Advertisement :)

I recently collaborated on this nextstop guide, started by one of our top users, Joana.  It’s really definitive, so check it out.  Also, check out the rest of nextstop and take a couple of minutes to recommend a few places you love.

Improving IDEs through Better Encapsulation and Organization

I don’t want header files that show just “public” methods and attributes, I want to filter what gets displayed in the main code window.  I might start out by only seeing the public methods.  Then, if I’m extending a class, for example, I might want to see the protected methods as well.  In either case I might just want to see the prototypes and their documentation comments — not the code.

On the other hand, if I’m working in a file, making changes, and I feel a method is getting over complicated, I might want to break it up into a few methods.  With a standard IDE, assuming I want a well-organized file, I have a couple choices: I could put this new function, right under the method that calls it (which works well if this new method is only going to be called by this one other method); or, I could put it in a “private” section at the bottom if the new method might be used by other methods in this file.  I could do other things to depending on the exact circumstances, but in any case there are a couple problems I see.  Let’s assume this new method is only to be called from the one other method.   First, I don’t care about this method when I’m browsing this file unless I’m specifically concerned about the method that calls it.  Second, I need to come up with some commenting or organizational mechanism or just remember that these two methods are connected.  That is, there’s no signal indicating a hard connection and to the IDE, the method call and the name of the new method are just pieces of text.  Something  that might help would be a drawing a box around the two methods and having the code of both be hidden unless specifically told otherwise.  In addition, the prototype for the new method could be hidden or ghosted as its secondary.   Lastly, I should be able to query the second function to see what methods call it and I should be able to query the first method to see which methods it calls.

The encapsulation offered by current languages and IDEs is too manual of a process.  I don’t want header files that show just “public” methods and attributes, I want to filter what gets displayed in the main code window.  I might start out by only seeing the public methods.  Then, if I’m extending a class, for example, I might want to see the protected methods as well.  In either case I might just want to see the prototypes and their documentation comments — not the code.

On the other hand, if I’m working in a file, making changes, and I feel a method is becoming overly complicated, I might want to break it up into a few methods.  With a standard IDE, assuming I want a well-organized file, I have a couple choices: I could put the helper methods right under the method that calls them (which works well if this new method is only going to be called by this one other method); or, I could put them in a “private” section at the bottom if the new method might be used by other methods in this file.  I could do other things too depending on the exact circumstances, but in any case there are a couple problems I see.  Let’s assume this new method is only to be called from the one other method.   First, I don’t care about this method when I’m browsing this file unless I’m specifically concerned about the method that calls it.  Second, I need to come up with some commenting or organizational mechanism or just remember that these two methods are connected.  That is, there’s no signal indicating a hard connection and to the IDE, the method call and the name of the new method are just pieces of text.  Something  that might help would be a drawing a box around the two methods and having the code of both be hidden unless specifically told otherwise.  In addition, the prototype for the new method could be hidden or ghosted as its secondary.   Lastly, I should be able to query the second function to see what methods call it and I should be able to query the first method to see which methods it calls.

In summary, what I’m looking for in an IDE are the following: keeping track of actual relationships between functions and variables such that tools can be developed to facilitate the automatic reorganization (at least of the appearance) of code and the querying of the relationships between pieces of code.

My Weekend Idea

Last weekend I had what I thought was a really cool file compression idea.  Suffice it to say, it didn’t work as well as hoped, but it’s sometimes useful to point out things that don’t work well.  Here was my idea:

Overview

The goal is to be able to compress and decompress files such that the compressed files

are significantly smaller than their original sizes while maintaining the same level of

digital information.  Decompression, should therefor, result in the exact same file as the

original.

Overview

The goal is to be able to compress and decompress files such that the compressed files are significantly smaller than their original sizes while maintaining the same level of digital information.  Decompression, should therefor, result in the exact same file as the original.

Technique

The first 8 bytes of a compressed file should be an unsigned int, the number of bytes in the original file.

There are 256 possible values for bytes.  In a large file, nearly all of these values will likely be used.  Bytes are normally arranged in order such that the location of a byte is only relative to the bytes that precede it.  If the values of bytes arenʼt evenly distributed throughout a file but rather have some localities in which they are more prominent than others, we should be able to achieve space savings by representing byte locations as encoded binary trees.

Example

Use a bit-field to represent the presence of a value in a binary tree node.  Assume an original file length of 16 bytes: (values in hex)

23 21 2f 75 73 72 2f 62     69 6e 2f 65 6e 76 20 70

In a file as small as this, no compression could be achieved using this method, but it will serve well as an example anyway.

Skipping the byte values that are not present and going through in ascending, numerical order, the byte values are:

20 21 23 2f(x3) 62 65 69 6e(x2) 70 72 73 75 76

Following are the binary tree representations as bit-fields, for each byte value.  Since we skipped non-present values, the first bit we show below for each value will always be 1, indicating presence of the value in the file.

The encoding of the trees in the following is level-based such that the first bit indicates the whole file, the 2nd and 3rd bits represent the first and second halves of the file, the 4th and 5th bits represent the first and second halves of the first-present half of the file (i.e. one quarter of the file, each), and so on.

20: 101010110 = 9 bits, 1 bit wasted

21: 110101001 = 9 bits, 1 bit wasted

23: 110101010 = 9 bits, 1 bit wasted

2f: 1111110010101101010 = 19 bits, 5 bits saved

62: 110010101 = 9 bits, 1 bit wasted

65: 101100101 = 9 bits, 1 bit wasted

69: 101101010 = 9 bits, 1 bit wasted

6e: 1011110100110 = 13 bits, 3 bits saved

70: 101010101 = 9 bits, 1 bit wasted

72: 110011001 = 9 bits, 1 bit wasted

73: 110011010 = 9 bits, 1 bit wasted

75: 110100101 = 9 bits, 1 bit wasted

76: 101011001 = 9 bits, 1 bit wasted

All skipped values would, in this case, incur a cost of 1 wasted bit. The total file size for this file would be: 8 bytes for the length encoding + 131 bits (bit as shown above) + 243 bits (skipped bits as described) = 55 bytes or 39 wasted bytes.

In larger files, the amount wasted should be significantly reduced.

Results

Sadly, the best results I found with this algorithm on a real file were about 5% compression (i.e. the compressed file was 95% the file size of the original).  On average however, the results were much, much worse, most files being around 170% of their original file size.  Oh well, fortunately it didn’t take long to code up.

Why are we still using text editors to code?

It’s interesting to realize that IDEs have changed so little.  They’re still basically fancy text editors.  They’ve added color coding, auto complete, and code folding, but don’t really help us understand our software any better.  It’s easy to understand why code is largely under-commented and under-decomposed, because the tools (and to some extent the languages) just aren’t designed around the way humans think about software development.  Here are some examples of features I’d like to see that only an IDE that helped depart from “text editor” status could bring:  Better encapsulation at all levels (file, package, application, library); micro parallelism; personalized display preferences; rich documentation; improvements for code reuse; access to online repositories; support for more advanced meta information.

I’ll add a separate blog post to address each of these topics.

Image Compression as Enumeration

Given the set of all useful images of a reasonable resolution, one could simply enumerate them and use the index as the image encoding.  For example, if there are 16777216^640*480 possible 24-bit 640×480 images, only a small fraction of those would ever conceivably be used as frames in a movie.  In fact, it would take every one on the planet (estimating 6.5 billion people) recording completely different stuff at 24fps, 1.82*10^2219415 years to create that many different frames.

Lets restrict the problem and say that useful images are those that would be used in Hollywood movies.  Assuming this is about 1000 films a year, each of which is around 2 hours at 24fps, this would be about 173 million frames produced per year or 173 billion frames produced if we produced movies for 1000 years.  Over 1000 years, this amounts to about 1.93*10^-2219423 of the possible frames.

To enumerate a single frame out of this set would take about 5 bytes, so a whole movie would take around 864000 bytes (or a little under 1MB).

Of course, it’s not really reasonable to expect one to be able to pick out just the frames that might be used in Hollywood movies.  Still, it might be possible to come up with some other definition of “useful” images, which is still likely to be a very, very small fraction.  Now the only trick is reproducing the image from the enumeration.

It would take a very clever person indeed to actually implement this…

ECMAScript to Native Compiler

I think it’d be really cool if someone developed a really good ECMAScript compiler that I could write server apps with.  I think the flexibility of the language and the number of people that are familiar with it (through JavaScript and ActionScript) warrants more investment in the language itself as a pure language.  I think it hands-down beats Python, for instance, in terms of ease-of-programming, power, and flexibility.  Further, variants like ActionScript show that optional type checking can be incredibly beneficial in many circumstances and allow for easier refactoring and code manipulation.

I could give examples as to why Python is very non-ideal, PERL is out-of-fashion, PHP just plain sucks, and Ruby is just another fad language that I won’t waste my time learning, but I’ll leave those as exercises for the reader…

Moving on.  Ideally, a compiler of this sort would produce a single executable that encapsulated the runtime environment in the binary so that people wouldn’t have to install a separate ECMAScript runtime library.

If a sufficiently well-supported and efficient product of this type was released, I believe it could really change the playing field for server scripting languages.  Next we just need jssp…

(If no one else works on this, I might make it a pet project of mine in a couple months — too busy at the moment.)

My First Idea!

Well, not really my first idea…

Problem

Bicubic interpolation always seems disappointing to me when trying to scale images to much larger sizes, going from screen resolutions to print resolutions for example.

Idea

Start with a large repository of high-quality photographic images (like all Flickr images).  From thumbnails that are much smaller than the original images (say 25% or 12.5%), start gathering, for neighboring colors, statistical information about the colors that make up the pixels in the original image.  In the images below, the letter ‘T’ is shown.  First in its original state, than in a blown up version of the 25% scaled image.  One can clearly see that the previously striking blacks have become various shades of gray.  In the third image, I’ve taken the 25% scaled image and rescaled it back to the original size using bicubic interpolation — no more hard lines, which in this particular example looks pretty bad.

With enough information about these relationships from enough images it seems that one could do better.  A human might say, a thin gray line between a dense black and dense white area should probably be divided up as segments of black and white, rather than blended with gray.  Being able to do this programmatically is clearly going to be difficult, but shouldn’t require AI — I think.

Clearly, it is impossible to recover the lost information caused by resizing an image with 100% accuracy.  However, the approach that bicubic interpolation takes is one of pure naïveté, which means there’s probably room for improvement.