A visibility and juxtaposability problem solved

I've been waiting for the point where my expert knowledge of Cognitive Dimensions will change what I would have done anyway (at least, in a conscious manner, rather than just general motivation). So yesterday, I figured out that there was a serious visibility/juxtaposability problem, resulting from references to layers that might have been "collapsed" out of the stack. It's possible (sometimes) to retrieve a layer by expanding the reference again, but this isn't an ideal option if your problem was simply that you'd forgotten what was on it (a trade-off between viscosity, visibility, or a hard mental operation - the three-way trade-off that Thomas described by analogy to the ideal gas law, back in the 1998 tutorial). In fact, even if the layer is somewhere in the stack, it might not be easy to find it - a fundamental problem with the whole layers idea. References have always had the default behaviour that they will navigate you to the right place in the stack, but this is annoying too - because you lose your previous context, which doesn't allow juxtaposability.

So the outcome today was to implement a function that allows you to hover over any layer reference, and get a preview of that layer's contents, whether or not the layer is currently present in the stack (illustration here - the ink spiral is a layer that has been deleted from the stack, but still has a reference button - I've hovered over that button to see the contents of the original layer).

As so often, the simple concept was harder to implement than I thought. JavaScript apparently supports a 'hover' callback as a standard attribute of UI widgets, but Java components do not. Implementing hover within my mouse interaction model meant that I had to grapple with threads more seriously, to figure out what to do when the mouse isn't moving. On the positive - 'craft' reflection - side, it's an oddly pleasing feeling when you finally have some purpose for the spurious empty implementations of interface methods that Java makes you leave around all over the place. This was the first time that I've had anything I wanted to do with the mouseMoved (as opposed to mouseDragged) method of the MouseMotionListener.

Exploded and collapsed layers

Now a range of regions can be "exploded" into new layers in the stack, and stacked layers can be collapsed again into sets of regions. First random test image of this behaviour behaved correctly, but is somewhat disturbing.

Wittgenstein vindicated - types are not a hierarchy

In the days when object-orientation was new to most programmers, we were occasionally surprised by apparently profound failings of metaphysical conception among our colleagues. In one case that was notorious among members of our team for a long time, an early draft of a programming tutorial described inheritance in terms of the parts of a car - the class "car" should inherit the class "wheel", the tutorial explained.

Wittgenstein might have observed a problem in the metaphysics of object-oriented programming, rather than an ontological disconnect in the life of that author (and of every person who had read that draft before I saw it). Is it really the case that we always know what category is a kind of what other category?

Well, I spent most of today resolving a similar conceptual reversal. Until today, I had assumed that "content" was a kind of "layer". Now I've decided that layer is a kind of content. Of course the real problem is that neither is fully a kind of the other. Most development projects involve epistemological compromises, and expert users of the software we create (if the category terms are revealed) manage to subtly change their previous understanding of those words, in that context, to anticipate the actual system behaviour. Unfortunately, when programming in Java, reversing the position of two classes in the inheritance hierarchy is far from a trivial exercise.

Standardising geometry

I resisted the temptation, when I started coding, to create my own basic geometry classes - the Java libraries are full of dependencies on basic classes like Point and Rectangle. But they seem to require far more mundane repetition than you might hope from an object oriented language (C++ operator overloading and auto-type casting, where are you when I need you).

I didn't mind at first, because it was kind of meditative, doing all those repeated operations of x + width, y + height and so on. But I finally cracked, and created some new Point and Rectangle classes (well, Location and BoundingBox) that do everyday stuff more conveniently. Of course, now that I have 10,000 lines of code, much of it doing geometry, I wished I'd started a lot earlier. And quite a few things didn't work afterward.

Never mind. Almost back to where I started, and my ink has bounding boxes again.

Defining selection regions

I wanted to be able to define a subset of regions within a layer as an interactively-edited selection. This turned out to provide a nice example of the tension between formalised algorithms and "intuitive" user intention.

The basic idea was that the user should be able to draw a line around the items of interest, or that a previously defined image layer could act as a region selection mask. In either case, an enclosed region is clearly intent to select something.

The enclosure could be defined computationally using either a flood fill, or a convex hull. However, neither of those corresponds very well to informal graphical conventions. A partially closed boundary still indicates containment, but couldn't be flood-filled, while a concavity should certainly be respected, if specifically drawn to exclude something.

As a result, I spent nearly a day inventing a more informal alternative to the well-known formal algorithms. It seems to work as expected in most cases, and while less efficient than a good flood fill, is faster than a naive convex hull. I doubt there is much future to the field of informal algorithm design, but hopefully this will be good enough to suffice for the rest of my project.

Adding a play button

I'm very impressed by the new Lua-based end-user game programming app for the iPad, Codify. Their approach to the need to distinguish between code manipulation and game interaction is to use a simple play button. That's what we expect with macro recorders, of course, though many environments complicate that metaphor. In Scratch, for example, you can still interact with the code while the program is executing, and it can control its own execution state through use of the green flag.

So after a final Cambridge drink with Sam Aaron and David Coyle last night, where Sam pressed me on exactly what kind of execution this language does, I thought I should add my own play button today. Fortunately, it worked as expected pretty much straight away, and created a whole bunch of new layers on the fly.

Some usable parameterised layers

So it's taken a few days, but now I have operations with value parameters, and a usable subdevice that can be used for viewing and modifying the order of the layer stack.

The result can be used to create basic scripted interactions, with some interesting behaviours resulting from layer dependencies (this image).

From here, the next step is a choice between two optons: a) adding a greater variety of parameter and operation types (e.g. parameterising the image thresholding, or adding rotate and scale operations to the current vector translate), or b) extending the computation model (e.g. processing layer regions as sets - they can already include references to other layers, or interactive commands, so this would extend the power of the language pretty substantially).