Exploded and collapsed layers

Now a range of regions can be "exploded" into new layers in the stack, and stacked layers can be collapsed again into sets of regions. First random test image of this behaviour behaved correctly, but is somewhat disturbing.

Wittgenstein vindicated - types are not a hierarchy

In the days when object-orientation was new to most programmers, we were occasionally surprised by apparently profound failings of metaphysical conception among our colleagues. In one case that was notorious among members of our team for a long time, an early draft of a programming tutorial described inheritance in terms of the parts of a car - the class "car" should inherit the class "wheel", the tutorial explained.

Wittgenstein might have observed a problem in the metaphysics of object-oriented programming, rather than an ontological disconnect in the life of that author (and of every person who had read that draft before I saw it). Is it really the case that we always know what category is a kind of what other category?

Well, I spent most of today resolving a similar conceptual reversal. Until today, I had assumed that "content" was a kind of "layer". Now I've decided that layer is a kind of content. Of course the real problem is that neither is fully a kind of the other. Most development projects involve epistemological compromises, and expert users of the software we create (if the category terms are revealed) manage to subtly change their previous understanding of those words, in that context, to anticipate the actual system behaviour. Unfortunately, when programming in Java, reversing the position of two classes in the inheritance hierarchy is far from a trivial exercise.

Standardising geometry

I resisted the temptation, when I started coding, to create my own basic geometry classes - the Java libraries are full of dependencies on basic classes like Point and Rectangle. But they seem to require far more mundane repetition than you might hope from an object oriented language (C++ operator overloading and auto-type casting, where are you when I need you).

I didn't mind at first, because it was kind of meditative, doing all those repeated operations of x + width, y + height and so on. But I finally cracked, and created some new Point and Rectangle classes (well, Location and BoundingBox) that do everyday stuff more conveniently. Of course, now that I have 10,000 lines of code, much of it doing geometry, I wished I'd started a lot earlier. And quite a few things didn't work afterward.

Never mind. Almost back to where I started, and my ink has bounding boxes again.

Defining selection regions

I wanted to be able to define a subset of regions within a layer as an interactively-edited selection. This turned out to provide a nice example of the tension between formalised algorithms and "intuitive" user intention.

The basic idea was that the user should be able to draw a line around the items of interest, or that a previously defined image layer could act as a region selection mask. In either case, an enclosed region is clearly intent to select something.

The enclosure could be defined computationally using either a flood fill, or a convex hull. However, neither of those corresponds very well to informal graphical conventions. A partially closed boundary still indicates containment, but couldn't be flood-filled, while a concavity should certainly be respected, if specifically drawn to exclude something.

As a result, I spent nearly a day inventing a more informal alternative to the well-known formal algorithms. It seems to work as expected in most cases, and while less efficient than a good flood fill, is faster than a naive convex hull. I doubt there is much future to the field of informal algorithm design, but hopefully this will be good enough to suffice for the rest of my project.

Adding a play button

I'm very impressed by the new Lua-based end-user game programming app for the iPad, Codify. Their approach to the need to distinguish between code manipulation and game interaction is to use a simple play button. That's what we expect with macro recorders, of course, though many environments complicate that metaphor. In Scratch, for example, you can still interact with the code while the program is executing, and it can control its own execution state through use of the green flag.

So after a final Cambridge drink with Sam Aaron and David Coyle last night, where Sam pressed me on exactly what kind of execution this language does, I thought I should add my own play button today. Fortunately, it worked as expected pretty much straight away, and created a whole bunch of new layers on the fly.

Some usable parameterised layers

So it's taken a few days, but now I have operations with value parameters, and a usable subdevice that can be used for viewing and modifying the order of the layer stack.

The result can be used to create basic scripted interactions, with some interesting behaviours resulting from layer dependencies (this image).

From here, the next step is a choice between two optons: a) adding a greater variety of parameter and operation types (e.g. parameterising the image thresholding, or adding rotate and scale operations to the current vector translate), or b) extending the computation model (e.g. processing layer regions as sets - they can already include references to other layers, or interactive commands, so this would extend the power of the language pretty substantially).

Pain and sub-devices (drag and drop)

I've never seen an API for drag-and-drop that I liked. The Java 2 version is truly horrendous. Unfortunately, if you create notational sub-devices and want the window manager to help with them (as per a recent blog entry - it was nice to have the layer stack sub-device in a separate pane, and perhaps make it scrollable in future), then you find yourself in the horrible world of wanting to coordinate graphical behaviour between different window system components.

Hence a morning spent (wasted?) trying to get to grips with the Java DnD API. I really don't recommend it. After giving up on that, I spent an afternoon trying to roll my own. Guess what - if you press the mouse in one component, then drag into another, the new component does not receive any events (not drag events, not motion events). You can tell it has entered, but then no more information. The really crude response (as implemented by yours truly) is to collect the events in the drag source, and pipe them over to the drop target. The astute reader will have noticed that this means recreating the complexity of the DnD architecture that I was trying to avoid in the first place (well not quite, because I have avoided data transfer between applications). Even if I had got Java DnD working, apparently it doesn't work under Windows anyway.

So as of the end of the day, I can finally drag items off the layer stack and onto another layer. Here's a picture of an interim work product, sometime during today (drop target rendering not quite sorted!).

A reflective note - one of the rants that I give my HCI lecture class is that most GUI applications do not really implement direct manipulation. This is a classic example of why they don't - it's just so difficult to for application developers to do it properly. It's in the interests of the API vendors to make every GUI function as much like a disguised command line as possible - because that is easier for them to implement.