A first bit of serendipity?

A little reflection on my discovery earlier this week ...

The idea that a "semantic" region of a photo could be indicated by a shadow so that it appears to float above the rest of the photo (blog post on Monday) was an accidental result of the fact that I was working on "treatment layers" and needed a second example of a treatment. The first had been experiments with a "rubbing" layer that would transfer painted parts of the layer under it (credit to Mike Trinder, who was the first I saw using that metaphor, in his PhD supervised by Paul Richens at the Martin Centre).

As a second treatment, I wanted to implement an algorithmic image filter that would have an effect a bit like scumbled paint finishes (interesting because those finishes blur more semantically diagrammatic boundaries in favour of rich textures). This was going to be my first pixel-shifting transformation, so I needed an example to draw on. The only example I had to hand of a context-based image filter was the Gaussian blur algorithm that I'd used to implement shadows a couple of weeks ago. This was pretty complex, so after 15 minutes exploring the code to see if I could make a scumbling version, I just thought I'd see what happens when the original shadow code was applied to my source photo. Nothing, as it turns out, because the shadow was calculated from the alpha channel, and the photographic test image had a uniform alpha. Fortunately, the rubbing layer I'd just implemented didn't have a uniform alpha, so I used that instead. This visually transformed the appearance of rubbing into floating patches of picture. I then played around with superimposing these on various backgrounds in order to see them better. When superimposed on the original picture, this resulted in the appearance of a "semantic" layer that had been specified by the previous user interaction of rubbing (the user naturally rubs areas that are "interesting" for some reason).

Serendipity relies on the observer being prepared to recognise the novel occurrence. In this case, my recognition of the potential application was prompted by one of the images I had created in my own PhD experiments. There, I had made a number of photographs of playground equipment look more "diagrammatic" by rendering some of the picture elements in higher contrast over a reduced contrast background layer of the original photo.

In my discovery on Monday, it had turned out that the shadows were easier to see if a further alpha-blended white layer was rendered between the photo and the shadow. This happened to create a similar contrast-reducing effect as in my PhD experiment, and resulted in the "rubbed" elements seeming clearly diagrammatic.

A journey to AbstractLand

Time for tubby bye-bye.

Semantic images

School half-term this week, so I have my resident end-user (Elizabeth) on hand to play with prototypes. This allowed me to make a first attempt at the "semantic layer" that Luke and I long regarded as an oxymoron. 

Finding performance limits

As I see it, Moore's law tells us that if a research prototype runs fast enough, then you aren't doing it right! I'm expecting that this thing will not run at the speed of a respectable language, at least not on the development platforms I'm using. I've therefore been waiting for my experiments to run into performance limits. I've finally found one, after realising that you can't have nicely antialiased lines if you are going to do affine transforms - for that to work, the transform has to be interpolated. Bicubic interpolation produced nice smooth results on this image (a first stack made of index cards rather than book pages), but runs slowly enough that the page turn animation was already finished before the pages were rendered. An entertaining (not) half-day ensued, as I played with ImageObservers and MediaTrackers in multiple animation threads. At the end of this, I just gave up - I don't think I understand how they work, and index cards don't need to bend anyway.

Tyranny of realism

After another day improving the rendering of my metaphorical book, I stop to think about the complaints by Alan Kay and Ted Nelson, back in 1985, about the detrimental effects of user interface metaphor. Kay argued that computer interaction should be magical rather than mundane. More extravagantly, Nelson compared over-arching conceptual metaphors to totalitarian regimes, constraining the (otherwise abstract) freedom of the designer. 

So in the course of making this more "compelling" (realistic), I learned quite a lot about animation, gradient fills, double buffering and so on. By the end of a day, I was just starting to implement a nice animation of pages that curl from the corner (building on a nice Java example by Arnoud Philip), when I remembered that I didn't really think a book was the right metaphor anyway. It was surprising how far I had been distracted by the appeal of naive realism. I'm still pretty certain that physical simulation does help users to become immersed in the abstract world (the physical realism of the touchscreen swipe-to-scroll is what really convinced me), so I'll return to this in future.

However, I'm not going to spend any more on the book. A stack of index cards might be a better starting point.

Making a metaphor

One of my key problems is how to present a concept of the "layer" that doesn't scare people off in the way that Photoshop layers seem to do. This brings us back to metaphor - much as I hate to admit it. So yesterday's task was a first attempt at providing a layer navigation visualisation that is more easily recognised through use of a physical metaphor. The geometry is a little crude here (using only efficient AffineTransforms), but interacting with this for a bit was sufficient to suggest that a book is probably not the right metaphor for the layer language.

There should be a name for ...

... that feeling you get when you've defined a new superclass, but more and more of the interface of the original class starts to migrate into it, as you realise that the abstraction wasn't sufficiently isolated from the specific properties of the original class. In the pre-object-oriented days, I could have called it "globalisation".

I spent quite a lot of yesterday doing this, having separated the language into "content" layers and "operation" layers. It kept turning out that these are harder to separate than you might think. Perhaps not a problem for me, if the language goes all the way to being a visual Lisp, with lots of run-time type checking to make my life easier (and trip up incautious users).