The perennial problem of abstraction

In the past few weeks, I've been grappling with some hard questions that were always on their way, but that I've managed to defer until now. After my second attempt at creating collection mapping operations ground to a halt, a third approach is proving more fruitful.

The essence of the problem is that, once operations deal with sets of values rather than single values, it is necessary to describe the structure and properties of those sets - a fundamentally abstract activity. In my case, collections of parameterised layers share some of their bindings, but not all. One approach to this has been to treat them as curried functions, but the challenge in doing so is specifying the curry bindings, and distinguishing them from the defaults that may be providing (user-perceived) desirable layer behaviours without having explicitly arisen from user choices. Most of my attempts to provide a user interface to this specification have been less than successful - complex, unreliable, and hard to plan in advance.

The latest approach has been to build more explicitly on the fact that parameters appear as graphical elements within the layer. Given the challenge of abstraction in this system is that the user must make a transition from interacting with images to interacting with bindings, I've allowed the bindings themselves to be manipulated as image elements, using the existing mask operations to select those parts of the binding set that should be preserved across a map. As I was building this, I was rather constantly reminded of the related challenges that Ken Kahn faced in ToonTalk, where Dusty the vacuum cleaner is used to specify value types by "sucking off" the value binding to leave the coloured pad. When Elizabeth used ToonTalk as a young child, this was one of the aspects that particularly upset her (along with everything that happened in the robot's thought bubble - the abstract world mode). A constant frustration was that a slip of the hand could easily delete the type, rather than the value. Easily undone, but an example of how error-proneness in the abstract notation carries significant weighted risk for attention investment. I hope that I've avoided this, by visualising the binding choices as a masked overlay that does not modify the original layer instance from which it is derived.

The next stage is to apply the binding mask overlay approach to cases where, rather than mapping a single layer (function) over a value collection, two collections are joined - typically with curried (function) layers in one (these may have resulted from a previous join), and value layers in the other. This seems like the right point to start experimenting with inference at last. In a previous attempt at the bind-then-play execution paradigm, I had created typed collections that could then be applied in contexts similar to those of single layers of the same type. However, the resulting constraints on users, for example that the collection type had to be maintained by constraining future addition of members, made this pretty cumbersome.

The inference approach that I'm about to start on, in contrast, allows users to place anything they like in a collection, while continuing to support aggregate bindings and maps. My intention is that the type of the collection will be determined (and visualised) statistically, based on the type of the largest proportion of its members. Members that are incompatible with this type can simply be passed over - the user may have intended them as secondary notation, or they may be accidental, or simply experiments to see what would happen. A user who is setting out to create an array value in a more systematic fashion can also do so, and the inferred type will be precisely as intended.

More computational toys for artists

Luke had developed the habit of calling his early experiments "computational toys" - image-based interactive systems that exhibit some kind of computational behaviour, thereby providing opportunities for creative experiences of flow and emergence. I think he recently referred to Palimpsest as a computational toy too, though it's rather more complex than anything of the kind we've built in the past. (Though to be honest, Michelle's positive experience was more by nature of a toy - the further complexities may yet turn out to be of minimal additional advantage).

Other blog readers have had their eye out for projects that they see as being related. All of them exhibit this same kind of behaviour, suggesting that the general category is recognisable, even to people without the precise theoretical fixations that Luke and I had. Two have come up recently:

• Sam Aaron pointed out the Recursive Drawing system built by Toby Schachman for his project Alternative Programming Interfaces for Alternative Programmers (I created the illustration above using this).
• Beryl Plimmer noted a presentation at CHI of the Vignette system from Igarashi's Design Interface Project, which uses gestures to replicate and fill with texture elements that can be dynamically created and modified while being used as drawing elements.

Both of these recent projects might be considered within the general class of procedural drawing systems - they are oriented toward artists rather than programmers, though Recursive Drawing is especially computational in its feel - as indicated by the project title. Vignette is sufficiently close to real professional design requirements that similar systems have been created fairly regularly in the past - from Sketchpad, to Paul Richens' Piranesi system for architectural rendering with its similar perspective and texture tools.

The challenging question for Palimpsest is, to what extent can these interactive procedural drawing systems be extended with more sophisticated computational abstractions, before they lose their appeal as computational toys? The answer relies on finding a particular balance between flow and attention investment, as well as sufficient quality of design. Recursive Drawing is rather similar to Palimpsest in its visual layout, including a side bar that resembles the Palimpsest layer stack. However it is far more elegant! Perhaps because its computational properties are quite specific and limited, but perhaps also because it is an art school product informed by computing expertise, rather than the other way round.

From naive physics to naive computation

Having done another couple of demos to interested observers, I've realised that the "translate" operation is not that useful. Translation is currently defined in terms of a vector, but the use of translation is most often to move something to the place you want it - the vector required to get it there is a byproduct, not the main point of interest. In fact, getting an object to a specified place is a real pain at present. The steps are:

• Create a point representation layer. The default behaviour, when a point representation is created over an image, is to use the centroid of the image as the default value, so this works OK.
• Create another point representation. The default behaviour, when another point is under this one, is to create a new randomised value. This is also OK, because an arbitrary translation can be used as the basis from which the user explores alternative values.
• Create a vector calculator. This will make a vector derived from the two points, which is also OK.
• Create a translation layer, which will use the original image, and the calculated vector, to move the image to the required point.

So there are four layers required to achieve one effect, which is arguably the most natural way to specify a translation anyway. Today I created a simpler variant of translation, which simply takes a point as a parameter and moves to there.

However, this does seem to point to a more general issue. My geometric operators are all nicely based on properly defined mathematical transformations. As a side-effect, this made them really easy to implement - each one simply corresponds to one of the Java Graphics2D AffineTransform operations. But perhaps we should be suspicious when the abstraction needed to implement a system function is too convenient - it's a sign that we might be imposing the programmer's mental model on the user.

In fact, every one of the geometric transforms has turned out to be not so closely related to the user applications that I've found interesting for those transforms. In a "naive geometry" approach, they could be described as follows:

• Translate -> "move it to here"
• Rotate -> "make it spin round" (usually as an animation that doesn't stop)
• Scale -> "stretch or squash" (not uniformly, but in various ways that drag handles can produce)

I suspect that I should discard the proper mathematical versions, replacing them with a move layer (which is what I had originally before making the more elegant mathematical transforms), a spin layer, and a layer that can reproduce any number of adjustments using the direct manipulation handles. This last will also overcome another problem. Although it was possible to generate a mathematical transform layer initialised according to handle manipulation, only the last manipulation was included - mainly because it would be so surprising to the user to see multiple layers of transform appear in response to a single button press.

Finally, a little reflection on "naive geometry". This takes me back to my MSc thesis, when I formulated a naive physics-style "qualitative trigonometry" that could be used for robust spatial reasoning by robots. Along with the rest of the naive physics / qualitative spatial reasoning movement in the mid 1980s, this could well have represented a user-orientation within the AI community, as we tried to create knowledge representations that were better aligned with common sense. At that time, the motivation was to replicate human problem-solving performance, rather than making computer "reasoning" easier for humans to understand, but the latter was  undoubtedly a side effect.

From a more HCI orientation, the Natural Programming project of Brad Myers and his students could be considered as an approach to defining a "naive computation" where program behaviour is described in common sense terms. I should probably have spotted this earlier, because I've been recommending it to Jamie Diprose, a student of Beryl's who is creating a visual programming language for healthcare robotics. I've encouraged Jamie to take an approach derived from John Pane's natural programming work, interviewing healthcare professionals to establish a vocabulary of domain concepts for use in his language. The healthcare robotics domain is sufficiently unlike the general purpose mechanical assembly domain of my own earlier work that I hadn't noticed the analogy to qualitative trigonometry, but now that I've noticed it, I could regard my project as creating a domain-specific language for exploratory image manipulation.

Some precedents

During James Noble's stay, he made a number of useful observations regarding precedents for some of the features in Palimpsest.

He thinks the overall feel is similar to that of early graphical constraint systems like Borning's ThingLab, or Smith's Alternate Reality Kit. The representation of constraints as "parameters" controlling the appearance of specific layer effects reminded him of the pointer constraints used in Myers' Garnet. Finally, the way that these are assigned default values from the context in which they are created (by type inference / extension from layers further down the stack) is like the implicit parameters in the Scala language.

References

Alan Borning. 1981. The Programming Language Aspects of ThingLab, a Constraint-Oriented Simulation Laboratory. ACM Trans. Program. Lang. Syst. 3, 4 (October 1981), 353-387.

Randall B. Smith. 1986. Experiences with the alternate reality kit: an example of the tension between literalism and magic. In Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface (CHI '87), John M. Carroll and Peter P. Tanner (Eds.). ACM, New York, NY, USA, 61-67.


Brad Vander Zanden, Brad A. Myers, Dario A. Giuse, and Pedro Szekely. 1994. Integrating pointer variables into one-way constraint models. ACM Trans. Comput.-Hum. Interact. 1, 2 (June 1994), 161-213.

Reducing abstraction hunger

It's a bit of a pilgrimage, out here into the forest, and we have few visitors other than family. So those who do make it over the mountains are guaranteed a demonstration of Palimpsest if they show any interest. Yesterday's guest, old friend Michelle Greenwood, was the first hands-on user apart from me and Elizabeth. I couldn't have asked for a more sympathetic third user - as an engineer and musician, Michelle is about as similar to my own motivations as could be managed. So where I advise students to find trial users as different to themselves as possible for critical assessment of their interaction design, I have managed to avoid any critique beyond the blindingly obvious.

Nevertheless, there is clearly some work to be done, even in addressing the blindingly obvious. A first priority is to make a smoother transition between direct manipulation of shapes and geometric transformations of those shapes. At present, it is possible to rotate, translate or scale a shape either by dragging handles (as in a conventional direct manipulation drawing editor), or by specifying a geometric transform (which can then be adjusted by directly manipulating its parameters). However, the more powerful of the two - the transform layer - must be requested explicitly. Only after doing this can the user modify the transform under program control. Use of the direct manipulation handles is transient, changing the state of the object, but with no opportunity to reproduce or adjust that state change.

In CDs terms, this represents a combination of abstraction hunger and premature commitment. The user can either create abstractions routinely (thus allowing them to be adjusted at any time), or create them only when necessary - but this involves premature commitment to know in advance whether the abstract alternative will be necessary. Michelle was (unsurprisingly) unsure about the difference between the options, and confused by the implicit state changes between them. I can deal with this by, whenever a shape is directly manipulated, saving the state changes as potential parameters for a transformation layer, and giving the user the option to create that layer.

This does actually return to one of my first blog discussions - probably going back to last October. The was the point at which I created a "move" layer that was automatically generated in response to the user dragging of any content. This turned out to be rather annoying, as move layers quickly accumulated, appearing rather "heavyweight" for minor adjustments (a form of abstraction hunger, even though the abstractions were created automatically, in programming-by-example style). As a result, I created the direct manipulation handles in December. Now I understand that they should always have been combined, allowing transition from direct to abstract.

On the positive side, Michelle liked using Palimpsest, even in its prototype form. She said it was a lot of fun to play with, and immensely superior to her son's current experience as a student of introductory graphics programming, where it has taken him weeks of work to achieve the visual effects that she could explore within seconds. If I had any faith in Likert-scale evaluations of user experience, I could report an early confirmation sample!

Manipulate, Automate, Compose

Margaret Burnett has been a great supporter of the Attention Investment model of abstraction use, in large part because it provided the motivation that led to her design strategy of Surprise, Explain Reward, that has proven so valuable in the development of end-user debugging systems. After many weeks wrestling with the problem of where the "language" is in my layer language, I realised that I have unconsciously been relying on my own design strategy, similarly motivated by Attention Investment, but until now not articulated.

We can call this strategy "Manipulate, Automate, Compose", in homage to Margaret's own three-part strategy for user experience design. (If you want to cite this, contact me first - there's a chance I might eventually decide to publish in a slightly revised form).

My hitherto unnamed, but analogous, strategy dates back to the invention of the "Media Cubes" over a decade ago - one of the first applications of Attention Investment. My reasoning at that point was that users would become familiar and comfortable with the operation of the individual cubes, in the course of operating them as simple remote controls. Once those unit operations had become sufficiently familiar in this way (perhaps over a period of months or years), the physical cubes would naturally start to be treated as symbolic surrogates for those direct actions, and used as a reference when automating the action (for example, setting a timer to invoke the relevant action). Once the use of references had become equally familiar, the user might even choose to compose sequences of reference invocations, or other more sophisticated abstract combinations. All of this is consistent with Piagetian education principles, and indeed with Alan Kay's original motivations in applying those principles to the design of the first GUIs.

What we have lost sight of since then is the second two steps in this process - most GUI users are stuck at the "Manipulate" phase, and are given little encouragement to move on to Automating and Composing - precisely the points at which real computational power becomes available. The various programming by demonstration systems (as in Allen Cypher's seminal collection) aim to move to the Automate step, while programming by example uses additional inference methods that Compose them as a map over different invocation contexts.

Typical approaches to programming language design often proceed in the opposite order - the mathematical principles of language design are fundamentally concerned with composition (for example in functional languages). Once the denotational semantics of the language are established, an operational semantics can be applied, so that the language can be applied to things that the user wants to automate. Finally, a programming environment is provided, in which the user is able to manipulate the notation that represents these semantics. After a language has been in use for a while, live debugging environments might even provide the user with the ability to directly manipulate objects of concern to themselves (rather than the elements of the language / notation, which for the user are a means to an end).

Those viewing the Layer Language up until this point (Beryl Plimmer's workshop in February, and James Noble's observations last week) have commented that I've provided a number of interesting user capabilities, but that they don't see where the "language" is. To be honest, I've had the same concern myself - it looks so unlike a normal language that it has taken some determination to persist along this path (not for the first time - the Media Cubes suffered from the same problem, to the extent that Rob Hague felt obliged to create a textual equivalent in order to prove that it was a real language, even though operating the direct manipulation capabilities of the system by typing in a text window would have seemed slightly ridiculous).

So after James' departure a couple of days ago, I returned to thinking about execution models. Before he arrived, I had implemented a simple event architecture that allows events to be generated from visual contexts (and hence automated), and during his recording session last week, I took the chance to implement a persistence mechanism for collections of layers (hence making composition more convenient). It's pretty clear that once these are working smoothly, they will provide a reasonable execution model, that is consistent with the visual appearance and metaphor of the multiple layers. Furthermore, users will be able to apply these in the same way as with Media Cubes and some of my other design exercises in the past - the system is quite usable for direct manipulation, with those experiences giving users the confidence for the attention investment decisions of automating their actions and composing abstract representations of them.

So this is the design strategy expressed in the Layer Language - the same one as in Media Cubes, and various other systems. The user can achieve useful results, and also become familiar with the operation of the system, through direct Manipulation that provides results of value. The notational devices by which the direct manipulation is expressed can then be used as a mechanism to Automate them, where the machine carries out actions on the user's behalf. Finally, all of the functions that the user interacts with in these ways can be Composed into more abstract combinations, potentially integrated with more powerful computational functions. The same Manipulate, Automate, Compose benefits can be seen in products such as Excel - hence the spreadsheet analogy that I have been making when explaining my intentions for the Layer Language.

Furthermore, I realised that the past 6 months work represent a meta-strategy for applying attention investment to design. I have intentionally deferred the specification of the language features for Automation and Composition until I had gained extensive experience of the Manipulation features. In part this comes from the hours of "flight-time" in which I've been using those features. But even more, it comes from the fact that I've been implementing, debugging and refining the direct manipulation behaviours as I've gone along. This has meant that the abstract aspects of the language design have been formed from my own reflective experience of the use and implementation of the direct manipulation aspects. A name for this meta-design strategy might be "Build, Reflect, Notate".

I suspect that this may be the way that many language designers go about their work in practice. I had several illuminating discussions with James about the work he and his collaborators are currently doing on the design of their Grace language. James has a great deal of expertise in architecture and usability of object-oriented languages (we had some enjoyable discussions on the beach, comparing my experiences of Java coding over the course of my project so far), so like most language designers, he is creating a language informed by his experiences as a Java "user". The difference between that kind of project and mine, however, is that the user domain in which his design work is grounded is the manipulation of OO programs by his students, in the context of teaching exercises to train them in OO programming. This is perfectly appropriate to his project, since Grace is intended as a teaching language. However, it means that the attention investment consequences arising from his use of this meta-design strategy are very different from mine. Rather than the end-user programming principles of Manipulate, Automate, Compose, his language will support some educational principles related to the the acquisition of programming skill (maybe Challenge, Analyse, Formulate). Perhaps Margaret's Surprise Explain Reward arose in a similar way from the same meta-design strategy - I look forward to discussing it with her at some point.



References

Surprise, Explain Reward: Aaron Wilson, Margaret Burnett, Laura Beckwith, Orion Granatir, Ledah Casburn, Curtis Cook, Mike Durham, and Gregg Rothermel. 2003. Harnessing curiosity to increase correctness in end-user programming. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '03).

Programming by Demonstration: Allen Cypher, with Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky. (1993). Watch What I Do: Programming by Demonstration. MIT Press

Media Cubes: Blackwell, A.F. and Hague, R. (2001). AutoHAN: An Architecture for Programming the Home. In Proceedings of the IEEE Symposia on Human-Centric Computing Languages and Environments, pp. 150-157.

Grace: The Grace Programming Language

http://gracelang.org/

Some (routine) "tricky corners" in Java

When Sam and I were trying to develop analogies between jazz improvisation and live coding, Nick Cook told us that preparation for performance might involve rehearsing "tricky corners" that can arise in a particular improvisation genre. Although I'm not coding anything live, there are some kinds of software development task that always leave me feeling a little nervous, no matter what language I'm working in. One of these is management of realtime threads. Another is interacting with a new file system for the first time.

So in the past week, I've taken some deep breaths, and plunged into both of these. Both are routine enough, and every Java programmer (or teacher of Java programming) knows what to do - but as always, newcomers have to learn it themselves.

I'd already created a number of Java animation threads during the project, and had got accustomed to the use of SwingWorkers and timers to ensure that the system still responds to user interaction while doing memory-intensive image shuffling. I'd speeded up some of these animations with better double-buffering of the layer rendering, separating fast animated components from relatively static elements that can be updated less often,

But lately there have been too many animations all running concurrently, now that some state variables are animated (the rate and event value types), and these control other animated layers (paths) that themselves interact with or control other layers. After some encouragement from James Noble (still visiting), I therefore aggregated all the animated updates into an update task list that is reviewed and updated in a single animation thread - James tells me this is how Smalltalk does it, and that's good enough for me! However, several days of thread debugging ensued. James was off at a recording session, and it turns out could have told me immediately what my problem was - but what I thought was a subtle deadlock problem turned out to be the simple fact that exceptions in background Java threads die silently if the thread invocation wasn't made within a try-catch block. So my problem was simply an invisible null pointer exception. Once I'd learned to catch this and dump a stack trace, it became relatively easy to debug the threaded animation.

The second plunge was into Java persistence, which I've never had any reason to use before. It seems about time that my "programs" (quote marks necessary until I persuade James that this really is a programming language) can be saved by the user - at least, it is now possible to create things sufficiently complex that I might want to recover them in future. The illustration on todays post is a "Mr Watson, come here" - the first layer successfully saved and dragged back into a new Palimpsest session. As with my complex animation thread, there had been good reason for nerves - the order in which persistent objects are written to a stream is not that easy to anticipate, meaning that getting a complex class hierarchy persisted involves several hours of trial and error, with many exceptions along the way.