Bret Victor's "Dynamic Drawing"

Sam Aaron pointed me to a talk by Bret Victor, a designer who has worked at Apple among other places. Victor says that "creators need an immediate connection to what they're creating" - the same principle of liveness that Luke, Chris and I have been promoting for programming languages more generally.

Checking out Victor's website, I see that he has already been playing with the idea of new languages to support this kind of experience when doing graphic design. He calls these "Dynamic Pictures" - an interesting comparison to my original proposal to create "Living Images" some years ago.

Some of the things he says in this little essay could easily be repeated as motivations for my current project:

• I believe that dynamic pictures will someday be the primary medium for visual art and visual explanations.
• Dynamic means that the picture changes when you change some input. A dynamic picture looks different in different situations
• Dynamic drawing means that the artist creates the picture by directly manipulating the picture itself, instead of working with some indirect representation that doesn't resemble the art-in-progress.
• With today's tools, dynamic design requires creating pictures by writing text. It is only because we are so accustomed to this situation that we don't recognize how bizarre, even barbaric, it is.
• A "user interface" is simply one type of dynamic picture. [Apple designers] were dependent on engineers to translate their ideas into lines of text. […] It's fashionable to rationalize this helplessness with talk of "complementary skillsets" and other such bullshit. But the truth is: An author can write a book. A musician can compose a song, a animator can compose a short, a painter can compose a painting. But most dynamic artists cannot realize their own creations, and this breaks my heart.

He offers the following illustrations of the text/picture divide in some typical artist/end-user environments:

Tools for sloppy programmers

Most programming languages have to keep things neat and tidy, but in the past week I've realised at a couple of points that I don't want to place too much responsibility for tidiness on my users - and that sometimes untidiness is likely to be positively welcome.

The first of these was was the outcome of a struggle to deal with circular references in dependency chains. Until today, I'd been trying to track these down at the time the circularity was created - every time a new reference is added, I search the dependency tree from that point, to see if there are any cycles. The problem with this was how to resolve the cycle. Initially, my approach was to save the user from themselves, deleting the final link that causes the cycle. However, this wasn't as sensible as it first seemed. I didn't want to delete the new link that had just been created - the cause of the problem was more likely to be an old link that was no longer so interesting to the user. But as it turned out, it was much harder than I thought to guess which dependency is uninteresting, or even "old". As more interactivity has been appearing in the language, dependencies accumulate pretty quickly, and many of them would be bad ones to disconnect (for example, between an a value interactor and the value that it manipulates).

The resolution to this was a realisation that circular dependencies are my problem, not the user's. I've decided to allow cycles in the graph, and I simply have to remember not to follow them when making updates. In fact, this is a lot easier than deciding how to fix them.

The second sloppy feature was discovered partly through boredom, as I was implementing and testing map functions. I was often creating multiple values to map over, and it was necessary to assign or adjust each one before carrying out the map. To save myself time, I decided to initialise each new value to a random number, rather than assume that the user will assign it or derive it from something sensible. As it turns out, this has been a lot of fun - I keep discovering interesting new colours that I wouldn't have thought of creating. Another case of a behaviour that wouldn't really be considered in a "serious" programming language, but for sloppy programmers, getting variables with random initial values seems kind of cool.

Computational Stuff - I need some types

After Beryl and Robert's challenge last week, I've been implementing some new layer types that are starting to look like more conventional CS. Monday devoted to creating a decision layer that either ignores or passes its source depending on comparison of two value parameters. Tuesday spent playing around with value substitution - at first interactive, then in response to a map treatment that maps a parameterised layer over a collection of alternative values.

All of these experiments have required more sophistication in the dynamic type processing. Until now, type equivalence (for example of value fragments and value layers) has been rather crude, with layer references and "promotion" of fragments simply hard-coded with "instanceof" to retain necessary types. A first attempt provided my own set of type identifiers, with all content elements being able to report their types. However, this is going to involve a lot of logic distributed around the place, detraining type equivalence, type conversion, and maintaining type/behaviour relations.

It's starting to look as though I should define value types as Java interfaces to the layers that support them, with type conversions and equivalence supported via multiple interfaces for some kinds of layer. In this case, Java reflection should provide all the dynamic type processing that I need, rather than constructing an independent type mechanism.

An Overview of the Language!

Beryl Plimmer welcomed me to Auckland with a one-day workshop, coinciding with a visit by Gem Stapleton from Brighton. Participants from Beryl's included new post-doc (previously PhD student) Rachel Blagoevich, with current PhD student Jamie Diprose and summer research assistant Ryan, as well as department lecturer Robert Sheehan. All are interested in notation design, with Robert being more oriented toward end-user programming, and the others theoretical and usability principles (especially future plans to use tangibles and multitouch alongside the group's existing expertise in sketch recognition).

This audience provided an excuse for my first attempt at an overview so far - I try to reproduce that summary below.

Motivation

This project is mainly inspired by my work with various collaborators in recent years, creating end-user language/notations for use by artists working with digital media. In the context of design research, collaboration with artists offers a valuable counterpoint to work with engineers, arguably a necessary element of exploring the full range of requirements for professional design tools. Working with artists also draws attention to particular parts of the attention investment / user experience equation, where end-users engage in construction activity for its intrinsic rewards (including, for example, flow experiences).

Observations of artist collaborators has also drawn attention to two interesting technical requirements. One is that many professional artists find the keyboard an obstacle to creative practice (thinking here of observations of Random Dance, and discussions with Bruce Gernand). This revives the long-standing challenge of the "purely visual" language, of which Dave Smith's Pygmalion, George Furnas's BitPict, Clayton Lewis's NoPumpG, and Wayne Citrin's VIPR have been stimulating if rather impractical examples. Although often mooted, such languages have seldom seemed compelling on machines that do, after all, have keyboards. In the age of tablets and touch interaction, keyboards have suddenly become a real inconvenience and hindrance in routine interaction, so this seems a better time than any to explore text-free notations. Everything that I have done so far, in principle, will be completely usable on a tablet (assuming I ever manage to port the graphical operations to Android Java).

Features

This language allows users to combine source images (photographs, simple shapes or ink) and treatments of those images.

Sources and treatments are layered on top of one another, in the manner of a palimpsest (a potential name for the language!).

All layers can be broken up into fragments, with these fragments being combined in a visual collage (another potential name).

The behaviour of treatments and fragments can be modified by adjusting values. Values themselves are also represented as fragments or layers - these can define visual parameters such as proportion/intensity, vectors, colours and so on.

Interaction with the system involves creating new layers, and superimposing them on layers already created. The resulting stack of source, values and treatments provides both a visual palimpsest, and a historical record of the process by which it was achieved.

The other element of the visual interface, apart from the main image manipulation area, is a visualisation of the stack of layers. The usability implications of the layer stack can be considered in terms of conjoining two standard features of Photoshop: the layer window and the history window. Some disadvantages of doing this can be understood by analogy to the (notoriously hard to understand) Photoshop history brush. The workshop audience expressed concern about this aspect of the user experience (they could have, but did not, express it in terms of premature commitment and viscosity). However, from my perspective, this mingling of process and product is also characteristic of artistic practice - our attempts to regularise it in the projects with Bruce Gernand and Random Dance provided them with history management mechanisms, but not mechanisms that they integrated fluently into their creative activity. We'll see how this enforced intermingling plays out.

Computational properties

The primary data structure facility at present is a layer whose fragments represent references to other layers. These fragments can be used either to modify or refer to particular image sources, treatments or values from other layers, or to preserve a whole stack as an ordered collection of fragments within a single layer.

The workshop audience was sceptical, that this data structure presents sufficient complexity to be described as a programming language. The behaviour of the system at present is rather reminiscent of Hypercard, but without any of the Hypercard script - from a traditional programming language perspective, it seems as though I have removed the only linguistic element of Hypercard (ignoring the notational analysis perspective, that the card representations and interaction environment provide a powerful notational system). I made some analogies to early programming languages - fragments as being like dictionary words in Forth, or stack sequences and references as like primitive LISP - with all symbols replaced by images. However, although these might support behavioural interpretation, they are not behaviours in themselves. A final poor analogy is that, as with the lambda calculus, these arrangements of images support a computational interpretation, in principle including sufficient expressive power to implement a Turing machine. (This interpretation won't help much in Auckland - it seems that the lambda calculus is not taught at undergrad level).

Nevertheless, the advice given by Robert and Beryl was that, if I wished to have this recognised as a programming language, it should have some behaviours that look like program execution. They recommended iteration, or evaluation of conditions. I've taken this on board, and that's where I'm going next ...

Back on board, from Karekare NZ

I should have warned readers (if there were any!) of the scheduled 2 months family time through December and January, as we visited with all the various branches of the family around New Zealand. Now back in action, installed deep in the bush at Karekare, and visiting with Beryl Plimmer's group at the University of Auckland.

The first day of coding on return was a little rusty (editor commands forgotten, basic maths skills lost), but I managed to add a couple of incremental functions - an interactive ellipse class to join the rectangle, and upgrading of the basic line class to allow interactive editing.

After this, time to get stuck in and add support for a fundamental new concept that I dreamt up during a jetlagged night in Singapore.  I've decided that the stack view of layers should be a first class object, interchangeable with collections of layer references. In order to make this obvious to users, I think this equivalence should be visible in a graphical transition from collection to stack. That's relatively straightforward, but still lots of geometric detail to be figured out in the animation. A quarter of yesterday spent on false starts for representing that geometry sensibly, and another quarter thinking that I had created a sublte reference contention problem in my animation thread (in fact, had just forgotten to initialise the reference at all).

An interesting consequence of the 2 month break is reluctance to get my hands dirty in refactoring code that I was blithely chucking around at the start of December. Perfectly happy now to make incremental changes or add features, but the core architecture has now acquired substantial viscosity after being "paged out" of my head. Hopefully the fluidity will be recovered before too long.