Making marks on the medium of time

Selection layer is working, and moved quickly on to a first draft of a layer that implements a move operation. Some more refinement to do on this (at present it's unary with a fixed vector), but already sufficient to make one of my key problems clear. It goes back to a long phone call with Thomas around 1999, subsequently reviewed with students including Darren, Chris and Luke, where my main advice to them was to avoid it!

Warning - the rest of this post descends into metaphysics of Cognitive Dimensions ...

Here is the issue - if every "notation" can be regarded as a transcribed notation-layer that is derived from another notation, then how do we characterise those notation-layers where an information structure is represented by a sequence of user actions? (Note that the idea of a notation layer in CDs has no relationship (I think) to the layers in the layer language). The example we used in 1999 was interaction with a telephone or other device with minimal user controlled display, where the "notation" is a sequence of button presses. Since we were formalising the concept of notation as marks arranged on some medium (building on Ittelson), the suggestion was that this interaction-as-notation could be regarded as marks (i.e. events) arranged on the medium of time.

However, not only is the idea of marks-on-time an unintuitive extension of the concept of notation, this would be a notation with very peculiar properties. It has low visibility and juxtaposability (you can't simultaneously experience events at different times), high viscosity (you can't change things in the past), severe imposed lookahead (time is relentlessly linear) and so on.

So the essence of the layer language is that it is an attempt to reconfigure this situation. I am explicitly representing user actions (so far, only selection and move) as a sequence of temporal layers, but providing users with the ability to view, manipulate and replay those layers. The result could be a new intermediate ground between direct manipulation and program execution.

Hence the two main challenges in my project. The first is technical - providing a sufficiently powerful yet lightweight mechanism to present the whole sequence of user actions, in their full visual context, for explicit interaction. The second is "philosophical" - will the counter-intuitive starting point result in an intuitive experience? We already have one example of how to make this puzzling for users - the History Brush in Photoshop. It would be interesting to interview Photoshop users to see how many people have figured out what it does. (Rather like the long-ago project carried out by Kerry Rodden, looking at mental models of browser history).

Perhaps many people have thought about this before - the Photoshop History Palette is visually similar to the Layers Palette, so others must have wondered whether the two could be integrated. (Much like the proposals Kerry, Natasa, Rachel and I developed to integrate the history pane and back button in Internet Explorer).