Len Lye, Swinging the Lambeth Walk, 1940

I've long been a fan of Len Lye's work. Lye was a member of the GPO film unit, whose landmark work was the wonderful soviet-influenced Night Mail (The GPO Film Unit was part of the Post Office). The GPO Film Unit collected works are available on DVD.

Lye's technique often involved painting directly onto celluloid. Full filmography.

Stumbled across a double DVD of Kenneth Anger's films (part 1, part 2). Totally great.

Most Fridays we hold talks known as 'osmotics' at Last.fm. Their purpose is to share understanding about the various things that are going on in different departments; an open session which anyone can use to present recent work. It's a response to our growth, as it's now impossible to know what's happening throughout the company at any given point in time.

A few weeks back I presented some of the work we've been doing on interaction path analysis. As web developers we're completely spoilt for usage information on our apps, and interaction histories are sitting in our apache logs the entire time. We've been using tools to mine these and better understand real interaction flows on our site, in order to improve our understanding of common use cases.

You'll often find that your users don't use your software exactly as you designed for. That's natural, and as web developers we need to be equipped to observe these flows and adapt/optimize our software accordingly. As Stewart Brand observed of buildings, so too good software should learn from usage patterns rather than working against them. A good app should flow.

We use a tool suite from Omniture - the pricing may well be out of reach of most startups, but anyone can mine apache logs to get similar information. Bear in mind the presentation assumes no technical knowledge and was prepared in about 20 minutes - it's more of a lightning talk than anything. I've blacked out some sensitive bits of information.

→ Interaction Path Analysis - slides with annotations (1.1MB)

Charles from Trampoline Systems organized a workshop on emergent democracy last week in Shoreditch. It was kind of interesting. I thought I'd post some quick notes.

First of all here's a rough definition of the phenomenon of emergence: A synergistic property or behaviour of a system that cannot be explained solely by the sum of its component parts; when the organization of a system exhibits dynamic behavioural properties that exist on a macro level in relation to its component parts. Systems that arise out of emergent phenomena are said to be complex.

One example of emergence is the development of the human foetus from the division of a single cell, through further division and specialization, into a complex human body. See Johnson's book Emergence for more examples.

Throughout the discussions it was clear there was a lack of clarity on the difference between strategies for refining democratic participation and a truly emergent system of governance. Emergence resists strategization because by definition it cannot be governed. In an emergent system it is precisely the lack of governance of the whole that allows complexity to arise from the behaviour of individual components reacting to particular constraints (governance at the component level).

This creates an interesting paradox of the expression emergent democracy. The question that the expression raises is this - what changes in participation and organisation will it take for democracy to produce a truly complex system of governance?

Let's take the complex system that arises out of traffic flows in a metropolis as an example of governing complexity. Individual agents (car drivers) are free within a set of constraints (the rules of the road, the placement of traffic lights, one-way streets etc) to make a set of choices about what route to take, how fast to drive, where to stop off and what time to travel. Car drivers really have a lot of freedoms in this system. Traffic flow is a complex system which urban planners try desperately to govern - with differing levels of success, because they don't often have good enough tools to predict the resulting behaviour of 100K+ car drivers. Because the simple rules governing the constraints on drivers mentioned above are increasingly not enough to deal with congestion in cities, governments begin adopting much more holistic governance rules (e.g. a congestion charge).

Admittedly, this is a field seeing a lot of research as a result, but it's an example of how complex systems and governance are often in tension. The rulesets that define cell replication and the instructions encoded in DNA do not attempt to govern the whole that is the human body. If you attempt to govern emergent behaviour with any top-down strategy, you eliminate the free agency of the component parts in the system. The analogy in the traffic flow example would be for urban planners to ban cars and allow people to only use public transport. Like this the level of governance would be such (the governance of routes and schedules so regulated) that the resulting traffic system may not exhibit properties of a complex system at all. What you've done is eliminate the free agents from the system. So it's difficult to imagine how you can successfully 'govern' emergent behaviour (by any accepted notion of governance) without extinguishing complexity; you probably have to re-define political 'governance' in order for it to be plausible.

In a truly emergent democracy, governance would play a role at the component level, in the rulesets that dictate how the individual citizen can participate, but would have to refrain from dictating, regulating or containing the emergent decision making patterns and structures that arise.

Charles highlighted a single change in the nature of voting that may drive emergent democracy: technology will force a shift from fixed place/time voting to continous voting in the near future. He backs this up with various historical reference points as to how technology has affected politics throughout human history. I'm inclined to agree with Charles, and this shift leads to all kinds of questions around representation and participation. Various other people contributed to an interesting discussion on the night:

Mako from Selectricity demonstrated their online polling system which has come out of research into civic technology at MIT; it supports several voting strategies (including preferential, condorcet and borda) and provides a wide variety of feedback on results. You can try out a quickvote for free or contact them for access to more advanced tools.

Sennse from Wikia provided an insight into arbitration and power dynamics at Wikipedia (which she worked at). Interestingly, Wikipedia takes the time to point out it's not an experiment in democracy or any other political system.

Edward from Involve also talked about their collaboration with local government in citizen engagement.

If you want to explore further try Joi Ito's essay on the subject.

In the interest of full disclosure, I should point out that Joichi was an early investor in Last.fm, whom I work for.

Sami Sänpäkkilä's sateenkaarisuudelma under the pseudonym ES. UK tour next month. Don't miss.

- Stewart Brand, How Buildings Learn (1995)

I'd heard of Stewart Brand's book How Buildings Learn before, but didn't know there was an accompanying TV series (with music by Brian Eno). The TV series is available in full online (episodes 1-6). By focusing on what happens to buildings after they're built, Brand articulates lots of principles that we use in software today. Brand's own notes even refer to it:

Most of the 27 reviews on Amazon treat it as a book about system and software design, which tells me that architects are not as alert as computer people.

It's interesting to see so many insights that tie together the two kindred fields of architecture and software design in one place. The fact is architects still struggle with the seeming permanence of their design decisions - even big memes of recent years like parametric design haven't laid the path for conscious design of adaptive buildings yet. And as Brand points out, it's not neccessarily a technological question, simply a re-definition of the architect's role and attitude.

We, on the other hand, have got it easy, working in a fairly pliable medium of pixels on screens, with tight feedback loops and the opportunity to evolve our design over time both available at a low cost.

- Aldo Van Eyck (dismayed with modernism)

Van Eyck designed and built about 700 playgrounds throughout Amsterdam from the 1950's onwards. Social network designers could learn from Van Eyck's thoughts on and approach to public space - he petitioned for these areas of play and dotted them throughout the city as spots where the 'seeds of community were sewn'. A whole generation of children found themselves playing in the very heart of their city, in all sorts of unlikely locations. The playgrounds were rarely cordoned off from the city around them - they were open, exposed areas that forced kids to come up with rules for play and security. They brought unlikely elements together, valued ambiguity and looseness in function and blurred borderlines. You can find more information on his playgrounds and another article here.

Adolf Loos, Ornament & Crime

This talk is worth a look. Forgive the title (which is a misnomer) and the way it frames REST; equating it to feeds designed to be consumed by polling clients - Newtonian physics to PubSub's quanta (poorly formed analogy). REST and RPC simply are suited to other types of services (I fail to see how RPC over XMPP is useful unless the time to process a request is long and callbacks are required). The talk goes on to use concrete examples that illustrate how hijacking Jabber servers and XMPP for generic push messaging using a pubsub architecture is far more efficient for lots of web services outside of IM that are currently pull-based. They even manage to fudge oAuth in for protected resources. It's the kind of pragmatism that smacks of real life problems solved (I salute).

I'd be very surprised if Last.fm's 'now playing' notifications didn't switch over to pubsub very soon.

Some great slides from a 1975 IBM presentation.

Boriska, a young boy over-seeing the casting of a bell, in Tarkovsky's Andrei Rublyov

Most developers aren't paid to think about what they build. This "divorce of head and hand", as Richard Sennett puts it, is counter-productive. The more pronounced this divorce, the less likely you are to innovate. Why? Because programming is a craft, and innovation happens in the midst of practising this craft. Sennett's latest book, The Craftsman, is a study of craftsmanship through the ages. I'm writing this as a collection of my after-thoughts on the book (which I'd recommend, despite its grammatically inept first edit).

The workshop ethic

Sennett discusses the history of the workshop and explores what he calls the 'material consciousness' of a craftsman who knows his tools and working materials intimately. In a workshop, authority is equivalent to skill. Historically, the most skilled craftsmen (masters) have had authority over journeymen and apprentices in their workshops. We could learn from a closer modelling of development teams as workshops - most developers are told what to build (and in some cases even how to build it) by managers who don't possess a 'material consciousness' relevant to software engineering, which often leads to problems.

On Engineering

Why does our industry separate out the 'lead engineer' from 'engineering management'? Dealing with software engineering as craft would indicate that a hybrid works better - craftsmen are motivated to work for better craftsmen, in order to improve their skills. In the traditional workshop, the master craftsman would interact with clients (read: product managers/executives) and manage the rest of the craftsmen whilst running the workshop.

By this logic the most effective software engineering manager is an experienced, highly-skilled programmer (hello Mr. Gates). By definition they should still be programming ("If you stop coding, you stop learning" - Beck), for the same reason a master craftsman never actually stops practising his craft. Because to stop practising is to stop evolving the material consciousness that makes you a great craftsman in the first place.

The arguments against this approach are obvious - how's a lead engineer also going to manage developers? Surely he can't do two jobs? Let's take a step back. In order to manage great developers effectively you need to gain their trust and respect. Without this you'll struggle to get anywhere. Upon which we come to a programming truism:

In the work-place, the only way to get the full respect of a skilled programmer is to prove yourself at least as skilled as them at programming, and the only valid evidence is actual working code.

I don't think I've ever met a developer for whom this isn't true. Bear in mind that your average software developer, not known for their humility, will often not dispense this kind of respect until you've coded alongside them (or in a visible forum such as that of an open source project) for a while.

Given this, you can't feasibly manage great devs without being one yourself. I mean you can (it happens all over), but it's a fruitless, unproductive and frustrating operation for all involved compared to what happens when you get it right. In this regard, I'm assuming you agree with the notion that programmer productivity is a completely non-linear phenomenon (see Paul Graham). If you address the issues of trust and respect at the core of a programmer's productivity, you open up a lot of potential. This overrides (by an order of magnitude) the fact that a hybrid lead/manager doesn't have as much time as two separate people, because it gives the individual developer a higher degree of agency in their work and a clear reporting line to a developer that a) speaks the same language as them, and b) which they are motivated to work for and with.

Pretty much all successful startups naturally possess someone who fulfills this hybrid role. It's only when companies grow that this 'workshop' ethic is lost.

Another argument against letting a lead actually manage anything is that the lead has lousy social skills. You'll hear a lot about skilled programmers that are completely socially inept. In application development, I don't even buy this notion. To be a great application developer you have to have superb natural language skills as the entire craft is equal parts natural language and logic. Algorithm-driven programming may well be a different kettle of fish, but app devs absolutely are measured by their capacity to communicate in natural language.

Design & engineering: hybrid minds

This kind of logic about craftsmanship leads to some more questions - particularly around the separation of 'front-end engineering' from 'web design'. Why? The syphoning off of these two disciplines is a symptom of this separation of head and hand; one that stifles the feedback loop of discovery and product evolution at the heart of craftsmanship (the 'designer' only has to think about the final product, not actually craft it). This design/engineering split is obviously evident across sectors (take architects and civil engineers), but rarely do you find a practice like software, where results can be near immediate and design/build feedback is quick. If designer and engineer are to be separated then they will need to work as a hybrid mind in order to innovate in any case (e.g. if Cecil Balmond, a lead at ARUP, and architect Rem Koolhaas hadn't developed this kind of relationship over decades, you wouldn't get results like this).

On technique and expression

Sennett explores the craftsman's worldview that technique "is intimately linked to expression". I believe this to be true. There's an obvious difference between a purely technical innovation (such as the evolution of object-oriented design paradigms in the design of Smalltalk and its dev community in the early eighties) and a software product innovation (e.g. Joshua deciding book-marking the web should be a social activity), but I intuit that the two are linked. Both these types of innovation (indeed, all types of innovation) are highly expressive acts, and it's impossible to be expressive without cultivating expertise in technique.

On 'theoretical' practice

This leads me to think about so-called 'theoretical' disciplines such as pure maths and quantum physics that, whilst being highly abstract, are expressive and require technical expertise. Can such theoretical practices display a 'material consciousness'? Physicist Richard Feynman is a good example of how they can. In Feynman's anecdotal autobiography, he talks about his need to come up with clear material examples when discussing any kind of physics problem, no matter how abstract. His grasp of mathematical techniques was rigorously practical (and as a result, unorthodox) as he was always trying to solve 'real world' problems. He was as skilled in his ability to grasp the right problems (it's no coincidence that the word 'grasp' alludes to the hand and the handling of materials) as he was in his ability to solve them. You can't put it better than Feynman when he says "What I cannot create, I do not understand".

The pyschology of the craftsman

Lastly, a note on what I think might be one of the most illuminating insights into the psychology of the craftsman in the history of art, Tarkovsky's epic film Andrei Rublyov, in particular the penultimate vignette, which depicts a young boy that takes on a commission to cast a huge bell for the Russian Prince in the 13th Century. Those 30 minutes of cinema express a lot more than can be summarised here, depicting in detail the process of founding and casting the bell, and the young boy's descent into a craftman's obsession (his life depends on the bell ringing true). I'd simply ask you to watch it. There are other observations on the artisan/craftsman and their relation to religion and art (a foreign concept in the period the film is set in) throughout the film, which is the (mainly fabricated) biopic of a famous Russian icon painter.

PS: Whenever i've used the term 'craftsman' or 'craftsmen' I actually mean men and women, obviously.

Glad to have been a part of this in some way (I handle all the API's). Now all we need is for Apple to approve the app for their store.

Update: It's been approved now so you can download it for free from the store. Toby has the full scoop.

Klaus Schulze has released more records than is probably good for him (40+), but 'Synthasy' on side 2 of 'Dig It' is great. I couldn't find it online, but I managed to find 'Nowhere Now Here' from his porn film soundtrack, Body Love (1976).