Most Fridays we hold talks known as 'osmotics' at Last.fm. Their purpose is to share understanding about the various things that are going on in different departments; an open session which anyone can use to present recent work. It's a response to our growth, as it's now impossible to know what's happening throughout the company at any given point in time.

A few weeks back I presented some of the work we've been doing on interaction path analysis. As web developers we're completely spoilt for usage information on our apps, and interaction histories are sitting in our apache logs the entire time. We've been using tools to mine these and better understand real interaction flows on our site, in order to improve our understanding of common use cases.

You'll often find that your users don't use your software exactly as you designed for. That's natural, and as web developers we need to be equipped to observe these flows and adapt/optimize our software accordingly. As Stewart Brand observed of buildings, so too good software should learn from usage patterns rather than working against them. A good app should flow.

We use a tool suite from Omniture - the pricing may well be out of reach of most startups, but anyone can mine apache logs to get similar information. Bear in mind the presentation assumes no technical knowledge and was prepared in about 20 minutes - it's more of a lightning talk than anything. I've blacked out some sensitive bits of information.

→ Interaction Path Analysis - slides with annotations (1.1MB)

If a building is allowed to fail small, early and often, and be corrected, the building as a whole can succeed

- Stewart Brand, How Buildings Learn (1995)

I'd heard of Stewart Brand's book How Buildings Learn before, but didn't know there was an accompanying TV series (with music by Brian Eno). The TV series is available in full online (episodes 1-6). By focusing on what happens to buildings after they're built, Brand articulates lots of principles that we use in software today. Brand's own notes even refer to it:

Most of the 27 reviews on Amazon treat it as a book about system and software design, which tells me that architects are not as alert as computer people.

It's interesting to see so many insights that tie together the two kindred fields of architecture and software design in one place. The fact is architects still struggle with the seeming permanence of their design decisions - even big memes of recent years like parametric design haven't laid the path for conscious design of adaptive buildings yet. And as Brand points out, it's not neccessarily a technological question, simply a re-definition of the architect's role and attitude.

We, on the other hand, have got it easy, working in a fairly pliable medium of pixels on screens, with tight feedback loops and the opportunity to evolve our design over time both available at a low cost.

This talk is worth a look. Forgive the title (which is a misnomer) and the way it frames REST; equating it to feeds designed to be consumed by polling clients - Newtonian physics to PubSub's quanta (poorly formed analogy). REST and RPC simply are suited to other types of services (I fail to see how RPC over XMPP is useful unless the time to process a request is long and callbacks are required). The talk goes on to use concrete examples that illustrate how hijacking Jabber servers and XMPP for generic push messaging using a pubsub architecture is far more efficient for lots of web services outside of IM that are currently pull-based. They even manage to fudge oAuth in for protected resources. It's the kind of pragmatism that smacks of real life problems solved (I salute).

I'd be very surprised if Last.fm's 'now playing' notifications didn't switch over to pubsub very soon.

Boriska, a young boy over-seeing the casting of a bell, in Tarkovsky's Andrei Rublyov

Most developers aren't paid to think about what they build. This "divorce of head and hand", as Richard Sennett puts it, is counter-productive. The more pronounced this divorce, the less likely you are to innovate. Why? Because programming is a craft, and innovation happens in the midst of practising this craft. Sennett's latest book, The Craftsman, is a study of craftsmanship through the ages. I'm writing this as a collection of my after-thoughts on the book (which I'd recommend, despite its grammatically inept first edit).

The workshop ethic

Sennett discusses the history of the workshop and explores what he calls the 'material consciousness' of a craftsman who knows his tools and working materials intimately. In a workshop, authority is equivalent to skill. Historically, the most skilled craftsmen (masters) have had authority over journeymen and apprentices in their workshops. We could learn from a closer modelling of development teams as workshops - most developers are told what to build (and in some cases even how to build it) by managers who don't possess a 'material consciousness' relevant to software engineering, which often leads to problems.

On Engineering

Why does our industry separate out the 'lead engineer' from 'engineering management'? Dealing with software engineering as craft would indicate that a hybrid works better - craftsmen are motivated to work for better craftsmen, in order to improve their skills. In the traditional workshop, the master craftsman would interact with clients (read: product managers/executives) and manage the rest of the craftsmen whilst running the workshop.

By this logic the most effective software engineering manager is an experienced, highly-skilled programmer (hello Mr. Gates). By definition they should still be programming ("If you stop coding, you stop learning" - Beck), for the same reason a master craftsman never actually stops practising his craft. Because to stop practising is to stop evolving the material consciousness that makes you a great craftsman in the first place.

The arguments against this approach are obvious - how's a lead engineer also going to manage developers? Surely he can't do two jobs? Let's take a step back. In order to manage great developers effectively you need to gain their trust and respect. Without this you'll struggle to get anywhere. Upon which we come to a programming truism:

In the work-place, the only way to get the full respect of a skilled programmer is to prove yourself at least as skilled as them at programming, and the only valid evidence is actual working code.

I don't think I've ever met a developer for whom this isn't true. Bear in mind that your average software developer, not known for their humility, will often not dispense this kind of respect until you've coded alongside them (or in a visible forum such as that of an open source project) for a while.

Given this, you can't feasibly manage great devs without being one yourself. I mean you can (it happens all over), but it's a fruitless, unproductive and frustrating operation for all involved compared to what happens when you get it right. In this regard, I'm assuming you agree with the notion that programmer productivity is a completely non-linear phenomenon (see Paul Graham). If you address the issues of trust and respect at the core of a programmer's productivity, you open up a lot of potential. This overrides (by an order of magnitude) the fact that a hybrid lead/manager doesn't have as much time as two separate people, because it gives the individual developer a higher degree of agency in their work and a clear reporting line to a developer that a) speaks the same language as them, and b) which they are motivated to work for and with.

Pretty much all successful startups naturally possess someone who fulfills this hybrid role. It's only when companies grow that this 'workshop' ethic is lost.

Another argument against letting a lead actually manage anything is that the lead has lousy social skills. You'll hear a lot about skilled programmers that are completely socially inept. In application development, I don't even buy this notion. To be a great application developer you have to have superb natural language skills as the entire craft is equal parts natural language and logic. Algorithm-driven programming may well be a different kettle of fish, but app devs absolutely are measured by their capacity to communicate in natural language.

Design & engineering: hybrid minds

This kind of logic about craftsmanship leads to some more questions - particularly around the separation of 'front-end engineering' from 'web design'. Why? The syphoning off of these two disciplines is a symptom of this separation of head and hand; one that stifles the feedback loop of discovery and product evolution at the heart of craftsmanship (the 'designer' only has to think about the final product, not actually craft it). This design/engineering split is obviously evident across sectors (take architects and civil engineers), but rarely do you find a practice like software, where results can be near immediate and design/build feedback is quick. If designer and engineer are to be separated then they will need to work as a hybrid mind in order to innovate in any case (e.g. if Cecil Balmond, a lead at ARUP, and architect Rem Koolhaas hadn't developed this kind of relationship over decades, you wouldn't get results like this).

On technique and expression

Sennett explores the craftsman's worldview that technique "is intimately linked to expression". I believe this to be true. There's an obvious difference between a purely technical innovation (such as the evolution of object-oriented design paradigms in the design of Smalltalk and its dev community in the early eighties) and a software product innovation (e.g. Joshua deciding book-marking the web should be a social activity), but I intuit that the two are linked. Both these types of innovation (indeed, all types of innovation) are highly expressive acts, and it's impossible to be expressive without cultivating expertise in technique.

On 'theoretical' practice

This leads me to think about so-called 'theoretical' disciplines such as pure maths and quantum physics that, whilst being highly abstract, are expressive and require technical expertise. Can such theoretical practices display a 'material consciousness'? Physicist Richard Feynman is a good example of how they can. In Feynman's anecdotal autobiography, he talks about his need to come up with clear material examples when discussing any kind of physics problem, no matter how abstract. His grasp of mathematical techniques was rigorously practical (and as a result, unorthodox) as he was always trying to solve 'real world' problems. He was as skilled in his ability to grasp the right problems (it's no coincidence that the word 'grasp' alludes to the hand and the handling of materials) as he was in his ability to solve them. You can't put it better than Feynman when he says "What I cannot create, I do not understand".

The pyschology of the craftsman

Lastly, a note on what I think might be one of the most illuminating insights into the psychology of the craftsman in the history of art, Tarkovsky's epic film Andrei Rublyov, in particular the penultimate vignette, which depicts a young boy that takes on a commission to cast a huge bell for the Russian Prince in the 13th Century. Those 30 minutes of cinema express a lot more than can be summarised here, depicting in detail the process of founding and casting the bell, and the young boy's descent into a craftman's obsession (his life depends on the bell ringing true). I'd simply ask you to watch it. There are other observations on the artisan/craftsman and their relation to religion and art (a foreign concept in the period the film is set in) throughout the film, which is the (mainly fabricated) biopic of a famous Russian icon painter.

PS: Whenever i've used the term 'craftsman' or 'craftsmen' I actually mean men and women, obviously.

Glad to have been a part of this in some way (I handle all the API's). Now all we need is for Apple to approve the app for their store.

Update: It's been approved now so you can download it for free from the store. Toby has the full scoop.

AS#Enumerable is a port of the Ruby Enumerable API for Actionscript 2.0. It provides the following enumerable methods for use in your flash applications:

• all,
• any,
• collect,
• detect,
• each,
• find,
• findAll,
• include,
• inject,
• map,
• max,
• member,
• min,
• pluck,
• reject,
• select,
• size,
• sortBy

It's provided as a subclass of the native Array object, but if you want all native Arrays to inherit the enumerable interface (a la Prototype) just add:

Array.prototype = new Enumerable();

to your Flash application. Some basic sample uses of Enumerable:

var enum = [];
enum.push(0.2);
enum.push(0.3);
enum.each(function(item,index) {
  trace(item + ':' + index);
});

(Using each for basic iteration).

var enum = [];
enum.push({value:'foo'});
enum.push({value:'bar'});
enum.push({value:'camp'});		
var item = enum.find(function(item,index) {
  return item.value == 'bar';
});

(Using find to fetch an object from a collection).

var enum = [];
enum.push('foo');
enum.push('bar');
enum.push('camp');	
trace(Math.floor(enum.inject(0,function(strlen,string) {
	return strlen + string.length;
})/enum.size()));

(Using inject and size to fetch the average string length in a collection of strings).

Go ahead and grab the source code, which will find its permanent home at /code, should you want to check back.

See the prototype enumerable API doc for further documentation, as the usage examples are identical in Actionscript 2.0. Thanks to Sam Stephenson of Prototype for the JS port.

[7:02pm] alex: Haskell users are all pretty nuts.
[7:02pm] johan: when the revolution comes they're first up against the wall
[7:02pm] alex: johan: Actually, they won't.
[7:02pm] alex: Haskell does not guarantee execution order.

it's not enough to just instruct a machine to do something that you need also to think about what will people read in what I write here, so it's programming more like a writer would write, than just thinking of a set of instructions for a machine.

- Kent Beck, OOPSLA interview

Kent has been the biggest single influence on my programming.

Sennett describes craftsmanship as "an enduring, basic human impulse, the desire to do a job well for its own sake". It's a great topic, which has come to my mind a lot with respect to software design. I have always viewed programming as a craft, amongst other things. There are reviews from The Times & The Guardian. I've ordered my copy. Grab yours.

This is the year in which hAudio ushers in the era of distributed music publishing and playback on the web. As enthusiastic as I am about OpenId, oAuth and DataPortability (the emerging protocols of the distributed social web), it's microformats that are going to be the building blocks of this shift in the short term. Microformats can be seen as "the nanotech of the semantic web" (Jeremy Keith) and the metaphor for usage is one of proteins and surface-binding. My hope is that this year microformats will have a big impact on media publishing. It's more than a hope because I will be dedicating some of my time to help make it happen.

Download the free AJAX Control Toolkit which includes over 30 AJAX controls including rounded corners...

- Microsoft Visual Web Developer Express website.

Uncanny similarity in conduct between Microsoft, Scientology and Neo-con Republicans here, all going ahead and creating their own new realities. Conclusion: positions of such power allow organisations to disregard the actual state of the universe. By this logic, Microsoft is purely an ideology machine and no longer a technology company.

- 'Silverlight Architecture' schematic, Microsoft

This illuminating 'architecture' drawing from Microsoft manages to pack six (awful) logos and two pieces of packaging onto one stupid schematic. If you're in Microsoft's marketing department, can you please do us all a favour and take extended leave? Please? And leave Visio alone. It's not for you.

How others compared you recently: "Who is hotter", you won 0 and lost 1 time.

- 'Compare People', Facebook App Email Notification

Thank you Facebook application platform, for all that you have done for me in 2007.

Scant time for any kind of thought pieces on here this year. I offer this post as a snapshot of my thoughts on sustainability - a running theme for me of late.

Problem Statement/ Few organisations have managed to create sustainable work environments. Modern industry has largely failed to create sustainable processes.

Thesis/ Corporate models need to be re-invented because people do their best work in sustained group environments. This is true in all co-operative forms of expression (e.g. in most forms of design and engineering). Sustainable industrial processes are key to ecological sustainability and the health of the biosphere.

Myopia/ The 20th Century as a catalogue of short-sightedness. Industry rewards myopia as 'success'.

Percolation/ Sustainability is a timeless design challenge. Its largely unattended problems penetrate all of social life, from the individual through to the corporation and the body politic.

Capital/ Long-term profitability comes from sustainable development. Historically, only 1 in 10 public companies last longer than four decades. [3]

Current/ The notion that market success comes from organisations that hold sustainability as a core principle at all levels, from product design to organisational structure to development processes, is gaining currency.

Child/ The modern corporation, with a history of no more than 150 years or so, is a child in terms of the history of production.

Delight/ Delight as continual learning. No sustainability without some measure of delight.

Diversity/ Diversity as a key ingredient of sustainability. Preservation of diversity in the biosphere as a moral, social, and political goal as well as a commercial one. Preservation of intellectual diversity in a workplace as a purely commercial goal.

Settler/ A sustainable financial, social and intellectual living set-up over a nomadic and erratic one.

Fractal/ Sustainability has a fractal relationship to the Nietzschean notion of 'eternal return' (the ability to live your life as you would wish to live it out an infinite number of times marks the ultimate affirmation of life).

Performance Metric/ Team health. Relative happiness of those expending the effort.

Perspective/ Contrast the modern Perspex industry - dependent on petrochemicals and with a product impossible to recycle without releasing pollutants - to that of Murano Glass - a non-toxic, largely recyclable product that has been produced since 1291.

Growth/ Through the lens of sustainability, Head count in an organisation is not a valid marker for success.

HR/ Modelling people on resources is bad design. People are too complex to be modelled on any contemporary metaphor.

Fossilization/ Vertical power structures fossilize products.

Happiness/ Sustainability as a key ingredient of happiness. We want to be involved in activities that are fundamentally sustainable, because we can then afford to engage in them fully.

Cultivation/ People embrace sustainable forms of product development because everything it implies on a human level - a serious cultivation of teams, retention of people, corporate independence, building products people have a personal investment in - is desirable to the individual. This observation is in sharp contrast to the nomadic form of professional working that has become popular in the western world.

Rhythm/ Keeping motivated as a team over a period of not just months, but years. Preserving momentum.

Consultants/ Companies are unable to cultivate specialized knowledge in all their interest areas since they are not modelled on self-organizing, adaptive organisms. Cue the age of consultants.

Abundance/ What industry calls 'innovation' is creative abundance. How to sustain creative abundance whilst meeting short-term needs (i.e. not going bust).

Democracy/ Democracy itself is a sustainable (political) process. Its principles can therefore be used as building blocks to other sustainable (commercial) processes.

Specialization/ Mechanisms by which specialized knowledge can be spread across organisations without people needing to leave: meet-ups & co-working locations. A consultant eco-system is ultimately problematic for product, departmental & organisational development.

Catalysis/ A horizontal structure hybridized with vertical paths of responsibility elected and erected on-demand. Encourage natural catalysis, intervention.

Co-operation/ Distributed ownership of an organisation feeds accepted responsibility.

Cycle/ Cycles are natural rhythms. Intelligence is a function of feedback bandwidth and quality. Sustainable development as an iterative cycle.

Command/ Companies continue to be based on military-inspired structural models. Tyranny over democracy. Command/divide (and conquer) is still the primary metaphor for conduct. It's an uncompetitive model for a free market.

Atomization of intellect/ An explosion in freelance working has been brought about by the problem statement.

Adaptivity/ How to create a team and process with the capacity to consistently develop not just one great product, but any great product in a given field or even multiple fields.

Self-Organization/ The people that compose a company should be able to shape their organisation's structure, with built-in review processes.

Retention/ Sustainable systems absorb nutrients and release others to nourish wider cycles that are indirectly beneficial to both the system and other co-dependant systems.

Autonomy/ Building a framework of services and tools that allow any new project in an organisation to get off the ground quickly.

Flourish/ Success as an execution of the model abiding by the principles of sustenance. A company as an adaptive organism that doesn't just survive but flourishes.

Accepted Responsibility/ "No single action takes the life out of people more than being told what to do." [4]

Nutrients

[1] Cradle to Cradle: Remaking the Way We Make Things, William McDonough & Michael Braungart, North Point Press

[2] The Ecology of Commerce, Paul Hawken, Collins

[3] Leading By Omission, Ricardo Semler, MIT School of Engineering Talks

[4] Extreme Programming Explained, Second Edition (accepted responsibility), Kent Beck with Cynthia Andres, Addison-Wesley

[5] Scrum Et Al (iterative development), Ken Schwaber (Google Talks)

[6] Small Giants: Companies That Choose to Be Great Instead of Big, Bo Burlingham, Portfolio Hardcover

[7] Birth of the Chaordic Age, Dee W. Hock, Berrett-Koehler Publishers

[8] A Thousand Years of Non-Linear History, Manuel De Landa, Zone Books

[9] Cybernetics: or the Control and Communication in the Animal and the Machine, Norbert Wiener, The MIT Press

If you stop coding, you stop learning.

Kent Beck, Smalltalk Best Practice Patterns

The 'Bundles' view in Zim gives you a simple way of grouping outlines.

Sony research group demo an 'augmented surface'

Spurred by Miranda's reference to Malcolm Gladwell's article, The Social Life of Paper and other articles on the subject ( MIT Press' The Myth of the Paperless Office, Passion for Paper and Bill Gates' paperless working methodology), i thought i'd throw in my $.02.

I really can't see a case for paper in the workplace outside of enhancing co-located group sessions/workshops (i've described some below - Gladwell points one out as well). I'm trying, but i don't see it. At least for software development, though i believe this view is defensible for a range of industries.

One of paper's great advantages lies in its physicality. As a result of its portability it can potentially radiate information to multiple people in different locations without any technology requirements other than a pen/pencil. It's spatially flexible and provides a low barrier to authorship.

When you access paper-based information, subtle authorship traits are clear (handwriting and drawing styles communicate myriad things to us which digital typography and rigid outlines do not). Paper media retains these traits so well because the paper/pen combination provides one of the greatest interfaces mankind has ever come up with. It allows for much self-expression. From an accessibility perspective, the printed press gives you high resolution fonts you can take to bed with you. No eye-strain included.

So it's not all bad for paper. But the paper-less office isn't a myth. I don't use any paper at work. The only pieces of paper i have touched involve HR processes such as holiday forms (arcane admin rituals).

I use OmniGraffle or a whiteboard for quick visualization of ideas. Whiteboards beat paper hands down for radiating information to groups in the workplace (see Cockburn). Textual collaborative, dynamic information resides in the web browser - accessible by all, manipulable at all times. Wikis, chat-rooms and project management tools sit there. The browser is our core communication medium.

For notes, I have OmniOutliner which i also use to deconstruct large problems into identifiable tasks. Technical designs can be drawn in applications pretty quickly. These designs can then be uploaded for others to comment, in their own time. I can re-organise and re-prioritise my tasks at a keystroke. I can take notes on a meeting with others, during a presentation (backchannel). Put all this together and paper is starting to look a little ragged.

In the processes of archiving and organising (search/sort), paper also makes a weak case for itself. Digital alternatives exist for tracking ideas, dialogues, designs, tasks and most other everyday information. The notion that paper is a more secure information storage mechanism is also often mistaken and i can only assume comes from a misunderstaning of modern technology (stories of giant electromagnetic guns and hackers making people nervous).

In contrast examine the issues that arise from using paper instead of digital tools, and accumulate these over time (no versioned history of events, unsearchable, restricted ability to re-organise, closed access). It's a cludge. Moreover, it's potentially harmful; paper is, after all, the medium of bureaucracy. Solutions to everyday design problems require agile, accessible information. You need to be able to mix and match approaches, sculpt them in a group environment and change them on the fly. Your chosen media should reflect this.

The only place i see for paper in software dev is in CRC cards and planning sessions (blitz planning, sprint planning or xp planning games) - specialized techniques peculiar to the field (though kaizen-blitz sessions are widespread in the manufacturing industry). Broadly speaking, these are group processes taking place in one location, in which members pursue a single goal of identifying and organising many discreet chunks of information in real-time. Paper's spatial properties come to the fore here. Note that the paper is often thrown away after the session is complete and results captured.

All of these group operations could be carried out on a screen, but screens are interfaces to personal computers, which are designed with the individual in mind. Personal Computers are not for groups (as the name suggests), and a network of such computers does not constitute a true group interface. Turkle has alluded to the solipsistic design of the modern computer in the past. I think as better groupware interfaces are commoditized, such as augmented surfaces (sony research video), the use of paper even in these group processes will become optional and ultimately deprecated.

From the ibm archives photo gallery.

My father worked with the IBM 1620, the Control Data Corporation's CDC 6600 & the Cray 1 supercomputer, which looks like a monolith.

These last two were designed by Seymour Cray.

The 1620 took punchcards and required the entire operating system to be fed in every time it was switched on, since it lacked anything but 60,000 digits of volatile storage. It got the nickname CADET (Can't Add Doesn't Even Try) through its use of lookup tables instead of adders to perform arithmetic.

I'm joining social music service last.fm as a full-time developer in a couple of weeks over at their new office in old st (it still looks like a mess - story + pics). As those who know me will confirm, i have views on social software and how culture can suck less under new classification, distribution and publishing models, so i'm really looking forward to getting stuck into moderation strategies, web services, folksonomy and data mining. Particularly psyched to find individuals i respect (e.g. Joi Ito) among the investors.

It should be a tremendous year for last.fm. Hopefully i can play my part in that.

Japanese for continuous and incremental improvement, a business philosophy about eliminating waste in working practices.

Kaizen is a daily activity whose purpose goes beyond improvement. It is also a process that when done correctly humanizes the workplace, eliminates hard work (both mental and physical), teaches people how to do rapid experiments using the scientific method, and how to learn to see and eliminate waste in business processes.

- Wikipedia

Japanese for continuous and incremental improvement, a business philosophy about eliminating waste in working practices.

Kaizen is a daily activity whose purpose goes beyond improvement. It is also a process that when done correctly humanizes the workplace, eliminates hard work (both mental and physical), teaches people how to do rapid experiments using the scientific method, and how to learn to see and eliminate waste in business processes.

- Wikipedia

Each week the four of us sit down and throw patterns in each others' faces. Last week i threw adaptor. Things got messy - the pdf presentation the only remaining evidence. UML is not evil btw. Big revelation. Growing pains.