Music has always served as a great metaphor for thinking about software development. At XP2010 during one of the most original key notes I’ve ever seen, Bjørn Halterhaug and John Pål Inderberg, professors of musicology and improvisation respectively, from the University of Technology and Science in Trondheim, Norway and both veteran jazz musicians, discussed, through music and words how they approach improvisational jazz, the mechanics for making it work, and the general implications on collaboration among artists. It’s tempting to romanticize improvisational jazz as a truly spontaneous creation but this isn’t exactly how it works in reality. For any improvisational jazz band to be successful its members must exhibit seven key characteristics.

1. Provocative Competence Interrupting Habit Patterns
2. Embracing Errors as a Source of Learning
3. Minimum Structures that allow Maximum Flexibility
4. Distributed Task: Continual Negotiation toward Dynamic Synchronisation
5. Reliance of Retrospective Sense Making as Form
6. Hanging out: Membership in Communities of Practice
7. Alternating between Soloing and Supporting

I propose that these seven characteristics apply directly to how teams who espouse emergent design should approach software architecture.

Music and Design

Experienced musicians have an attenuated ear which enables them to hear music in ways that novices can’t. Drawing on a well-developed playbook of clichés, an expert in the standup bass for example can not only follow the minimally, well-defined structure of a song but also weave pleasing bass licks into the musical ramblings of his band mates, adjusting his actions based on the paths his band mates choose with appropriate responsiveness as the song develops. Part of it is musical maturity, a taste or style developed over years of playing, part of it is also trust both in himself as a musician but also in the rest of the group, that they too will not overreact to mistakes and are also able to follow a musical flow as it develops.

Well-defined but minimalist structure. Contrast this with the music played by a symphony which is also well-defined but has explicit and detailed structure.

When I hear this I immediately think of architecture as seen through the lens of agile software processes which advocate strongly for emergent design. Agile software development processes and agile culture in general strongly encourages teams to become improvisational architects. Create a minimalist, well-defined design of the system and then allow the developers to evolve the system, creating the design as they develop, playing off the code written by their fellow teammates as the system emerges. The romantic view of this is appealing to some: “Let’s go have a code jammin’ session, you dig?” But the results of an emergent design can be just as varied as improvisational jazz bands; some designs are elegant and spectacular successes, some designs are flaming piles of disaster, most turn out somewhere in between – usually good enough for customers who aren’t paying too much and who expected a somewhat unpredictable but functionally adequate outcome.

So if you enjoy the thrill of experiencing once-in-a-lifetime moments of creation, the outcome of which can never be predicted or known ahead of time, then emergent design is probably right up your alley. As demonstrated by the near exclusive emphasis on code and shipping, many agile developers fancy themselves as code jammin’, improvisational architects.

The exact opposite of emergent design, of course, is the easy to hate (and rightly so) enemy of creating things that actually work, Big Design Up Front, where a heavy cost is levied against a project in the beginning without producing any usable code whatsoever. Sticking with the music metaphor, the well-defined, explicit, and detailed structures of a symphony might take a composer months or even years to create. But in exchange for this planning you get a piece of music that is guaranteed to be executed (for all intents and purposes) in a nearly identical way in every performance in which it is played. I am not saying you should use a Big Design Up Front approach, in reality, even Mozart tried out music and made incremental deliveries to his sponsors as he wrote.

While certainly a degree of improvisation happens when we develop software, customers paying for software usually expect more predictability in the outcome. So, while it might be ok to spend 30 bucks for a fun first date with a girl at a jazz club – if only for the experience of it – when I pay 100 bucks a ticket to see the symphony play Mozarts’s Requiem, I would be more than a little disappointed, angry, and confused if the first chair violinist “felt a groove” in the middle of the second movement and began jamming. The greater the cost of a project, the more highly valued predictability in it will be.

Who can be an Improvisational Architect?

Mozart was a true genius, a master of composition. To play Mozart, you need to be well practiced but you don’t need to have his level of genius. On the other hand, most members of a great improvisational jazz group must be experienced musicians – experienced in composition as well as technical playing ability. Even then, a great group will usually avoid disaster but occasionally it will still happen because of the nature of improvisation.

In my experience, teams that plan to allow their designs to emerge do an ok job of providing the minimal structure that is essential for evolving a design. But only a few developers have the necessary experience and background to truly improvise a design. The implication? Most teams are terrible at improvising architecture because they have neither a solid enough understanding of the core fundamentals for software architecture nor the experience with using architectural design to reason about a system.

I believe that most of the seven characteristics of improvisation are deeply ingrained in agile culture, but agile teams often fail to fully embrace all seven characteristics of improvisation when designing a software system. A team without the experience, who doesn’t understand design cliché’s (architectural styles and patterns) will have an unpredictable outcome when allowing the design of a system to emerge as the system is developed. This is something the agile community must work to resolve.

Bonus

Cory Foy was thinking ahead and had the foresight to record video of the performance/keynote.  It comes in 6 parts.

• Part 1 – introduction number
• Part 2 – introduction to improvisational jazz, link to XP, perspectives and group dynamic, a glimpse of how they teach jazz and improvisation
• Part 3 – Second number, application of theory discussed in part 2
• Part 4 – Beginning to introduce the idea of minimalist structures, awesome body percussion number
• Part 5 – communication, the seven characteristics of improvisational jazz, demonstration of “minimal structure for maximum flexibility”
• Part 6 – closing remarks and the final number

Ingvald Skaug was also inspired by this talk and wrote an interesting essay which explains kanban as agile jazz. Check it out for a process perspective on the idea of “minimal structure for maximum flexibility.” If you believe Conway’s Law, then it isn’t coincidence that we would apply the characteristics of improvisation in jazz to both process and design.

The Megalomaniac Architect: “You will implement the system as I have designed it because I am the most important person on this team, the project will fail without me, and I am the only one smart enough to know everything that needs to be known!”

The Benevolent Dictator: “Implement the system precisely as I have designed it and I will make your work easier.” [via Dennis Mancl]

The “Everything’s a Nail!” Architect: All problems can be solved by a single tool, idea, or way of thinking. He’s got a hammer…and everything he sees is a nail.

The Diva: “I’m the architect of this system! Are you questioning my ability to design? If you’re not going to appreciate my talents, maybe I’ll just go to another team.”

The Magician: Excellent at high level abstraction, vague on how the design actually achieves architectural drivers.

The Over-Accommodating Architect: “You want that change? No problem.” There are no trade-offs to consider when making architectural decisions.

The Chess Master: To achieve victory, it is simply a matter of being able to see enough moves ahead. Every element in the architecture can be strategically placed in a perfect position to checkmate your opponent: requirements changes.

The PowerPoint Architect: “Architecture is just pretty pictures!”

The Ninja: Infiltrates a project from the outside, crafts an amazing design, then disappears into the shadows before anyone realizes he’s gone. Ninjas do not suffer fools and their designs are technically correct, thorough, and beyond your ability to comprehend.

The Navigator: Creates a map (with legend) and uses it to plot a course through implementation, testing, and deployment.

The Movie Producer: Leads the team indirectly by providing technical design support. [via George Fairbanks]

The Coach: Teaches the team about architectural practices and concepts as well as the design.

The Puppeteer: Able to manipulate multiple levels of abstraction simultaneously to design the architecture, the Puppeteer effortlessly manages the various threads influencing the design (commonly known as architectural drivers).

The Seasoned Veteran: He’s tried a lot of things and is pretty good at all of them including programming, design, processes, and management. Thanks to his experience, having walked 10,000 miles, he understands the role of architecture across the lifecycle, has seen many different situations, and is an excellent technical practitioner. [via Bill Opdyke]

The Long-Bearded Wise Man: A little philosophical but always willing to share an enlightened thought that will help resolve whatever concerns are ailing the project, though somewhat indirectly.

The Team Captain: A bona fide member of the team, playing on the field with everyone else, leading the team from the field.

The Design Evangelist: Excited about the architectural drivers and architecture to the point of near fanaticism. His enthusiasm for architecture and the system’s design is infectious and helps maintain conceptual integrity of the system.

The Student: Takes time to learn about the problem domain and the customer’s needs to the greatest extent possible so that he understands and solves the right problems. [via Gail Harris]

What other archetypes have you worked with? Which archetypes do you think might combine to make the perfect software architect?