When I was a kid my brother and I used to play a game called Make Believe. My favorite variant of the game was simple. Together we would build some kind of fortress and then one person gets the fort and the other person tries to invade the fort. In theory, the game ends when the fort has been overtaken by the invader. What made the game fun was that as the invasion began, the rules of the game always changed. The first thing to go was any notion of death. If one of us was “killed” in battle then near instantaneous respawning was created. Shortly after that we skipped respawing and simply became invincible. Soon the fort became invisible which means the invader just has to run around trying to find it. Sometimes someone gained super strength or the ability to force other people to move in slow motion. We almost always created super weapons (such as a hand held Death Star) which for some reason could always be defended against. Nearly every game ended in tragedy, someone crying or upset: “That’s not fair! You can’t do that! I’m invisible! You can’t do that!”

Kid’s stuff right?

A lot of software projects with teams made up of working adults still play this game. The scenario goes something like this. A team is put together to build some software. Neither the clients nor the team talk about the objectives of the project other than building “some software”. After a few months, something goes wrong or someone doesn’t like what’s happening so someone changes the rules. Before too long, one side or the other is upset that they can’t win, somebody throws a fit, and goes home. Instead of summoning invisible armor, software projects change the rules by cutting features, adding more requirements, moving due dates, wasting resources, and things like that.

We make believe that we’re software engineers.

While Make Believe was a fun game as a kid, changing the rules when there’s real money on the line isn’t as fun. My brother and I ran into problems as kids because we got the objectives of the game wrong. Actually, there were no common objects, which is why we could change the rules so easily. The same thing happens on a software project when the objectives aren’t well known.

Defining and committing to a clear picture of success establishes the common ground rules for a project by making the basic project goals explicit. The technique is known as Threshold of Success.

Defining What Success Looks Like

The Threshold of Success for a project is the minimum set of conditions that must be met for the project to be considered successful. If the team fails to meet even one of the conditions then the project is a failure. A good Threshold of Success is made up of about 3-4 SMART goals (no more than a few bullets on a single PowerPoint slide). SMART is a mnemonic which stands for Short/Specific, Measurable, Achievable, Relevant, and Time bound.

Some other pointers for defining a Threshold of Success:

• The Threshold of Success should be built as a team. Since this is the measure by which you will define success or failure, everyone on the team must buy into it. If you can include your client that’s even better.
• Threshold of Success goals should be challenging, but it’s important that they are achievable. If the goals are too easy, victory will be meaningless, too difficult, elusive.
• Once the Threshold is established, don’t change it! The only reason to modify the Threshold of Success is if the project has changed so drastically that the Threshold no longer makes sense (for example if someone leaves the project).
• Revisit the Threshold of Success regularly (a good time is when planning iterations) so everyone remembers what success looks like. Put it on your team wiki so that it’s readily accessible.
• Be sure that the goals in your Threshold are SMART! The point of defining a Threshold of Success is to take away the wiggle room for defining what it means to succeed or fail. The goals you define should make this black and white. The more specific the goal is the better.

Building a Threshold of Success

The easiest way to create a Threshold of Success is to first create a minimum picture of failure, then convert failure into success. Here’s an example:

Failure for my current project might look something like this.

• Essential features are not ready by the end of the second quarter.
• Team members are dissatisfied or bored with their jobs.
• Newly hired team members don’t feel like they’re part of the team by March 31.
• There isn’t enough money to continue development after this fiscal year and we have to fire people.

Now that I know what failure looks like, seeing success is easy. I don’t want any of these things to happen. The threshold of success for my current project might look something like this.

• By the end of the second quarter, all “Must Have” features are implemented and pass acceptance tests with no known critical defects.
• All team members give average score of 5 or better on a job satisfaction survey taken quarterly.
• By March 31, the team has successfully executed at least three team building activities with all team members present.
• Funds of at least $1 million are secured by December 31 to allow for future development without a reduction in team size.

Notice that only 1/4 of the success goals in this example are related to software functionality. While goals might come from anywhere, teams traditionally focus on goals related to people and relationships, process, resources (such as budget or schedule), and product (software functionality and quality).

As this technique originated with the Software Engineering Institute (pdf), nearly every studio team in the Carnegie Mellon Master of Software Engineering program creates a Threshold of Success for their projects. The MSE Studio Archive has extensive examples of both good and bad pictures of success that teams have created. The Square Root Team’s threshold (my team) is a good place to start, but there are plenty of other examples.

There might be many goals for a project. In the Team Software Process you actually identify at least three different kinds! But there is only one threshold of success for a project. Knowing what success looks like gives you a better chance of actually achieving it.  Without it, you’re just pretending that you know what’s going on.

Late in the afternoon on December 25, during one of the loudest, howling winter storms I’ve ever experienced we lost power. Normally this wouldn’t be a big deal except I was in a vacation house with 20 other people, basically my wife’s entire extended family.

After the power went out, the heat did not fire up. The tree went dark. So did the TV, DVD player, and Wii.

Using a pair of LED headlamps my wife and I received for Christmas the 20 of us took turns rolling and stuffing homemade ravioli dough for dinner. Christmas ravioli making is extremely serious business and I had finally been promoted to “unmonitored ravioli stuffer” this year. Luckily we had manual pasta rollers to flatten the dough. There’s always talk of “upgrading” from the hand cranked system but this year, tradition trumped technology.

Once ravioli are stuffed, they have to dry for a few hours before cooking. In the years past this was the time to play with new games or watch a new movie. Of course, without electricity most of our new toys were rendered useless.

Since the heat was off and the vacation house was gigantic, the fireplace in the living room was our only option for warmth. Lighting the room was the handful of candles we had, originally intended for dining ambiance, and the low glow from the fire.

To pass the time, we sang Christmas carols in the near dark. It was all sort of surreal, a setting I am certain we would not have created on our own had we not lost electricity, had we not been so completely pushed out of our comfort zone. Even more fantastic was that after over 20 years of putting it on my wish list, someone actually gave me a pair of night vision goggles for Christmas. Surprisingly, this “toy” is the real deal. I was able to wander around a completely pitch dark house with no problems. The night vision, along with the head lamps (which single handedly saved dinner), were by far the best gifts of the day. The electricity turned on around 1:00am that morning and by the next morning, everything went back to the way it was.

Sitting in the dark on Christmas night got me thinking. This was an experience that I never would have chosen, a situation I would never have intentionally embraced, but it turned out to be pretty fun. How many opportunities have I missed because my default attitude was to stay inside my comfort zone? While happy accidents are great, can’t I do more to create opportunities rather than relying on happenstance? Given the time of year it only seems appropriate to ask these sorts of questions.

I’m a technologist, a scientist, and an engineer at heart. I love playing with gadgets, tinkering with software, and working on interesting and challenging problems. Sitting in the dark with no power, my normal pursuits removed, it was easy to remember that people are important too. Sure, when building software everyone always talks about how people are important, but when building software we call people “users” essentially reducing their humanity. After all, there are only two industries where the customers are called “users” – software is one of them.  Is it really appropriate to use the same term to describe software clients and drug addicts?

My lesson from the night: the technology and gadgets and programming and processes and everything else are awesome, but they are meaningless if they fail to create a genuine relationship with people.  I do believe that relationships matter and that people are important. I’d like to do a better job of thinking about people first this year. In the rush of excitement surrounding every new technological achievement, it’s sometimes easy to forget that helping people is why I build software.  It’s too bad it took a harsh winter snow storm to remind me of this.

Almost a full year ago, I concluded that formal methods simply aren’t worth the effort:

For almost every project in the world, I think formal methods should be generally avoided. Given the option of spending money and time on mathematicians or extremely smart coders I would chose the latter. With smart coders, code inspection is a fun and effective defect filtering option. And let’s face it. Why would you have your amazing coders do something other than write amazing code?

But what if formal methods didn’t have to be so…formal?

While traditional formal methods such as Z might be a little difficult to pick up if your predicate logic is rusty, applying formalisms that are close to the code doesn’t seem that outrageous in perspective. Design-by-contract language extensions such as Spec# for C# or Plural for Java have been stealthily making their way into IDEs for years now without anyone really knowing the wiser, without anyone really thinking about these tools as being formal methods. State machines in UML and even some uses of domain specific languages might be considered formal methods when these tools are used for specification and analysis purposes.

Creating formal methods that are close to the code is one of the best ideas to come out of the formal methods arena in the past 20 years. Just like how adding cherry flavor makes it easier for kids to take icky tasting medicine, putting formal methods in the code makes it more likely that I, as a programmer, will actually use them. Because no matter how good for me something is, if it’s difficult to use or generally unpleasant, it won’t get used.

This is why domesticating formal methods (pdf) is extremely important for formal method adoption.

Wild formal methods can be difficult to work with. No matter what anyone tells you it does take time to get the hang of Z. Formal methods such as Z have limited use when talking with customers and, though a formal specification is an excellent tool for gaining better understanding of functional requirements, the payoff for creating the specification isn’t seen for weeks or months making it difficult to justify their use in Agile development environments.

But is turning what was once a ferocious and unpredictable wolf into an obedient and trusted canine companion safe? In selecting the features for our domesticated formal methods, did we accidentally breed out the most important benefits?

Anthony Hall outlined seven myths of formal methods (pdf), ideas about formal methods erroneously taken as truth.

Myth 1: Formal methods can guarantee that software is prefect
Myth 2: Formal methods are all about program proving.
Myth 3: Formal methods are only useful for safety-critical systems.
Myth 4: Formal methods require highly trained mathematicians.
Myth 5: Formal methods increase the cost of development.
Myth 6: Formal methods are unacceptable to users.
Myth 7: Format methods are not used on real, large-scales software.

Any formal method worth using should not uphold any of Hall’s seven myths, but domesticated formal methods have the additional burden of usability. Just like we expect certain behavior from our four-legged friends, I don’t think it’s too much to ask that a domesticated formal method be friendly and obedient. Any method which bites the master’s hand obviously can’t be trusted. Crashing, making it easy to make mistakes, poor affordances, difficult to read output, impossible to maintain specifications – these are all reasons for not trusting a domesticated formal method.

I think informal formal methods are great. When done well, it doesn’t even feel like I’m using a formal method and I get many of the same benefits a wild formal method would give me – clarity, understanding, and maybe even a little automated verification depending on the language and method. Domain specific languages and state charts are great for working with end users and clients. It’s even plausible to skip other testing or verification measures such as unit testing or inspection.

And the best part is that since many domesticated formal methods are close to the code, I’ve already got the necessary training on how to use the methods because I already know how to program.  Domesticated formal methods are a win-win for programmers who want to do more engineering and another practical tool for your silver toolbox.

It’s easy to convince a group of people to follow you when you’ve got a great idea that you’re passionate about. Passion is like a highly communicable virus, easily spreading from one host to another. Something funny happens, though once enough people are on board with an idea: new ideas become less infectious over time as if the group has built up antibodies against risk. It’s always unfortunate to see this happen since most organizations are initially brought together by an idea that was so risky and so contagious that everyone wanted to be a part of it. Eventually, if an organization is not careful, it becomes a place where Big Ideas go to die, a sort of idea graveyard.

Kathy Sierra explains it best:

Any time a new idea is brought to the table, especially a Big Idea, the group acts like white blood cells, attacking the Big Idea as if it were a foreign invader, reducing the idea to a benign and much less exciting version of itself. The end result is something that no one can really get all that excited about. It may get the job done but it certainly doesn’t inspire anyone.

It’s really difficult to fight off this group tendency. It’s much, much easier to say no to something than yes, especially when there’s an established status quo. Once a group (company, club, team, non-profit, whatever) feels safe, group risk aversion magnifies problems into insurmountable barriers. Kathy Sierra (boy, do I miss her blog!) talked about this too, avoiding risks leads to the safest route, but the safest route, ultimately will only yield incremental improvements:

In most cases, the barriers preventing new Big Ideas from being achieved are no bigger than the barriers overcome by the group when it formed. The main difference is that there’re more people to get over the wall now than when it was just you and your Big Idea which brought the group together.

As a group it’s important to fight risk aversion which is often reinforced by groupthink. It’s a difficult job, but if growth and innovation are goals, taking risks is an essential part of success. Figuring out how to work with Big Ideas rather than fight against them (and in turn deal with risk as an organization) makes it easier to jump the wall as a group when and if the time comes. Delegating is a great way to build practice taking risks If you’re able to trust someone else to do something that you could reasonably do yourself you’re on your way to letting others run with their Big Ideas.

As an individual with a Big Idea, it’s important to quickly figure out whether your idea is something that has legs or whether it’s something better left to the back burner. Seth Godin’s The Dip has some simple advice but ultimately it’s up to you to determine. If you yourself are not prepared to pull each and every member of the group over the wall that is blocking you from realizing your ultimate vision then it may not be time to unveil your idea to the group.

Both the group and the innovator have a part in killing a Big Idea or making it fly. Recognizing that you have to work together is the first step toward achieving greater success.

During a brainstorming session in a recent OOPSLA workshop I participated in we were discussing what qualities make for a “good” architect in an agile world. I jotted down this list during the discussion based on the group brain dump. Some archetypes are “good” while others are “bad” but I don’t think any of them necessarily describe the perfect architect. When you’re designing software, what kind of architect are you? What kind do you prefer to work with?

The Megalomaniac Architect: “You will implement the system as I have designed it because I am the most important person on this team, the project will fail without me, and I am the only one smart enough to know everything that needs to be known!”

The Benevolent Dictator: “Implement the system precisely as I have designed it and I will make your work easier.” [via Dennis Mancl]

The “Everything’s a Nail!” Architect: All problems can be solved by a single tool, idea, or way of thinking. He’s got a hammer…and everything he sees is a nail.

The Diva: “I’m the architect of this system! Are you questioning my ability to design? If you’re not going to appreciate my talents, maybe I’ll just go to another team.”

The Magician: Excellent at high level abstraction, vague on how the design actually achieves architectural drivers.

The Over-Accommodating Architect: “You want that change? No problem.” There are no trade-offs to consider when making architectural decisions.

The Chess Master: To achieve victory, it is simply a matter of being able to see enough moves ahead. Every element in the architecture can be strategically placed in a perfect position to checkmate your opponent: requirements changes.

The PowerPoint Architect: “Architecture is just pretty pictures!”

The Ninja: Infiltrates a project from the outside, crafts an amazing design, then disappears into the shadows before anyone realizes he’s gone. Ninjas do not suffer fools and their designs are technically correct, thorough, and beyond your ability to comprehend.

The Navigator: Creates a map (with legend) and uses it to plot a course through implementation, testing, and deployment.

The Movie Producer: Leads the team indirectly by providing technical design support. [via George Fairbanks]

The Coach: Teaches the team about architectural practices and concepts as well as the design.

The Puppeteer: Able to manipulate multiple levels of abstraction simultaneously to design the architecture, the Puppeteer effortlessly manages the various threads influencing the design (commonly known as architectural drivers).

The Seasoned Veteran: He’s tried a lot of things and is pretty good at all of them including programming, design, processes, and management. Thanks to his experience, having walked 10,000 miles, he understands the role of architecture across the lifecycle, has seen many different situations, and is an excellent technical practitioner. [via Bill Opdyke]

The Long-Bearded Wise Man: A little philosophical but always willing to share an enlightened thought that will help resolve whatever concerns are ailing the project, though somewhat indirectly.

The Team Captain: A bona fide member of the team, playing on the field with everyone else, leading the team from the field.

The Design Evangelist: Excited about the architectural drivers and architecture to the point of near fanaticism. His enthusiasm for architecture and the system’s design is infectious and helps maintain conceptual integrity of the system.

The Student: Takes time to learn about the problem domain and the customer’s needs to the greatest extent possible so that he understands and solves the right problems. [via Gail Harris]

What other archetypes have you worked with? Which archetypes do you think might combine to make the perfect software architect?

Most software processes are light in two areas: quality assurance and process improvement. Most processes prescribe specific techniques for ensuring the production of quality code. XP for example advocates unit testing with TDD, continuous integration with smoke tests, pair programming, and acceptance tests written by the customer (using something like FIT so you can run automated regressions). Process improvement in XP is accomplished in a ’round the campfire, Kumbaya singing, get in touch with your feelings brainstorming session. Assumptions abound and there is no systematic way of ensuring that either testing or process improvement is handled adequately.  As we know, assumptions are never good enough.

The biggest assumption in XP (and indeed most software processes) is about bug tracking. Common sense dictates that you will create some kind of bug database. Hopefully it will at least be some kind of third party bug tracker such as Bugzilla or Bug Genie. Excel will work in a pinch but quickly becomes unsuitable for teams larger than one developer.  But how does the bug tracking actually work? What bugs get reported? Will you record issues from inspections in your bug tracker or only “true” bugs? What is the process for fixing a bug? What is the process for closing a bug? Who has access to the bug tracker? What information is required in your bug database and what information is optional? How do you determine defect priorities? Or the severity of bugs?

The majority of software processes provide answers for almost none of these questions. You are largely on your own to make up whatever you think makes the most sense for your development environment based on the best practices for your software process and your understanding of “good” software quality assurance practices.

No matter what quality process you follow, you will need a defect control philosophy. Once again, in absence of guidance I turn to the Team Software Process, one of the few processes to define what it means to track defects. In the Team Software Process, defects are treated as blight, a horrific mistake injected through the ineptitude of a developer. To remove these blights, the TSP relies on a series of filters in the form of code reviews, code inspections, unit tests, function tests, and so on. XP has a similar, though somewhat less rigorous set of filters in the form of pair programming, TDD, continuous integration, and acceptance tests. Each filter is meant to remove more and more defects, until finally “all” defects are removed from the system by the time the software has passed through all the filters. Generally each filter is intended to remove different types of defects, though it is conceivable to capture escaped defects from a previous filter in a later filter.

Just as water passing through layers of sand and rock will remove debris, so too will code passing through layers of unit tests and inspections sift out injected defects.

Defect Data

With these ideas in mind, bug tracking has three basic goals.

1. Record defects so they can be analyzed and fixed.
2. Identify the means by which defects are injected.
3. Identify the means by which defects are removed.

We write down bugs so we can go back and fix them later. This is obvious. But bug data can be used to measure process improvement also.

Since many parts of a software process are dedicated to filtering out the defects we’ve injected, understanding how defects are injected is essential to preventing similar defects from being seen in the future. The idea is that we want to learn from our mistakes. To achieve this, record the type of defect and the reason it was injected. The TSP gives us a good starting point for each of these, shown in the tables below. You should feel free to modify the types and reasons so they make sense for you, your team, and your project.

The defect type characterizes what kind of defect is injected and captures the essence of what is needed to fix the defect.

The defect’s reason characterizes why the defect was inject.

Assigning the reason can be a little tricky. I have found it is best if the person who injected the bug assigns the reason. On small teams (or if you’re following the PSP), this is generally pretty easy. It’s not about rubbing their nose in the problem – well, actually it is. Ok, it’s not about embarrassing or punishing the person but creating an opportunity for learning from our mistakes. If you can understand the reason why a defect was injected, it’s possible to prevent the defect from occurring again. For example, if there seems to be a rash of education related defects in a particular module, perhaps some training is in order.

Ideally we’d also like to know when the defect was injected, as in at what phase of development. It is possible to realize this information with additional analysis but I’ve found the return to be rather diminutive. Basically, you’ll learn what we’ve known all along – that the longer a defect is in the system, the harder it is to get out and that the most expensive defects are injected during the earlier phases of development (e.g. design defects are costly). Rather than track when things were injected I think it makes more sense to track when defects are detected. The point is to gain an understanding of how well the quality process filters out defects. To accomplish this, simply write down what you were doing when you found the bug. If you’re using XP, the list might include designing, writing new code (in a pair), writing new code (alone), refactoring, unit testing, integration, and acceptance testing. With this information you should be able to determine how effective each activity filters defects and over time whether the quality process is having issues. For example, I would expect interface defects to be detected during integration. If they are being detected earlier, say during unit testing, or later, say during acceptance testing, then my continuous integration and smoke test suite might not be as robust as it should be.

Better Bug Tracking

The strategies I’ve outlined here are a little more sophisticated than your average bug tracker, but add a lot of punch for very little effort. Tracking defect type, reason injected, and phase detected allow you to get a better handle not only on how defects are being injected into the software, but also how they are being detected. Both these chunks of information are necessary for understanding how defects are making their way into the system and how your process is helping you ferret them out of your system.

Van Halen may have known more about project management than most program managers. Van Halen’s legendary “No Brown M&Ms Rider” is simultaneously the greatest example of rock star excess and project signaling I’ve ever seen. As David Lee Roth puts it:

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say “Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . .” This kind of thing. And article number 126, in the middle of nowhere, was: “There will be no brown M&M’s in the backstage area, upon pain of forfeiture of the show, with full compensation.”

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

In economics, signals are indicators that convey specific meaning between producers and consumers. For example, when you see THX on the side of a set of speakers, you know the speakers are going to probably be of audiophile quality. The THX logo is the speaker manufacturer’s signal to you, the consumer, that these speakers are really good. To David Lee Roth and the Van Halen road crew, the presence of brown M&Ms indicated that the hosting venue had not understood all details of the contract and had very likely made a mistake in configuring the set. One mistake in this case could cause malfunctions during the show or even the death of a crew member.

As it turns out, signaling software projects isn’t that difficult. The 12 step Joel Test is a reasonable signal for software development companies. While the Joel Test is nice for getting a feel for a company before you work for them, the concept is still useful once you’ve got the job and the project is in full swing.

Ultimately signals, also known as tripwires or triggers, are really just binary metrics for uncovering potential problems your project might be facing before the problems explode in your hands. When some condition is met (the signal), you know it has specific significance and prompts certain actions to prevent a problem from occurring. Triggers are most often used with risk management but their use should not be exclusive to that practice. In fact, if you’re collecting real data, you have even more opportunities for identifying signals outside of risk management.

On past projects I’ve used signals for a variety of issues. Here are some examples.

• During the past 3 iterations the team identified between 15 and 20 defects. I expect a similar number of defects to be detected for this iteration. If more defects are detected, there may be a disconnection in understanding between requirements, design, and implementation. If fewer defects are detected, tests may not have been as rigorously defined as they should have been.
• A Fagan inspection completed in less than one hour with a rate of 400 LOC/hour. Since most inspections have covered only 250 LOC/hour it is likely that this inspection was not effective and the results not reliable since the inspection team sped through the code.
• When evaluating potential open source libraries, Source Forge projects without a website shows a general lack of dedication to the project and indicates that the software is probably of poor quality or ill-maintained; the library is worth neither the time nor effort to use.
• Tasks that have been estimated to require longer than 9 hours have probably not been thoroughly thought through.
• No risks have been identified for this project or risks have not been updated for several iterations. This implies that the team doesn’t have a realistic understanding of what problems the project faces.

In each of these examples, when the signal is heard, I knew there was going to be a problem on the project.

Work with your team to establish signals for your project. The best part is that once you’ve decided on the signals for your team, when triggers are tripped you can throw a Van Halen sized rock star fits in your cubicle! Well, try to resist throwing your monitor out the window anyway.

Software developers are, in their heart of hearts, dataphiles – people who are absolutely in love with data. When was the last time you had a passionate discussion about frame rates, hardware benchmarks, gadget specs, sports statistics, dungeons and dragons, the merits of high def…the list goes on. Face it, you love data. You love comparing things using data. You don’t feel comfortable making decisions without a comprehensive comparison of data.

Why then do most software developers treat software development differently?

Tom DeMarco recently brought his own famous quote into question (pdf), musing that not only is it possible to control what you can’t measure, but the most important stuff you need to control on a software project is impossible to measure. Once again, DeMarco is wrong (in my opinion anyway). To prove his point DeMarco pointed at Wikipedia, something extremely valuable that was built without the use of metrics or formal control. This is a romanticized view of Wikipedia.

Wikipedia is one of the most controlled projects on the planet

On the surface, Wikipedia is the Wild West of online content. Not only can anyone edit any page, but content from Wikipedia is widely proliferated in the media and (sadly) school reports. Wikipedia is the single greatest success of user generated content in the history of mankind (“The Internet,” as the medium, doesn’t count). What started with a dozen humble articles has evolved into the most comprehensive encyclopedia ever created and includes everything from the fundamentals of science to the definitive source on Babylon 5.

What folks seem to forget is that even in the Wild West, there were laws and there were lawmen. Though we love to think romantically about such brigands and gunslingers as Jesse James, Billy the Kid, and Butch Cassidy, most stories about these historic figures are greatly exaggerated. So too is the case with Wikipedia.

Let’s take a closer look at the Wikipedia entry for Billy the Kid. This article belongs to a number of internal WikiProjects, visible from the top of the article’s talk page. The WikiProject Biography is not unlike most projects in Wikipedia. There are defined processes for assessing articles and conducting peer reviews. There are rubrics defined for assessing the quality of articles within the project. People even take on specific roles and responsibilities within the project. The collection of processes and information serves as the main means of coordination for contributors and helps the group control articles within the scope of the project.

The WikiProject Biography even collects metrics on articles which it then uses to make decisions concerning the articles under the project. The metrics are derived from quantifiable data and help control the project.

As it turns out, Wikipedia is not the lawless territory of the internet it has been made out to be.

You can measure the immeasurable

Wikipedia works because people were able to figure out ways to measure things that usually can’t be measured. The fundamental principle that many people overlook is that binary is a metric too. Yes or no questions can be just as effective a measure as any complex metric. Did everyone fill out their task data today? Yes or no. Did the estimate match the actual? Yes or no. Did the test pass? Yes or no. Is the project done? Yes or no. Have we identified risks? Yes or no. Has this risk become a problem? Yes or no.

At the heart of every complicated metric is really a series of yes or no, binary questions. When considering whether the project is done, you have to define done. One way of defining done is in terms of a checklist. Is feature 1 done? Is feature 2 done? Defining done for a feature could be as simple as checking whether all the tests have passed for the feature, again a binary measure.

For more subjective assessments, you can rely on observation-based, experience-defined rubrics. Does the team get along with one another? In the simplest form, this could be a binary metric (Am I friends with everyone on the team?) but it could also be more complicated relying on gut feelings and a guiding rubric (“we never hang out together and don’t trust one another” might indicate low harmony while “we hang out often and feel comfortable sharing personal stories” could indicate high harmony). Teachers use rubrics and experience to judge subjective assignments everyday. The difference is that they slap a grade on it and send it home as a report card.

While DeMarco is correct that many of most critical things in a project are the most difficult to measure, it is possible to create measurements if you feel it is important enough to do so. How would you assess whether you have a good architecture that solves the problem at hand? Rubrics might play a part but so too might binary gates based on quality attribute scenarios or intricate observations concerning design trends over time. If you think hard enough, you’ll find that it’s extremely easy to find measuring points for nearly every aspect of a software project.

Whatever you do, don’t become a mindless, data-driven robot

I love data and I know you do to. While it’s tempting to inject data collection and derive metrics for every aspect of a project (because it’s fun and informative!) don’t. Collecting data and calculating metrics can be expensive. Not so expensive that you shouldn’t use it, but expensive enough so that you shouldn’t use it on everything. I like to compare using metrics to eating out at restaurants. Once or twice a week isn’t that big a deal, but it’s not something you should do every day if you’re trying to watch your budget.

DeMarco is right about one thing: control is not the end-all-be-all of software engineering. Consider carefully, what are the most risky parts of my project? What are the parts of my project that even require control? What are the parts in which I need more insight or want to improve? Strategically develop metrics for these areas and don’t worry about measuring the rest. Trust me, the world won’t end. If you don’t know what you’re doing, start with a simple binary measure. And above all, if something isn’t working, change it.

I was extremely disappointed to read a recent article on Coding Horror reflecting on an IEEE editorial written by Tom DeMarco. If you have not already, please read Tom DeMarco’s article now. It’s only two pages and it’s well written.

With all due respect, Tom DeMarco is wrong.

And Jeff Atwood made things worse.

According to Atwood’s interpretation of DeMarco, since we can’t control software projects, there is no sense in trying to engineer software.

What DeMarco seems to be saying — and, at least, what I am definitely saying — is that control is ultimately illusory on software development projects. If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.

Atwood’s conclusion simply is not supported by DeMarco’s article. DeMarco made two points in his piece.

1. We don’t have as much control over software as we think we do — even when we can measure the software on which we work.
2. We should be focusing more on the upfront “conception” activities than the areas that currently receive the most attention, construction.

My interpretation of “conception” activities are things like requirements, architecture, and design — details that ultimately help you figure out whether it makes sense to build the thing you think you want to build. By framing DeMarco’s argument as “craftsmanship” vs. “engineering” Atwood misses the whole point and reopens the tired art or engineering debate.

Overlooked by Atwood, DeMarco never questioned the idea that software should be engineered.

I’m gradually coming to the conclusion that software engineering is an idea whose time has come and gone. I still believe it makes excellent sense to engineer software. But that isn’t exactly what software engineering has come to mean. The term encompasses a specific set of disciplines including defined process, inspections and walkthroughs, requirements engineering, traceability matrices, metrics, precise quality control, rigorous planning and tracking, and coding and documentation standards. All these strive for consistency of practice and predictability.

DeMarco is really saying that the engineering part of software engineering has become overshadowed by a collection of best practices for building software. In my mind this isn’t necessarily a bad thing. All it means is that what has become known as “software engineering” is different than the original definition intended by the NATO Conference on Software Engineering.

But by discounting current software engineering practices, DeMarco dismisses the real engineering that went into advancing the field to where it is today.

DeMarco seems to imply that what we really want software engineering to be–the application of systematic, disciplined, quantifiable approaches to the development of software–and what software engineering has become cannot coexist. Essentially, to reach a state where metrics and measures, quantifiable approaches, are used correctly and consistently by the software development community we must stop using the term “engineering” to describe the current set of practices.

This is backwards thinking.

Engineering is more than something you do; it’s also a way of thinking about problems and solutions. Reaching the point in software engineering that we are today required systematic, disciplined, and quantifiable thinking. Over time results of this thinking have been codified into the set of best practices that most developers now take for granted.

For example, we know that there is a 100x or more difference in costs between defects discovered later in the software lifecycle than earlier. We know that certain practices can effectively remove defects at different costs and at different times throughout the lifecycle (for example, inspection vs. prototyping vs. unit testing vs. system testing). We also know that historical data is an excellent indicator of future performance on software projects.

Systematic, disciplined, quantifiable thinking was required to make these discoveries.

Because of these codified best practices, it is not always necessary to conduct experiments on a project to trust that they are working. I know unit testing combined with regular system integrations will flush certain defects from my software before those defects become a problem during system testing. I know that statistical analysis of collected task tracking data will help me better predict how long future tasks of a similar size will take. It doesn’t matter whether I completely understand the engineering behind the practice or whether I simply follow the process or use the tool. The benefits will be the same.

Does that make me less of an engineer? I don’t think so.

Using best practices codified as processes, methods, or tools on a software project means you are engineering software whether you like it or not. With many of these practices, the control mechanisms are already built in so you don’t realize that you’re already controlling your project. As DeMarco points out, it simply isn’t necessary that every engineering detail be painstakingly scrutinized for a project to be successful. For many projects, the essence of the project is sufficient to overcome the accidents encountered when engineering by coincidence.

But just because you engineer by coincidence it doesn’t make you a software craftsman. To prove it, I’m calling Jeff Atwood out. Jeff, I dare you and the Stack Overflow team to take the PSP Challenge. Take a course on the Personal Software Process, honestly give it a try — use actual software engineering for a few weeks — then tell me that software engineering is dead. But don’t knock it until you’ve tried it.

Building a great software product is only half the battle. The relationship the user builds with your software has the single greatest influence on how awesome that software is. Sorry. It’s not the language the software is written in. It’s not the algorithms. It’s not all the processes you used behind the scenes to create it. If the relationship sucks, your software sucks. Period.

To prove this I’m going to use a tangentially-related personal experience that has absolutely nothing to do with software. In this case, it’s ok, because realistically, this bit of reflection actually has nothing to do with software – remember it’s about relationships. Let’s talk about two concerts I’ve seen this summer.

Coldplay: “You’re Awesome!”

I’ve heard Coldplay’s radio hits but I was never really a huge fan of the band. Had it not been for a friend’s invitation I would never have gone to this concert and I would have missed out on one of the best live shows I’ve seen. Acceptable radio hits became handcrafted masterpieces, more amazing live than on FM. The most interesting part of the show, and the thing that left the greatest impression on me, was the band’s absolute humble devotion toward the fans in attendance. At one point the band walked out to mini-stages set up in the seated area under the pavilion and way out in the lawn, giving all fans an equal chance to be a part of the show, not just the diehards that purchased overpriced tickets.

Coldplay’s attitude created a completely positive vibe that made me want to enjoy myself.  After they had exhausted themselves for close to two hours, doing everything possible to create a positive experience for the audience, Coldplay thanked us, not for coming to see them, but for allowing them to play for us, “You’re awesome, thank you for letting us play for you tonight!”

Dave Matthews Band – “We’re Awesome!”

I didn’t realize that Dave Matthews Band live shows are really just one long meandering jam session.  Every one of the band members is an amazing musician, but 14+ minute songs start to wear after a while.  Dave Matthews rarely spoke (when he did it was incomprehensible) and most of the band wore sunglasses throughout the entire show. Overall, it was a very mediocre concert that was better suited to an afternoon picnic setting than an evening at a large outdoor amphitheatre.  From a technical perspective it was amazing but overall it failed to entertain me.

The most interesting thing about this concert was the crowd. In spite of the fact that the band seemed to be playing solely for their own amusement, the audience was full of absolute fanatics, complete with their own dress code – plaid shorts and tie dyed shirts (though rarely together) with sandals – and a shared emotional connection to every song the band played. Dave Matthews Band is here to play music – If you like it, whatever, if you don’t, whatever.  “We had a fun time tonight and I hope you did too!”

It’s all About the Relationship

It turns out that the relationship the bands built with the audience during the show was directly related to how much I enjoyed it. It should be pretty obvious how this extends to software.  The relationship your software builds with your users will determine how much they enjoy using it. Is your software helping users be amazing or does your software expect users to change their behaviors to use it? Are you building something for yourself that others might happen to like or are you targeting a specific audience?

Software that is technically the best will always fall short to similar software that builds a relationship. Kathy Seirra expressed a similar sentiment with her Dating Rules for Software. But there’s a small catch to Kathy’s rules as exhibited by the Dave Matthews Band. Apple tends to act this way too. Sometimes if you do your own thing and you actually are great, you can afford not to focus on the relationship. It’s extremely difficult but sometimes it works. Sacrificing the relationship creates a razor thin margin for error but when you get it right, you’ll breed fanatics that will love you (almost) unconditionally and critics who will absolutely hate you – there is no mediocrity.

Sacrifice the relationship and you could be the best, but only until someone almost as good comes along who thinks about others before themselves.  I think Microsoft has learned that lesson the hard way.  No matter what you do, relationships rule the day.