Reflections on Software Engineering

I’ve had to help more than a few teams get their version control systems sorted out over the past few years, and so I thought it would be easier if I just wrote down the philosophies I use for initializing a repository and getting the whole system set up. If you’re looking for some specific advice on how to set up and use a specific version control system, the Pragmatic Starter Kit Series for CVS, Subversion, or Git is a great place to start.

What should go into a source code repository?

The short answer: the repository should contain everything necessary to perform a clean build of your system. In most cases, this includes the code, third-party binaries necessary for building, tests, and documentation.

It’s ok to assume that everyone has their build environment “properly configured” for building. To make sure, make a list of everything that must be setup in the environment to build the software and put it on the team wiki. These things don’t need to be stored in the repository but you should at least write down what the standard build environment is supposed to look like. Depending on what the required software is, it might also be a good idea to keep a copy of it, just in case something happens to vendor in the future. The last thing you want is for a vendor to stop supporting the version of something you need, forcing you to upgrade because your hard drive crashed and you had to setup a new environment.

Include at least the following in your standard build environment list:

• Compiler versions
• Team sanctioned IDEs
• Required frameworks, toolkits, and build tools
• IDE extensions that the team has decided are so critical/awesome to the project they have to be used. Critical/awesome IDE extensions might enable a required tool-kit (such as GWT in Eclipse) or configure the IDE is specific ways (such as coding styles or static analysis settings)

Putting code and tests in the repository is fairly obvious, but third-party binaries (e.g. libraries) might not be. Put these in version control so that it’s easy to check out a project from source control and build without monkeying around with anything. I’ve found it best to create an “ext_lib” folder for storing all the external libraries. This way there is no confusion over what versions to use, and all the build paths can be set so that anyone can build just by checking out the code.

Here’s a real life example. Let’s say you’re writing a web application using the Google Web Toolkit and you rely on a caching library. The caching library should go into your ext_lib folder and you should tuck a zip of the GWT version you use away in a safe place just in case you need it later. Say your team is also using JUnit. Put the version you use in the ext_lib folder. This way everyone can build and use whatever GUI they want to run tests, be it the JUnit GUI or an Eclipse Plug-in.

Another real life example. Let’s say you use the excellent Sharp AutoUpdated component. Should you version the binary or the source? That was a trick question since it depends. The best answer is to only keep the binary of the library, but this isn’t always possible. One of the awesome things about open source software is that you have access to the source if you need it. So, let’s say you find a bug in the AutoUpdater and for some reason the maintainers aren’t responding quickly enough for your immediate needs. You can’t live with this bug so you have no choice but to fix it yourself. Congratulations, you just took ownership over your own fork of the AutoUpdater component. You now are responsible for maintaining the code – either in your version control library or in a public fork, and merging with the original code base may be more difficult in the future.

What doesn’t go into the source code repository?

Remember the DRY Principle for writing code (Don’t Repeat Yourself)? Well, that applies to your version control system too. Anything that can be derived shouldn’t be held under version control. Since your source code is already in the repository, storing the built binary is a violation of the DRY Principle. The penalty? Confusion, mistakes, and avoidable headaches. Third party libraries in the ext_lib folder don’t violate DRY since you can’t build them – you don’t own the source.

Also, do your fellow developers a favor and keep your personal stuff out of the repository. If you’re testing, nobody else wants to see your test reports. When you run the application, keep your logging messages to yourself. Also keep anything related to how you set up your personal environment in your personal environment. The last thing I want is to open up my IDE and see the last tabs you had open because you committed your personal user settings.

The easiest way to keep these sorts of undesirables out of the repository is by setting up an ignore list. Share it among the team.

How often do I commit?

Generally you should commit your changes anytime you think you’ve finished something useful that doesn’t introduce problems into the system. On the average, you should be committing changes at least once a day.

There’s two parts to this commit rule. “Finished something useful” might mean many things. This is by design. When you’ve finished a logical chunk of code that does something, feel free to commit it. “Doesn’t introduce problems” is a common courtesy to your fellow developers. Make sure, at a minimum, the system builds and passes any automated tests you have. And always update before you commit. Depending on your team size and how important the code is, you might establish a checklist for committing. Google has theirs automated. Every change the system has to build, pass tests, and pass a peer review before it can be committed.

Remember this mantra: Commit early, commit often.

But if everyone is committing all the time, isn’t that going to cause problems?

When you’re working with people and coordinating effort, problems will inevitably arise. Just remember, if you’re going to fail, fail early. It’s better to cause a conflict today through miscommunication while there’s plenty of time to fix it than the day before it’s time to deploy. Why? The conflicts will be smaller since you’re incrementally growing your code base. Also, since you made the changes recently they are fresh in your head and easier to work with. Code more than a week old might as well have been written by someone else.

Taking a risk management approach makes mitigating this easy. The risk: “Developers use a shared repository and commit changes frequently; might cause code conflicts that break the build.” The source of this risk is communication; therefore anything which helps facilitate communication can reduce the likelihood of this risk becoming a problem. Daily stand-up meetings are perfect for getting the word out about what everyone is working on. Automatically generated email updates from the version control system keep folks abreast throughout the day as changes are made. Continuous integration acts as a smoke test for uncovering integration problems while they’re small. Good merge tools can help reduce the impact of the consequence.

Once everyone gets used to the update-then-commit cycle, most of these problems go away. In my experience, big problems with code in the repository are usually a symptom of larger problems such as poor communication or failing processes.

What are some of your version control philosophies? What helps you keep things organized so you can get things done?

I was recently asked by a coworker how software risk management is different from traditional SWOT analysis. SWOT is a technique commonly used for strategic planning where the strengths, weaknesses, opportunities, and threats facing a group are compiled and analyzed to determine an appropriate course of action. Software risk management (as defined using the continuous risk management paradigm from the Software Engineering Institute) is similar in that risk management can be used for strategic planning but risks yield much different information which is applied in a very different way.

The first step when performing a SWOT analysis is to define the business objectives. This is very similar to defining a threshold of success in software risk management. The main difference is a business objective takes the form of the desired end state whereas the threshold of success is the minimum objectives necessary for the project to be successful. For example, a perfectly valid business objective might be to deliver all 100 story points by the end of the year while the threshold of success might be to deliver the core functionality (worth only about 50 story points). Would more stories completed be better? Of course, but what if you end up only completing 75 story points by the end of the year? How did you do? You missed your goal, but you still succeeded right? It’s difficult to tell without understanding the difference between wants and needs.

The main part of a SWOT analysis consists of a group session where strengths and weaknesses internal to the group and opportunities and threats external the group are identified. People like to put SWOTs into a 4×4 grid so it’s easier to look at. While there is some great advice out there for understanding what goes into a SWOT, the analysis is largely subjective, relying on a teams’ gut feelings to know the strengths from the weaknesses, the opportunities from the threats. Software risk management can be a much more systematic approach to understanding the potential dangers that face a project based on known facts when tools such as the SEI’s Taxonomy Based Questionnaire for risks (pdf) are used. Guts still come into play, but there is enough engineering in place to help people make the right decisions.

Risks are specifically actionable – depending on the risk you might be able to mitigate it by manipulating the timeline, impact of the consequence, probability of the risk occurring, or by addressing the condition. You might transfer the risk to someone else or simply accept the risk. SWOT by itself is merely a collection of statements relative to internal or external entities which may or may not actually be true. Are you good at testing? How do you know that? Is Bing really a threat to Google Search? Should you do anything about your weaknesses? Will they prevent you from achieving your business objectives? Without further analysis there really is no way to know and other than prioritizing there really is no way to analyze a SWOT, nor is there any clear direction for next steps.

Look, when planning a project you really need both SWOT analysis and risk management. SWOT is a tool for assessing capabilities while risk management is a tool for assessing the likelihood of success. Each technique serves a very different purpose. SWOT is most useful at the beginning of a project to help you figure out what you’re doing and come up with an overall strategy. Risk management, though is an ongoing activity that makes sure you don’t fall flat on your face in trying to achieve your business objectives.

Everyone has a chore they hate doing. For me it’s cleaning the dishes. I’m a busy guy so I usually don’t get around to cooking and eating dinner until fairly late. Rather than cleaning anything, I stack the dishes in the sink and maybe soak a pan if something burned to the bottom. If the dishwasher has space, I’ll load it and set it going but most nights I leave a big pile of dirty dishes sitting around. After two or three nights of this, all the pots and pans are dirty and there’s no room for cooking thanks to the piles of dirty dishes. It’s actually kind of disgusting.

My wife takes a slightly different tact. After the meal is finished she immediately cleans all dishes, pots, pans, and utensils used while she was cooking. She has the forethought to run the dishwasher beforehand so there is plenty of room to load dirty dishes after the meal. She even wipes down the counter and stove so everything is ready for the next meal we cook. She’s quite amazing actually and a good cook to boot (especially when she’s following a recipe).

Professional chefs take matters a step further still. They clean as they cook.

I procrastinate doing something I hate and the result is a monumental, exhausting chore which takes an hour or more to finish. My wife spends 10, 15 minutes tops a night “tidying up” and though she hates doing dishes just as much as I, she makes it seem effortless. Professional chefs make miracles in the kitchen minutes at a time.

So it is with software.

Integrating software, even with a small team can be a chore.  Which would you prefer?  Approach A: write a lot of code, get everything working individually, and then do a big bang integration at the end; or Approach B: write a little code and integrate a little. While putting off integrating might satisfy your immediate needs, much like skipping dishes and moving straight to dessert, Approach A is going to cost more than Approach B in the end. Why?

• Integration problems aren’t uncovered until you integrate (profound, I know) so the longer you wait to integrate, the longer it takes to find out if there is a problem. Of course, no one ever plans for problems…
• Conflicts have further reaching consequences the longer you wait to fix them. Modern version control systems usually do a pretty good job merging changes but even magic has its limits.
• The full power of refactoring can’t be realized because the turnaround time on changes is too long.  The side effect is that you don’t refactor which means the code becomes more brittle over time.
• More code changes means more time to bring it all together and a higher likelihood of introducing defects through integration.  Unless you’ve planned knowing that integration will take time, chances are good you’re going to ship late.

Even better than big bang and nightly builds: continuous integration, cleaning as you code. Automated build servers have made huge advancements over the past few years. I highly recommend Hudson. It’s super easy to install and get started and has plug-ins for practically everything. There are even Hudson plug-ins for C# and just about every version control system you could want (and even some you don’t).

Doing chores sucks, but don’t make it worse than it has to be by putting things off. Continuous integration is a no brainer. If I had a dishwashing machine that constantly washed dishes as I finished using them I would dance naked in the streets, celebrating the marvels of modern technology. (You should be thankful such a machine doesn’t exist.)

You really don’t have an excuse for not using continuous integration.

When I was a kid my brother and I used to play a game called Make Believe. My favorite variant of the game was simple. Together we would build some kind of fortress and then one person gets the fort and the other person tries to invade the fort. In theory, the game ends when the fort has been overtaken by the invader. What made the game fun was that as the invasion began, the rules of the game always changed. The first thing to go was any notion of death. If one of us was “killed” in battle then near instantaneous respawning was created. Shortly after that we skipped respawing and simply became invincible. Soon the fort became invisible which means the invader just has to run around trying to find it. Sometimes someone gained super strength or the ability to force other people to move in slow motion. We almost always created super weapons (such as a hand held Death Star) which for some reason could always be defended against. Nearly every game ended in tragedy, someone crying or upset: “That’s not fair! You can’t do that! I’m invisible! You can’t do that!”

Kid’s stuff right?

A lot of software projects with teams made up of working adults still play this game. The scenario goes something like this. A team is put together to build some software. Neither the clients nor the team talk about the objectives of the project other than building “some software”. After a few months, something goes wrong or someone doesn’t like what’s happening so someone changes the rules. Before too long, one side or the other is upset that they can’t win, somebody throws a fit, and goes home. Instead of summoning invisible armor, software projects change the rules by cutting features, adding more requirements, moving due dates, wasting resources, and things like that.

We make believe that we’re software engineers.

While Make Believe was a fun game as a kid, changing the rules when there’s real money on the line isn’t as fun. My brother and I ran into problems as kids because we got the objectives of the game wrong. Actually, there were no common objects, which is why we could change the rules so easily. The same thing happens on a software project when the objectives aren’t well known.

Defining and committing to a clear picture of success establishes the common ground rules for a project by making the basic project goals explicit. The technique is known as Threshold of Success.

Defining What Success Looks Like

The Threshold of Success for a project is the minimum set of conditions that must be met for the project to be considered successful. If the team fails to meet even one of the conditions then the project is a failure. A good Threshold of Success is made up of about 3-4 SMART goals (no more than a few bullets on a single PowerPoint slide). SMART is a mnemonic which stands for Short/Specific, Measurable, Achievable, Relevant, and Time bound.

Some other pointers for defining a Threshold of Success:

• The Threshold of Success should be built as a team. Since this is the measure by which you will define success or failure, everyone on the team must buy into it. If you can include your client that’s even better.
• Threshold of Success goals should be challenging, but it’s important that they are achievable. If the goals are too easy, victory will be meaningless, too difficult, elusive.
• Once the Threshold is established, don’t change it! The only reason to modify the Threshold of Success is if the project has changed so drastically that the Threshold no longer makes sense (for example if someone leaves the project).
• Revisit the Threshold of Success regularly (a good time is when planning iterations) so everyone remembers what success looks like. Put it on your team wiki so that it’s readily accessible.
• Be sure that the goals in your Threshold are SMART! The point of defining a Threshold of Success is to take away the wiggle room for defining what it means to succeed or fail. The goals you define should make this black and white. The more specific the goal is the better.

Building a Threshold of Success

The easiest way to create a Threshold of Success is to first create a minimum picture of failure, then convert failure into success. Here’s an example:

Failure for my current project might look something like this.

• Essential features are not ready by the end of the second quarter.
• Team members are dissatisfied or bored with their jobs.
• Newly hired team members don’t feel like they’re part of the team by March 31.
• There isn’t enough money to continue development after this fiscal year and we have to fire people.

Now that I know what failure looks like, seeing success is easy. I don’t want any of these things to happen. The threshold of success for my current project might look something like this.

• By the end of the second quarter, all “Must Have” features are implemented and pass acceptance tests with no known critical defects.
• All team members give average score of 5 or better on a job satisfaction survey taken quarterly.
• By March 31, the team has successfully executed at least three team building activities with all team members present.
• Funds of at least $1 million are secured by December 31 to allow for future development without a reduction in team size.

Notice that only 1/4 of the success goals in this example are related to software functionality. While goals might come from anywhere, teams traditionally focus on goals related to people and relationships, process, resources (such as budget or schedule), and product (software functionality and quality).

As this technique originated with the Software Engineering Institute (pdf), nearly every studio team in the Carnegie Mellon Master of Software Engineering program creates a Threshold of Success for their projects. The MSE Studio Archive has extensive examples of both good and bad pictures of success that teams have created. The Square Root Team’s threshold (my team) is a good place to start, but there are plenty of other examples.

There might be many goals for a project. In the Team Software Process you actually identify at least three different kinds! But there is only one threshold of success for a project. Knowing what success looks like gives you a better chance of actually achieving it.  Without it, you’re just pretending that you know what’s going on.

Late in the afternoon on December 25, during one of the loudest, howling winter storms I’ve ever experienced we lost power. Normally this wouldn’t be a big deal except I was in a vacation house with 20 other people, basically my wife’s entire extended family.

After the power went out, the heat did not fire up. The tree went dark. So did the TV, DVD player, and Wii.

Using a pair of LED headlamps my wife and I received for Christmas the 20 of us took turns rolling and stuffing homemade ravioli dough for dinner. Christmas ravioli making is extremely serious business and I had finally been promoted to “unmonitored ravioli stuffer” this year. Luckily we had manual pasta rollers to flatten the dough. There’s always talk of “upgrading” from the hand cranked system but this year, tradition trumped technology.

Once ravioli are stuffed, they have to dry for a few hours before cooking. In the years past this was the time to play with new games or watch a new movie. Of course, without electricity most of our new toys were rendered useless.

Since the heat was off and the vacation house was gigantic, the fireplace in the living room was our only option for warmth. Lighting the room was the handful of candles we had, originally intended for dining ambiance, and the low glow from the fire.

To pass the time, we sang Christmas carols in the near dark. It was all sort of surreal, a setting I am certain we would not have created on our own had we not lost electricity, had we not been so completely pushed out of our comfort zone. Even more fantastic was that after over 20 years of putting it on my wish list, someone actually gave me a pair of night vision goggles for Christmas. Surprisingly, this “toy” is the real deal. I was able to wander around a completely pitch dark house with no problems. The night vision, along with the head lamps (which single handedly saved dinner), were by far the best gifts of the day. The electricity turned on around 1:00am that morning and by the next morning, everything went back to the way it was.

Sitting in the dark on Christmas night got me thinking. This was an experience that I never would have chosen, a situation I would never have intentionally embraced, but it turned out to be pretty fun. How many opportunities have I missed because my default attitude was to stay inside my comfort zone? While happy accidents are great, can’t I do more to create opportunities rather than relying on happenstance? Given the time of year it only seems appropriate to ask these sorts of questions.

I’m a technologist, a scientist, and an engineer at heart. I love playing with gadgets, tinkering with software, and working on interesting and challenging problems. Sitting in the dark with no power, my normal pursuits removed, it was easy to remember that people are important too. Sure, when building software everyone always talks about how people are important, but when building software we call people “users” essentially reducing their humanity. After all, there are only two industries where the customers are called “users” – software is one of them.  Is it really appropriate to use the same term to describe software clients and drug addicts?

My lesson from the night: the technology and gadgets and programming and processes and everything else are awesome, but they are meaningless if they fail to create a genuine relationship with people.  I do believe that relationships matter and that people are important. I’d like to do a better job of thinking about people first this year. In the rush of excitement surrounding every new technological achievement, it’s sometimes easy to forget that helping people is why I build software.  It’s too bad it took a harsh winter snow storm to remind me of this.

Almost a full year ago, I concluded that formal methods simply aren’t worth the effort:

For almost every project in the world, I think formal methods should be generally avoided. Given the option of spending money and time on mathematicians or extremely smart coders I would chose the latter. With smart coders, code inspection is a fun and effective defect filtering option. And let’s face it. Why would you have your amazing coders do something other than write amazing code?

But what if formal methods didn’t have to be so…formal?

While traditional formal methods such as Z might be a little difficult to pick up if your predicate logic is rusty, applying formalisms that are close to the code doesn’t seem that outrageous in perspective. Design-by-contract language extensions such as Spec# for C# or Plural for Java have been stealthily making their way into IDEs for years now without anyone really knowing the wiser, without anyone really thinking about these tools as being formal methods. State machines in UML and even some uses of domain specific languages might be considered formal methods when these tools are used for specification and analysis purposes.

Creating formal methods that are close to the code is one of the best ideas to come out of the formal methods arena in the past 20 years. Just like how adding cherry flavor makes it easier for kids to take icky tasting medicine, putting formal methods in the code makes it more likely that I, as a programmer, will actually use them. Because no matter how good for me something is, if it’s difficult to use or generally unpleasant, it won’t get used.

This is why domesticating formal methods (pdf) is extremely important for formal method adoption.

Wild formal methods can be difficult to work with. No matter what anyone tells you it does take time to get the hang of Z. Formal methods such as Z have limited use when talking with customers and, though a formal specification is an excellent tool for gaining better understanding of functional requirements, the payoff for creating the specification isn’t seen for weeks or months making it difficult to justify their use in Agile development environments.

But is turning what was once a ferocious and unpredictable wolf into an obedient and trusted canine companion safe? In selecting the features for our domesticated formal methods, did we accidentally breed out the most important benefits?

Anthony Hall outlined seven myths of formal methods (pdf), ideas about formal methods erroneously taken as truth.

Myth 1: Formal methods can guarantee that software is prefect
Myth 2: Formal methods are all about program proving.
Myth 3: Formal methods are only useful for safety-critical systems.
Myth 4: Formal methods require highly trained mathematicians.
Myth 5: Formal methods increase the cost of development.
Myth 6: Formal methods are unacceptable to users.
Myth 7: Format methods are not used on real, large-scales software.

Any formal method worth using should not uphold any of Hall’s seven myths, but domesticated formal methods have the additional burden of usability. Just like we expect certain behavior from our four-legged friends, I don’t think it’s too much to ask that a domesticated formal method be friendly and obedient. Any method which bites the master’s hand obviously can’t be trusted. Crashing, making it easy to make mistakes, poor affordances, difficult to read output, impossible to maintain specifications – these are all reasons for not trusting a domesticated formal method.

I think informal formal methods are great. When done well, it doesn’t even feel like I’m using a formal method and I get many of the same benefits a wild formal method would give me – clarity, understanding, and maybe even a little automated verification depending on the language and method. Domain specific languages and state charts are great for working with end users and clients. It’s even plausible to skip other testing or verification measures such as unit testing or inspection.

And the best part is that since many domesticated formal methods are close to the code, I’ve already got the necessary training on how to use the methods because I already know how to program.  Domesticated formal methods are a win-win for programmers who want to do more engineering and another practical tool for your silver toolbox.

During a brainstorming session in a recent OOPSLA workshop I participated in we were discussing what qualities make for a “good” architect in an agile world. I jotted down this list during the discussion based on the group brain dump. Some archetypes are “good” while others are “bad” but I don’t think any of them necessarily describe the perfect architect. When you’re designing software, what kind of architect are you? What kind do you prefer to work with?

The Megalomaniac Architect: “You will implement the system as I have designed it because I am the most important person on this team, the project will fail without me, and I am the only one smart enough to know everything that needs to be known!”

The Benevolent Dictator: “Implement the system precisely as I have designed it and I will make your work easier.” [via Dennis Mancl]

The “Everything’s a Nail!” Architect: All problems can be solved by a single tool, idea, or way of thinking. He’s got a hammer…and everything he sees is a nail.

The Diva: “I’m the architect of this system! Are you questioning my ability to design? If you’re not going to appreciate my talents, maybe I’ll just go to another team.”

The Magician: Excellent at high level abstraction, vague on how the design actually achieves architectural drivers.

The Over-Accommodating Architect: “You want that change? No problem.” There are no trade-offs to consider when making architectural decisions.

The Chess Master: To achieve victory, it is simply a matter of being able to see enough moves ahead. Every element in the architecture can be strategically placed in a perfect position to checkmate your opponent: requirements changes.

The PowerPoint Architect: “Architecture is just pretty pictures!”

The Ninja: Infiltrates a project from the outside, crafts an amazing design, then disappears into the shadows before anyone realizes he’s gone. Ninjas do not suffer fools and their designs are technically correct, thorough, and beyond your ability to comprehend.

The Navigator: Creates a map (with legend) and uses it to plot a course through implementation, testing, and deployment.

The Movie Producer: Leads the team indirectly by providing technical design support. [via George Fairbanks]

The Coach: Teaches the team about architectural practices and concepts as well as the design.

The Puppeteer: Able to manipulate multiple levels of abstraction simultaneously to design the architecture, the Puppeteer effortlessly manages the various threads influencing the design (commonly known as architectural drivers).

The Seasoned Veteran: He’s tried a lot of things and is pretty good at all of them including programming, design, processes, and management. Thanks to his experience, having walked 10,000 miles, he understands the role of architecture across the lifecycle, has seen many different situations, and is an excellent technical practitioner. [via Bill Opdyke]

The Long-Bearded Wise Man: A little philosophical but always willing to share an enlightened thought that will help resolve whatever concerns are ailing the project, though somewhat indirectly.

The Team Captain: A bona fide member of the team, playing on the field with everyone else, leading the team from the field.

The Design Evangelist: Excited about the architectural drivers and architecture to the point of near fanaticism. His enthusiasm for architecture and the system’s design is infectious and helps maintain conceptual integrity of the system.

The Student: Takes time to learn about the problem domain and the customer’s needs to the greatest extent possible so that he understands and solves the right problems. [via Gail Harris]

What other archetypes have you worked with? Which archetypes do you think might combine to make the perfect software architect?

Most software processes are light in two areas: quality assurance and process improvement. Most processes prescribe specific techniques for ensuring the production of quality code. XP for example advocates unit testing with TDD, continuous integration with smoke tests, pair programming, and acceptance tests written by the customer (using something like FIT so you can run automated regressions). Process improvement in XP is accomplished in a ’round the campfire, Kumbaya singing, get in touch with your feelings brainstorming session. Assumptions abound and there is no systematic way of ensuring that either testing or process improvement is handled adequately.  As we know, assumptions are never good enough.

The biggest assumption in XP (and indeed most software processes) is about bug tracking. Common sense dictates that you will create some kind of bug database. Hopefully it will at least be some kind of third party bug tracker such as Bugzilla or Bug Genie. Excel will work in a pinch but quickly becomes unsuitable for teams larger than one developer.  But how does the bug tracking actually work? What bugs get reported? Will you record issues from inspections in your bug tracker or only “true” bugs? What is the process for fixing a bug? What is the process for closing a bug? Who has access to the bug tracker? What information is required in your bug database and what information is optional? How do you determine defect priorities? Or the severity of bugs?

The majority of software processes provide answers for almost none of these questions. You are largely on your own to make up whatever you think makes the most sense for your development environment based on the best practices for your software process and your understanding of “good” software quality assurance practices.

No matter what quality process you follow, you will need a defect control philosophy. Once again, in absence of guidance I turn to the Team Software Process, one of the few processes to define what it means to track defects. In the Team Software Process, defects are treated as blight, a horrific mistake injected through the ineptitude of a developer. To remove these blights, the TSP relies on a series of filters in the form of code reviews, code inspections, unit tests, function tests, and so on. XP has a similar, though somewhat less rigorous set of filters in the form of pair programming, TDD, continuous integration, and acceptance tests. Each filter is meant to remove more and more defects, until finally “all” defects are removed from the system by the time the software has passed through all the filters. Generally each filter is intended to remove different types of defects, though it is conceivable to capture escaped defects from a previous filter in a later filter.

Just as water passing through layers of sand and rock will remove debris, so too will code passing through layers of unit tests and inspections sift out injected defects.

Defect Data

With these ideas in mind, bug tracking has three basic goals.

1. Record defects so they can be analyzed and fixed.
2. Identify the means by which defects are injected.
3. Identify the means by which defects are removed.

We write down bugs so we can go back and fix them later. This is obvious. But bug data can be used to measure process improvement also.

Since many parts of a software process are dedicated to filtering out the defects we’ve injected, understanding how defects are injected is essential to preventing similar defects from being seen in the future. The idea is that we want to learn from our mistakes. To achieve this, record the type of defect and the reason it was injected. The TSP gives us a good starting point for each of these, shown in the tables below. You should feel free to modify the types and reasons so they make sense for you, your team, and your project.

The defect type characterizes what kind of defect is injected and captures the essence of what is needed to fix the defect.

The defect’s reason characterizes why the defect was inject.

Assigning the reason can be a little tricky. I have found it is best if the person who injected the bug assigns the reason. On small teams (or if you’re following the PSP), this is generally pretty easy. It’s not about rubbing their nose in the problem – well, actually it is. Ok, it’s not about embarrassing or punishing the person but creating an opportunity for learning from our mistakes. If you can understand the reason why a defect was injected, it’s possible to prevent the defect from occurring again. For example, if there seems to be a rash of education related defects in a particular module, perhaps some training is in order.

Ideally we’d also like to know when the defect was injected, as in at what phase of development. It is possible to realize this information with additional analysis but I’ve found the return to be rather diminutive. Basically, you’ll learn what we’ve known all along – that the longer a defect is in the system, the harder it is to get out and that the most expensive defects are injected during the earlier phases of development (e.g. design defects are costly). Rather than track when things were injected I think it makes more sense to track when defects are detected. The point is to gain an understanding of how well the quality process filters out defects. To accomplish this, simply write down what you were doing when you found the bug. If you’re using XP, the list might include designing, writing new code (in a pair), writing new code (alone), refactoring, unit testing, integration, and acceptance testing. With this information you should be able to determine how effective each activity filters defects and over time whether the quality process is having issues. For example, I would expect interface defects to be detected during integration. If they are being detected earlier, say during unit testing, or later, say during acceptance testing, then my continuous integration and smoke test suite might not be as robust as it should be.

Better Bug Tracking

The strategies I’ve outlined here are a little more sophisticated than your average bug tracker, but add a lot of punch for very little effort. Tracking defect type, reason injected, and phase detected allow you to get a better handle not only on how defects are being injected into the software, but also how they are being detected. Both these chunks of information are necessary for understanding how defects are making their way into the system and how your process is helping you ferret them out of your system.

Van Halen may have known more about project management than most program managers. Van Halen’s legendary “No Brown M&Ms Rider” is simultaneously the greatest example of rock star excess and project signaling I’ve ever seen. As David Lee Roth puts it:

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say “Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . .” This kind of thing. And article number 126, in the middle of nowhere, was: “There will be no brown M&M’s in the backstage area, upon pain of forfeiture of the show, with full compensation.”

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

In economics, signals are indicators that convey specific meaning between producers and consumers. For example, when you see THX on the side of a set of speakers, you know the speakers are going to probably be of audiophile quality. The THX logo is the speaker manufacturer’s signal to you, the consumer, that these speakers are really good. To David Lee Roth and the Van Halen road crew, the presence of brown M&Ms indicated that the hosting venue had not understood all details of the contract and had very likely made a mistake in configuring the set. One mistake in this case could cause malfunctions during the show or even the death of a crew member.

As it turns out, signaling software projects isn’t that difficult. The 12 step Joel Test is a reasonable signal for software development companies. While the Joel Test is nice for getting a feel for a company before you work for them, the concept is still useful once you’ve got the job and the project is in full swing.

Ultimately signals, also known as tripwires or triggers, are really just binary metrics for uncovering potential problems your project might be facing before the problems explode in your hands. When some condition is met (the signal), you know it has specific significance and prompts certain actions to prevent a problem from occurring. Triggers are most often used with risk management but their use should not be exclusive to that practice. In fact, if you’re collecting real data, you have even more opportunities for identifying signals outside of risk management.

On past projects I’ve used signals for a variety of issues. Here are some examples.

• During the past 3 iterations the team identified between 15 and 20 defects. I expect a similar number of defects to be detected for this iteration. If more defects are detected, there may be a disconnection in understanding between requirements, design, and implementation. If fewer defects are detected, tests may not have been as rigorously defined as they should have been.
• A Fagan inspection completed in less than one hour with a rate of 400 LOC/hour. Since most inspections have covered only 250 LOC/hour it is likely that this inspection was not effective and the results not reliable since the inspection team sped through the code.
• When evaluating potential open source libraries, Source Forge projects without a website shows a general lack of dedication to the project and indicates that the software is probably of poor quality or ill-maintained; the library is worth neither the time nor effort to use.
• Tasks that have been estimated to require longer than 9 hours have probably not been thoroughly thought through.
• No risks have been identified for this project or risks have not been updated for several iterations. This implies that the team doesn’t have a realistic understanding of what problems the project faces.

In each of these examples, when the signal is heard, I knew there was going to be a problem on the project.

Work with your team to establish signals for your project. The best part is that once you’ve decided on the signals for your team, when triggers are tripped you can throw a Van Halen sized rock star fits in your cubicle! Well, try to resist throwing your monitor out the window anyway.

Software developers are, in their heart of hearts, dataphiles – people who are absolutely in love with data. When was the last time you had a passionate discussion about frame rates, hardware benchmarks, gadget specs, sports statistics, dungeons and dragons, the merits of high def…the list goes on. Face it, you love data. You love comparing things using data. You don’t feel comfortable making decisions without a comprehensive comparison of data.

Why then do most software developers treat software development differently?

Tom DeMarco recently brought his own famous quote into question (pdf), musing that not only is it possible to control what you can’t measure, but the most important stuff you need to control on a software project is impossible to measure. Once again, DeMarco is wrong (in my opinion anyway).

Wikipedia is one of the most controlled projects on the planet

One the surface, Wikipedia is the Wild West of online content. Not only can anyone edit any page, but content from Wikipedia is widely proliferated in the media and (sadly) school reports. Wikipedia is the single greatest success of user generated content in the history of mankind (”The Internet,” as the medium, doesn’t count). What started with a dozen humble articles has evolved into the most comprehensive encyclopedia ever created and includes everything from the fundamentals of science to the definitive source on Babylon 5.

What folks seem to forget is that even in the Wild West, there were laws and there were lawmen. Though we love to think romantically about such brigands and gunslingers as Jesse James, Billy the Kid, and Butch Cassidy, most stories about these historic figures are greatly exaggerated. So too is the case with Wikipedia.

Let’s take a closer look at the Wikipedia entry for Billy the Kid. This article belongs to a number of internal WikiProjects, visible from the top of the article’s talk page. The WikiProject Biography is not unlike most projects in Wikipedia. There defined processes for assessing articles, conducting peer reviews. There are rubrics defined for assessing the quality of articles within the project. People even take on specific roles and responsibilities within the project. The collection of processes and information serves as the main means of coordination for content contributors and helps the group control articles within the scope of the project.

The WikiProject Biography even collects metrics on articles which it then uses to make decisions concerning the articles under the project. The metrics are derived from quantifiable data and help control the project.

As it turns out, Wikipedia is not the lawless territory of the internet it has been made out to be.

You can measure the immeasurable

Wikipedia works because people were able to figure out ways to measure things that usually can’t be measured. The fundamental principle that many people overlook is that binary is a metric too. Yes or no questions can be just as effective a measure as any complex metric. Did everyone fill out their task data today? Yes or no. Did the estimate match the actual? Yes or no. Did the test pass? Yes or no. Is the project done? Yes or no. Have we identified risks? Yes or no. Has this risk become a problem? Yes or no.

At the heart of every complicated metric is really a series of yes or no, binary questions. When considering whether the project is done, you have to define done. One way of defining done is in terms of a checklist. Is feature 1 done? Is feature 2 done? Defining done for a feature could be as simple as checking whether all the tests have passed for the feature, again a binary measure.

For more subjective assessments, you can rely on observation-based, experience-defined rubrics. Does the team get along with one another? In the simplest form, this could be a binary metric (Am I friends with everyone on the team?) but it could also be more complicated relying on gut feelings and a guiding rubric (”we never hang out together and don’t trust one another” might indicate low harmony while “we hang out often and feel comfortable sharing personal stories” could indicate high harmony). Teachers use rubrics and experience to judge subjective assignments everyday. The difference is that they slap a grade on it and send it home as a report card.

While DeMarco is correct that many of most critical things in a project are the most difficult to measure, it is possible to create measurements if you feel it is important enough to do so. How would you assess whether you have a good architecture that solves the problem at hand? Rubrics might play a part but so too might binary gates based on quality attribute scenarios or intricate observations concerning design trends over time. If you think hard enough, you’ll find that it’s extremely easy to find measuring points for nearly every aspect of a software project.

Whatever you do, don’t become a mindless, data-driven robot

I love data and I know you do to. While it’s tempting to inject data collection and derive metrics for every aspect of a project (because it’s fun and informative!) don’t. Collecting data and calculating metrics can be expensive. Not so expensive that you shouldn’t use it, but expensive enough so that you shouldn’t use it on everything. I like to compare using metrics to eating out at restaurants. Once or twice a week isn’t that big a deal, but it’s not something you should do every day if you’re trying to watch your budget.

DeMarco is right about one thing: control is not the end-all-be-all of software engineering. Consider carefully, what are the most risky parts of my project? What are the parts of my project that even require control? What are the parts in which I need more insight or want to improve? Strategically develop metrics for these areas and don’t worry about measuring the rest. Trust me, the world won’t end. If you don’t know what you’re doing, start with a simple binary measure. And above all, if something isn’t working, change it.