As I’m writing this I am making final preparations to leave for Trondheim, Norway to present an experience report at the 11th International Conference on Agile Software Development and Extreme Programming, or XP2010. After my experience report was accepted, the conference organizers opened up a second round for lightning talk proposals, and in a moment of whimsy I decided to propose another talk. That was accepted as well. So I’ll be giving two talks at XP210.

If you’re at the conference I would very much like to meet you! This is my first time presenting at a conference and only my second conference attended (my first was OOPSLA this past October) so I’m a little nervous and unsure what to expect. Above all I hope that I’m able to share some information that is useful, interesting, and inspirational. I think I have some interesting insights and perspectives that can help make the way we build software just a little bit better and I want to hear your opinions, your thoughts and experiences too. If you can’t make it to my talks, come find me, I’d be glad to discuss any ideas with you. I plan to have a copy of the paper and slide decks available on this website along with a brief synopsis of the talks shortly after the conference. And of course I’ll be looking for people to eat meals with and hang out while I’m here on Wednesday and Thursday. I’m here to present, to learn, and to discuss cool ideas with other practitioners.

Here’s a summary of the two talks I’m giving along with some background information. The background isn’t necessary but just interesting information and context related to the talks.

Put it to the Test: Using Lightweight Experiments to Drive Process Improvement

This experience report tells the story of how my team ran a lightweight experiment to figure out whether we should use pair programming or program alone and review our code with Fagan inspection. With only a few hours of work and only a few weeks time we discovered that pair programming was instrumental to our eventual success.

In this talk I will discuss what we learned about setting up and running the experiment so you can run lightweight experiments of your own on whatever topics your team finds most interesting or pressing. Experimentation doesn’t have to be this overbearing, lofty, academic thing that it has become. My hope is that teams around the world will use this technique to discover just a little more about how software engineering works and that they’ll share what they learn in white papers, blog posts, and future experience reports. We can help close the gap between research and industry with just a sprinkling of scientific thinking.

Background Information

• Paper and talk abstract
• Pair programming experiment work plan as executed by the team (archived)
• Square Root’s team reflection blog
• A brief analysis of why pair programming seemed controversial to the team while we were planning.
• Complete Square Root project archive (You’ll find data from the experiments along with tons of context information on the project. This is a very raw archive, but there’s tons of great data if you’re willing to root around.)

Lessons from a Software Engineering Dojo: The MSE at Carnegie Mellon University

At OOPSLA in Orlando, Florida this past October I heard proclamations like this more than a few times: “The only way to teach software engineering is through experience. What we really need is a software engineering program that uses a capstone project, a non-trivial, long term project that lets you practice what you learn. No university programs currently have such a project in their curriculum.” I agree. In fact, I agree so much that I attended the only university in the world in which a long-term, realistic (in both scope and complexity), team-based capstone project is an essential part of the software engineering curriculum.

I have two goals with this lightning talk. First, I aim to spread the word about the existence of the Master of Software Engineering program at Carnegie Mellon University. Carnegie Mellon is on the forefront of software engineering education research and the Master of Software Engineering program has been teaching professional software engineers to become true masters in the field for over 20 years. Second, since the studio component (the capstone project) of the MSE degree is so similar to an industrial setting, there’s a lot industry teams can use for training and educating software engineers as you work. There are tons of lessons that can be taken from the MSE in the form of both research and battle tested experiences. This second goal will be the greater emphasis of the talk.

Background Information

• Lightning talk abstract
• The 9 crafts of software engineering proposed by Alistair Cockburn
• Master of Software Engineering homepage
• Wikipedia entry about the MSE program
• MSE studio projects archive

Most software processes are light in two areas: quality assurance and process improvement. Most processes prescribe specific techniques for ensuring the production of quality code. XP for example advocates unit testing with TDD, continuous integration with smoke tests, pair programming, and acceptance tests written by the customer (using something like FIT so you can run automated regressions). Process improvement in XP is accomplished in a ’round the campfire, Kumbaya singing, get in touch with your feelings brainstorming session. Assumptions abound and there is no systematic way of ensuring that either testing or process improvement is handled adequately.  As we know, assumptions are never good enough.

The biggest assumption in XP (and indeed most software processes) is about bug tracking. Common sense dictates that you will create some kind of bug database. Hopefully it will at least be some kind of third party bug tracker such as Bugzilla or Bug Genie. Excel will work in a pinch but quickly becomes unsuitable for teams larger than one developer.  But how does the bug tracking actually work? What bugs get reported? Will you record issues from inspections in your bug tracker or only “true” bugs? What is the process for fixing a bug? What is the process for closing a bug? Who has access to the bug tracker? What information is required in your bug database and what information is optional? How do you determine defect priorities? Or the severity of bugs?

The majority of software processes provide answers for almost none of these questions. You are largely on your own to make up whatever you think makes the most sense for your development environment based on the best practices for your software process and your understanding of “good” software quality assurance practices.

No matter what quality process you follow, you will need a defect control philosophy. Once again, in absence of guidance I turn to the Team Software Process, one of the few processes to define what it means to track defects. In the Team Software Process, defects are treated as blight, a horrific mistake injected through the ineptitude of a developer. To remove these blights, the TSP relies on a series of filters in the form of code reviews, code inspections, unit tests, function tests, and so on. XP has a similar, though somewhat less rigorous set of filters in the form of pair programming, TDD, continuous integration, and acceptance tests. Each filter is meant to remove more and more defects, until finally “all” defects are removed from the system by the time the software has passed through all the filters. Generally each filter is intended to remove different types of defects, though it is conceivable to capture escaped defects from a previous filter in a later filter.

Just as water passing through layers of sand and rock will remove debris, so too will code passing through layers of unit tests and inspections sift out injected defects.

Defect Data

With these ideas in mind, bug tracking has three basic goals.

1. Record defects so they can be analyzed and fixed.
2. Identify the means by which defects are injected.
3. Identify the means by which defects are removed.

We write down bugs so we can go back and fix them later. This is obvious. But bug data can be used to measure process improvement also.

Since many parts of a software process are dedicated to filtering out the defects we’ve injected, understanding how defects are injected is essential to preventing similar defects from being seen in the future. The idea is that we want to learn from our mistakes. To achieve this, record the type of defect and the reason it was injected. The TSP gives us a good starting point for each of these, shown in the tables below. You should feel free to modify the types and reasons so they make sense for you, your team, and your project.

The defect type characterizes what kind of defect is injected and captures the essence of what is needed to fix the defect.

The defect’s reason characterizes why the defect was inject.

Assigning the reason can be a little tricky. I have found it is best if the person who injected the bug assigns the reason. On small teams (or if you’re following the PSP), this is generally pretty easy. It’s not about rubbing their nose in the problem – well, actually it is. Ok, it’s not about embarrassing or punishing the person but creating an opportunity for learning from our mistakes. If you can understand the reason why a defect was injected, it’s possible to prevent the defect from occurring again. For example, if there seems to be a rash of education related defects in a particular module, perhaps some training is in order.

Ideally we’d also like to know when the defect was injected, as in at what phase of development. It is possible to realize this information with additional analysis but I’ve found the return to be rather diminutive. Basically, you’ll learn what we’ve known all along – that the longer a defect is in the system, the harder it is to get out and that the most expensive defects are injected during the earlier phases of development (e.g. design defects are costly). Rather than track when things were injected I think it makes more sense to track when defects are detected. The point is to gain an understanding of how well the quality process filters out defects. To accomplish this, simply write down what you were doing when you found the bug. If you’re using XP, the list might include designing, writing new code (in a pair), writing new code (alone), refactoring, unit testing, integration, and acceptance testing. With this information you should be able to determine how effective each activity filters defects and over time whether the quality process is having issues. For example, I would expect interface defects to be detected during integration. If they are being detected earlier, say during unit testing, or later, say during acceptance testing, then my continuous integration and smoke test suite might not be as robust as it should be.

Better Bug Tracking

The strategies I’ve outlined here are a little more sophisticated than your average bug tracker, but add a lot of punch for very little effort. Tracking defect type, reason injected, and phase detected allow you to get a better handle not only on how defects are being injected into the software, but also how they are being detected. Both these chunks of information are necessary for understanding how defects are making their way into the system and how your process is helping you ferret them out of your system.