I used to think hacking was bad. It was something you did when you didn’t have a plan, when you didn’t know what you were doing; it’s what amateurs do, noodling around the code without clear direction or intent. Or it was something you did, quick and dirty, just to get something out the door. Hacking is the best, surefire way to waste time and create mountains of unmaintainable code you’ll have to rewrite anyway. I’ve always struggled with this since I feel so drawn to hacking culture – DIY, playing with things just to see how they work, ignoring the instruction manual, reveling in the aesthetic of a job well done, making what you can within limited constraints – and many of the brightest minds in software development speak quite passionately about the merits of hacking.

Thinking about this for a while, I’ve come to the conclusion that there’s nothing wrong with hacking. Quite the contrary, hacking could be one of the most positive things a team can do. In fact, many of the agile processes and techniques that have worked for me in the past seem to be designed to cultivate teams of hackers.

It seems I had fallen for some of the myths about hackers.

Myth: Hackers don’t have a plan

Hackers do have a plan but they acknowledge that the plan will change and then plan with this in mind. This means creating plans only to the level of detail to which you are currently comfortable making plans. For example we all know that making a detailed plan at the start of a project, down to the code level, for all but the tiniest programs, is silly since the plan is mere conjecture of a positive outcome. Rather than writing a fantasy on paper, hackers lay plans based on the current knowledge. These plans then expose areas for further exploration (hacking!), the resultant knowledge of which can be turned into more detailed plans.

In XP these detailed plans are created just in time during the iteration planning game, sometimes even later than that. That’s not to say that there wasn’t a plan before the iteration began. Not so. A great degree of planning, communication, and coordination was required to be able to plan so adeptly, so quickly.

Myth: Hackers don’t know what they’re doing

In the literal sense, a lot of times hackers don’t know what they are doing. That is why they are hacking. The purpose is to figure out how something works or the best approach for something. And that’s the key. At a meta-level, hackers know exactly what they are doing – they are searching for information, for knowledge, for a means to achieve an end.

Action research is a relatively new-to-software-engineering research method that seemed to take agile researchers by storm at XP2010. The basic premise is that a researcher works closely with a consenting organization to observe practices in software engineering. Rather than acting as a passive observer, as is traditionally done, the researcher injects practices directly into the team and observes before and after reactions. For example, in recent work by Tor Erlend Fægri, having observed issues with team collaboration he suggested several techniques to help solve the problem. Some suggestions worked, some didn’t. Fægri was able to take the time to understand what happened and why because he was dedicated to understanding almost as a participant, watching the events unfold. This take on research makes me think of Jane Goodall’s work with chimpanzees in which she directly interacted with wild primates in order to better observe their behavior.

When I first heard about action research it sounded a lot to me like hacking – trying things out to understand how something works.

Myth: Hackers don’t create value

Yes, the code that comes out of a hard core hacking session might be a big pile of spaghetti, but consider the purpose of this work in context. Another perspective is that hacking may incur technical debt which sooner or later will need to be paid off. But it’s important to keep in mind that not all technical debt is bad. When you are hacking, you are either in learning mode or in finish mode. Learning mode is never wasted effort. Finish mode, where you are just trying to get something finished so it can be used, can leave scars on your code, but the immediate value provided to customers can be enormous. When you have a desperate and immediate need, anything that works today is better than something perfect that works next week.

The same goes for architecture. Of course, many of the architecture researchers use words and phrases like “experiments” or “trade-off analysis” to describe this same act of quickly creating something for the sake of knowledge. That’s really just hacking too.

Sometimes the artifacts you produce are throwaway. Sometimes the working solution is good enough and you’ll never have to worry about it again. Sometimes you’ll have to pay off the technical debt. But hacking for the right reasons is never wasted effort.

Agile Development Helps Cultivate Good Hackers

Agile software development, be it controlled through time boxes or flow, is all about ensuring that work doesn’t teeter out of control a day at a time. The focus on providing customers with value creates an environment that encourages action and creation. The agile manifesto gently nudges hacking behaviors. Frequent releases of working software, welcoming changing requirements, focus on simplicity and continuous attention to technical excellence, helping the customer understand what they need and then doing your best to make it for them. Even the idea of continuous process improvement is about hacking your team to make yourselves better. Hacking is the very lifeblood of agile software development.

Hacking isn’t a dirty word. Hacking could be the most important thing you do all day.

[This post is a recap on the second talk I gave at XP2010. This was the big one, the experience report talk, one of 15 experience reports published at XP2010. You can download the slides (pdf) or the full paper (pdf) from this website or from XP2010.org.]

Process improvement is important for nearly all teams but it can sometimes be difficult for a team to know what is working, what isn’t working, and what to techniques or methods to try when attempting to improvement. Performing a scientific experiment is one way help overcome these problems but as academic research has shown us, while experimentation can yield interesting results, running an experiment is time consuming, expensive, and requires some serious thinking and control to pull off. From a practitioner’s standpoint this means that experimentation is a non-starter.

Of course, that’s only if you run experiments like an academic.

Back Story

Just over a year ago, my MSE studio team at Carnegie Mellon had a problem. We had decided we would use Extreme Programming for the construction phase of our project but some team members had doubts concerning pair programming. We had decided that we would use some kind of peer review, having already seen the many benefits of inspection when reviewing other artifacts. The dispute arose over whether pair programming would give similar enough results. Also, not all team members had experience with pair programming but everyone on the team knew and enjoyed solo programming.

The number one concern was whether pair programming would allow us to meet our very strict deadline. We had just over three months to complete the construction phase of the project. According to our threshold of success this meant implementing all “must have” requirements with a minimum level of quality. Did we really have time to waste by having two people working on the same code at the same time? Wouldn’t working independently and inspecting code on an as needed basis allow us to get more work done faster?

At the time it just so happened that I was taking a reading class with Mary Shaw and in that class we discussed some research findings that might help settle this debate. Research from Laurie Williams, Ward Cunningham, Barry Boehm, and many others showed that pair programming requires more effort (but never double the effort) but is faster than programming alone (pdf). Also pair programming creates code of about the same quality as coding alone with inspection (pdf). Of course, the research may not apply to us since Square Root is closer to a professional team working on a large project with a real client, not undergrads working on short term toy projects.

After an iteration where some teammates used pair programming and others refused, we decided to try an experiment to see which practice actually worked better. The original idea was that we might be able to validate some of the research but decided instead that it was more important just to resolve our own internal conflicts and figure out which processes worked better.

Conducting a Lightweight Experiment

With the scientific method as our guide we planned and executed a lightweight experiment which pitted programming alone against pair programming. The results were amazing (and you can find the raw data in our project archive). In conducting the experiment we used a set of novel techniques which I think can be useful in conducting other lightweight experiments. There’s more background in the experience report so I’m only putting the meaty stuff in this post.

Focus narrowly on a single question – The essential key to keeping an experiment light is to only tackle one thing at a time. In this case we focused on comparing and contrasting a single technique, pair programming, rather than multiple techniques or an entire process (such XP vs. TSP).

Divide work, not teams – If I were comparing pair programming to programming alone in an academic setting, I would put together two teams of about the same experience and have them each build their own version of the same software, one team using pair programming, the other programming alone. In a business setting this is a complete waste and few companies can afford to have two teams duplicating effort. By dividing work instead of teams you may lose some control over variables in the experiment but in most cases isolating more variables doesn’t add any further clarity to helping answer the narrowly focused question. To divide work successfully you need to have some way of estimating work units for division. We used use case points as shown in the figure depicting our modified planning game.

Continue making releases – Since we still needed to make a comparison, rather than dividing into teams and duplicating effort we divided the features that were released each iteration. In this way we built about half the features released during an iteration using each technique. Working on about half the features using pair programming meant that at least some features were being built by individuals. At the time this was a risk reduction decision to make sure that if pair programming completely failed we’d still have something to ship at the end of the iteration. Explicitly managing risks is the only way to know if the lightweight experiment may cause problems for making releases. Also, we had a strictly defined cut-off for stopping the experiment if it ever stopped us from shipping to our client.

Use the data you have – In almost all cases we were able to get the data we needed to evaluate our hypothesis from our current process. When we couldn’t, we only had to make minor modifications to our data collection practices, for example adding a check box to our SharePoint server for indicating whether a task was paired or individual.

One of the more interesting things we did was to create a “tally sheet” for collecting pair programming issue detection statistics in real time, as the issues were discovered. Given the near instantaneous code-inspect-fix cycle when programming in pairs, this was the only way to collect similar data for comparing pair programming to inspection.

Statistical significance is overrated – The whole point of running a lightweight experiment is to collect just enough data to help you make a better decision or validate your gut feeling. This technique is not meant for uncovering universal truths or proving something to the rest of the world. In exchange for keeping the experiment light, the results will only apply to your team. Over the course of an iteration or two, 4-6 weeks, you’ll only get enough data to start to see trends. In our case the results were not statistically significant using individual T-tests but that didn’t matter. The most important thing is that we had data that could be used for comparison, data that everyone felt good about and that helped us gain clarity into what we did and how well it worked.

Retrospectives get immediate value – The whole reason the experiment is light is to reduce cost and decrease the lag time to providing value to the team. Just to give you a little perspective, it took us 6 weeks to run the experiment and had enough data and casual observations to make a decision during the retrospective when the analyzed data was shared. That event occurred in early August of 2009. This experience report required almost nine full months of gestation from the paper proposal to the talk I gave at the conference. The gestation period on “universal truth” research can be even longer. We, as practitioners, don’t have to wait for those universal truths to be born to get value from research. By running your own quick and dirty, lightweight experiments, you can get results in a timely fashion that you know will apply to your team because your team was the subject of the experiment. It’s all about closing the gaps between research and practice and taking the information you need now instead of waiting for academic research to catch up.

Overall Conclusions

For the Square Root team it turned out that pair programming was faster, cheaper, and produced code that had more predictable albeit slightly worse quality. The more important lesson is that we discovered a technique, lightweight experimentation, for learning other interesting things about our team and about software engineering in general.

My paper and this blog post were all about trying to describe the technique, using our experiment as an example. I think it would be awesome if teams around the world conducted lightweight experiments on a variety of topics. If enough folks share what they learn, we might start to see trends emerge across teams that could lead to universal truths, validate research, or at least discover some great rules of thumb.

What else might make for a great experiment? Anything you’ve got a question about on your team!

• What is the clearer way to write requirements, user stories or use cases?
• Which estimation technique is more accurate of X and Y?
• Can we skip unit testing if we use inspection (looking at quality, knowledge sharing)?
• Is UML a better design notation than the one we made up as a team?
• What else…?

If you do a lightweight experiment, let me know! Share what you learn as a blog post or whitepaper. Let others know what you’ve learned! Even if the specific results only apply to your team and the way you’ve executed your project, your experiences help form a baseline, a sort of shared understanding for how software development works, how some of these practices work. And there’s so much about software engineering that we have yet to learn.

Acknowledgements

This paper was my first experience report and it was an awesome journey. Naturally a lot of folks helped me along the way and I would like to take a moment to make sure they know that I appreciate their influences and support. The Square Root team: Marco Len, Yi-Ru Liao, Abin Shahab, and especially my fellow experiment co-champion Sneader Sequeira for having the guts to go along with this idea in the first place. Some of the faculty at Carnegie Mellon: Dave Root and John Robert (my studio mentors) for bringing up the idea of writing a paper, and Jonathan Aldrich for helping review my proposal. Artem Marchenko was my XP2010 paper shepherd after the proposal was accepted, and the quality of each draft only improved because of his inputs. A group of my fellow employees at Net Health Systems sat through an early draft of the presentation I gave and shared valuable feedback for improving it. And finally I thank, Marie, my wife, who was with me from start to finish and read more drafts and sat through more practice talks than anyone else. She’s probably as much an expert on this subject by now as I.

A Final Aside

I wrote the initial draft of this paper as my final reflection paper for my Master of Software Engineering degree (pdf). That draft has a very different tone, approach, conclusion, and direction than what I eventually published for XP2010. This is half due to there not being a hard page limit but also I had a lot more time to think about what was really important when writing for XP2010. There’s some interesting information, mostly in the lessons learned, that might prove interesting to those who are interested. You should check out my Square Root teammates’ reflection papers as well since they are all interesting and well written.

As I’m writing this I am making final preparations to leave for Trondheim, Norway to present an experience report at the 11th International Conference on Agile Software Development and Extreme Programming, or XP2010. After my experience report was accepted, the conference organizers opened up a second round for lightning talk proposals, and in a moment of whimsy I decided to propose another talk. That was accepted as well. So I’ll be giving two talks at XP210.

If you’re at the conference I would very much like to meet you! This is my first time presenting at a conference and only my second conference attended (my first was OOPSLA this past October) so I’m a little nervous and unsure what to expect. Above all I hope that I’m able to share some information that is useful, interesting, and inspirational. I think I have some interesting insights and perspectives that can help make the way we build software just a little bit better and I want to hear your opinions, your thoughts and experiences too. If you can’t make it to my talks, come find me, I’d be glad to discuss any ideas with you. I plan to have a copy of the paper and slide decks available on this website along with a brief synopsis of the talks shortly after the conference. And of course I’ll be looking for people to eat meals with and hang out while I’m here on Wednesday and Thursday. I’m here to present, to learn, and to discuss cool ideas with other practitioners.

Here’s a summary of the two talks I’m giving along with some background information. The background isn’t necessary but just interesting information and context related to the talks.

Put it to the Test: Using Lightweight Experiments to Drive Process Improvement

This experience report tells the story of how my team ran a lightweight experiment to figure out whether we should use pair programming or program alone and review our code with Fagan inspection. With only a few hours of work and only a few weeks time we discovered that pair programming was instrumental to our eventual success.

In this talk I will discuss what we learned about setting up and running the experiment so you can run lightweight experiments of your own on whatever topics your team finds most interesting or pressing. Experimentation doesn’t have to be this overbearing, lofty, academic thing that it has become. My hope is that teams around the world will use this technique to discover just a little more about how software engineering works and that they’ll share what they learn in white papers, blog posts, and future experience reports. We can help close the gap between research and industry with just a sprinkling of scientific thinking.

Background Information

• Paper and talk abstract
• Pair programming experiment work plan as executed by the team (archived)
• Square Root’s team reflection blog
• A brief analysis of why pair programming seemed controversial to the team while we were planning.
• Complete Square Root project archive (You’ll find data from the experiments along with tons of context information on the project. This is a very raw archive, but there’s tons of great data if you’re willing to root around.)

Lessons from a Software Engineering Dojo: The MSE at Carnegie Mellon University

At OOPSLA in Orlando, Florida this past October I heard proclamations like this more than a few times: “The only way to teach software engineering is through experience. What we really need is a software engineering program that uses a capstone project, a non-trivial, long term project that lets you practice what you learn. No university programs currently have such a project in their curriculum.” I agree. In fact, I agree so much that I attended the only university in the world in which a long-term, realistic (in both scope and complexity), team-based capstone project is an essential part of the software engineering curriculum.

I have two goals with this lightning talk. First, I aim to spread the word about the existence of the Master of Software Engineering program at Carnegie Mellon University. Carnegie Mellon is on the forefront of software engineering education research and the Master of Software Engineering program has been teaching professional software engineers to become true masters in the field for over 20 years. Second, since the studio component (the capstone project) of the MSE degree is so similar to an industrial setting, there’s a lot industry teams can use for training and educating software engineers as you work. There are tons of lessons that can be taken from the MSE in the form of both research and battle tested experiences. This second goal will be the greater emphasis of the talk.

Background Information

• Lightning talk abstract
• The 9 crafts of software engineering proposed by Alistair Cockburn
• Master of Software Engineering homepage
• Wikipedia entry about the MSE program
• MSE studio projects archive

“Better is the enemy of good enough” is a phrase often held up as the reason for not making changes on a team. If everything seems “good enough,” the effort to make something better is regarded as waste. A lot of times, “good enough” is defied in terms of “providing value to the customer,” often stated as the “shipping working software” metric. So if you are shipping working software and receiving generally positive feedback from your customers, then what you’re doing is good enough and there is no need to do things differently.

But if good enough is really all you need, why is it so dissatisfying?

I’ve mentioned before that it’s a good idea to have a project threshold of success, a set of minimum goals that must be completed for a project to be considered successful. Failure to meet all goals in the threshold of success means you’ve failed the project. So while you will succeed if you meet your threshold goals, only meeting the goals means you’ve done the absolute minimum amount of work necessary to be successful. Satisfying the threshold of success for a project is like getting a C in school. It’s certainly good enough, but it isn’t exactly awesome.

While “good enough” is perfectly acceptable (You didn’t fail!), it always feels nice to achieve more. It feels good when I’m able to gently exceed my client’s expectations. It also feels good to do things right and not merely get things done. This is why merely shipping software, the minimum requirement for succeeding on a project, is not enough for me.

Equally important to me is how the work is done, not just that the work gets done. Is overtime or heroic effort necessary? Are we reflecting on our work and attempting to improve? Does my team work together well, take risks, and innovate in many different areas of the project? Do we try to use data to understand what is happening in the project? Are teammates given the support they need to grow as professionals? Am I having fun, looking forward to work every day, and happy with my contributions on the team? Is the software something I am proud of and actually useful to people? Is the product well supported by documentation? Is my code beautiful and maintainable? And most importantly, did I either learn something new or achieve something great while working on this project?

This is actually a lesson in understanding what “good enough” really means and why tools such as Threshold of Success are so important. Only when everyone agrees at a conceptual level that what the team is doing is “good enough” can everyone on the team move forward and be happy. Sometimes this will mean getting working software out the door no matter the cost. For most teams “good enough” will be a mix between a working software product, a happy and healthy team, and laying foundation for future work.

Making improvements in how my team operates or to the software itself just feels good. It’s fun. So while “better is the enemy of good enough,” avoiding change is paramount to avoiding the very things that makes me happy. The trick is making sure that I’m not making changes just for the sake of change. Sound engineering and a good understanding of the project’s threshold of success can help to avoid this fate. Once you’ve met your threshold, earn some extra credit by improving your code base or making process improvements. Or, better still, choose a threshold of success which requires you to do just a little more than the bare minimum. Because no one should settle for good enough when awesome is within reach.

This post is a recap of a talk I gave this weekend at the Carnegie Mellon University Master of Software Engineering 20th Anniversary Mini-Conference. I’ve made the paper this talk is based on (pdf) as well as the slides I used during the talk (pdf) available. I’ve also linked to as many of the primary sources I used in research as I could so please, check out those papers if this is something that interests you.

About a year ago I discussed the idea of using affordances to help figure out how to make software processes work more smoothly for a team. Back then the idea spawned from a moment of crisis and self-reflection on my studio team, but having thought about it for a while and noticing the phenomena occurring on other teams I decided to revisit this idea and see if there is a way to proactively use affordances to avert problems rather than merely explaining problems as they occur. As it turns out there is a precedent for affordance-driven design in object engineering. With a few basic assumptions I think affordance-driven design can be extended to software processes as well.

To better explain how to proactively use affordance to pick or tailor a software process it helps to know a little about the Theory of Affordances (pdf). If you’ve read The Design of Everyday Things by Donald Norman then you probably already have a pretty good understanding of how to use affordances in the context of object design and usability. From an ecological psychology perspective, affordances are a way of helping explain or predict how an animal will behave in the context of their environment. In the context of humans, our environment is influenced by our experience, culture, and background as well as our current goals within that environment.

A simple example is the play button on a DVD player. A triangle to the right means play. This is obvious to us because we know what a DVD player does (we have experience with similar devices), we turn to the device when we want to watch movies (we’re looking for the play button), and culturally, our notion of time is left to right (so a triangle pointing right means play while a triangle facing left means reverse). But what if you come from a culture where time is generally represented as passing from down to up vice left to right? In this case, a triangle facing up might be a better symbol on a play button than a triangle pointing right. The value of the affordance changed based on a cultural bias and the user’s background.

This is all well and good, but what does it have to do with software process? For the Theory of Affordances to apply to software processes (or any process) we have to assume that process is a part of the environment. This is an interesting proposition since process really only exists in our minds. Process is something that we make up, and like our understanding of an object your knowledge and experience with a process will influence your perception of that process as an environmental influence. As long as you believe in, understand, and follow a process, the steps of that process exist as much as any other object in the real world. Logically this seems to make sense. Even the US patent office will grant patents for a process just as it will a physical invention.

Back to the core problem of identifying affordances, something I did not have an answer for in my original post a year ago. As best as I can tell, the only way to identify affordances is to reverse engineer a process focusing on the affordances using a technique known as Affordance Driven Design. Affordance Driven Design has three basic steps (pdf).

1. Identify a user’s needs in terms of functions.
2. Identify the desired functional affordances necessary to achieve the previously identified user’s functions.
3. Choose affordances to design into artifacts which are mostly likely to help achieve those desired functional affordances.

As an example, consider the task of blending a drink using a blender (pdf). Typical functions a user might want to perform are preparing the blender, blending, and cleaning up afterward. Functional affordances might include the “countertop-ability” (the ease with which a blender can be moved to a countertop), the “clean-ability” (the ease with which a user can clean a blender), and “transportability” (the ease with which a user can move a blender around). A person might choose any number of blenders to perform the desired functions but each blender will fulfill the functional affordances in different ways. A hand blender is extremely portable while a gas-powered “whacker” (powered by a 26cc engine, complete with motorcycle throttle – it makes the smoothest margarita you’ll eve have) isn’t really intended for indoor use.

To software engineers, functional affordances should look very familiar – they’re essentially quality attributes. Thinking about affordances from this perspective gives us a huge advantage since, as software engineers, we are already extremely familiar with quality attributes and quality attributes scenarios. Affordances, therefore either promote or inhibit desired quality attributes in your process.

Thinking about software processes, the functions will all be related to software: writing, designing, testing, and releasing are just a few possibilities. Some process quality attributes might include:

• Plan-ability (How far ahead does a process help you to plan?)
• Predictability (How well can you see into the future?)
• Changeability (When the course of a project needs to shift, how well does the process support chaging plans or direction?)
• Quality (The degree to which your process promotes “quality”)
• Cost (The amount of resources you’re willing to spend on process to achieve specific functions)
• Harmony (How well the team gets along)
• Reliability (How consistently the process helps you perform)
• Performance (Could be speed of development or quantity of code – define what you mean in the quality attribute scenario.)

As an example, say changeability is a desired quality on your team. A specific changeability scenario might go something like this. In order to meet business needs in an aggressive market, the team needs to be able to shift focus and answer competitors’ challenges within five business days. What are the things that might get in the way of this kind of rapid change? Heavy documentation could be one thing, as it nudges teams into keeping a single course. Long iterations also make it difficult to shift focus since more effort is required to make longer term plans. Going light and having short iterations, on the other hand promotes a team’s ability to change. But like every design decision there are trade-offs. Going light in terms of documentation might make it harder to achieve certain kinds of quality, for example.

The main idea here is that it’s relatively easy to identify process affordances by thinking like a designer and applying the skills we’ve already acquired as software engineers. I propose that evaluating process affordances as I’ve discussed here is a great way to pick a process and also to tailor processes. When tailoring simply identify affordances that are helping the team (be sure to keep those), and identify the affordances that are nudging the team in the wrong direction (replace those with affordances that help you do the right things). And above all, remember that if things are going wrong, it isn’t always your fault. The process is a part of the environment and if your process is giving you the wrong cues for your project or team, then it’s the wrong process for you. So change it!

I’ve had to help more than a few teams get their version control systems sorted out over the past few years, and so I thought it would be easier if I just wrote down the philosophies I use for initializing a repository and getting the whole system set up. If you’re looking for some specific advice on how to set up and use a specific version control system, the Pragmatic Starter Kit Series for CVS, Subversion, or Git is a great place to start.

What should go into a source code repository?

The short answer: the repository should contain everything necessary to perform a clean build of your system. In most cases, this includes the code, third-party binaries necessary for building, tests, and documentation.

It’s ok to assume that everyone has their build environment “properly configured” for building. To make sure, make a list of everything that must be setup in the environment to build the software and put it on the team wiki. These things don’t need to be stored in the repository but you should at least write down what the standard build environment is supposed to look like. Depending on what the required software is, it might also be a good idea to keep a copy of it, just in case something happens to vendor in the future. The last thing you want is for a vendor to stop supporting the version of something you need, forcing you to upgrade because your hard drive crashed and you had to setup a new environment.

Include at least the following in your standard build environment list:

• Compiler versions
• Team sanctioned IDEs
• Required frameworks, toolkits, and build tools
• IDE extensions that the team has decided are so critical/awesome to the project they have to be used. Critical/awesome IDE extensions might enable a required tool-kit (such as GWT in Eclipse) or configure the IDE is specific ways (such as coding styles or static analysis settings)

Putting code and tests in the repository is fairly obvious, but third-party binaries (e.g. libraries) might not be. Put these in version control so that it’s easy to check out a project from source control and build without monkeying around with anything. I’ve found it best to create an “ext_lib” folder for storing all the external libraries. This way there is no confusion over what versions to use, and all the build paths can be set so that anyone can build just by checking out the code.

Here’s a real life example. Let’s say you’re writing a web application using the Google Web Toolkit and you rely on a caching library. The caching library should go into your ext_lib folder and you should tuck a zip of the GWT version you use away in a safe place just in case you need it later. Say your team is also using JUnit. Put the version you use in the ext_lib folder. This way everyone can build and use whatever GUI they want to run tests, be it the JUnit GUI or an Eclipse Plug-in.

Another real life example. Let’s say you use the excellent Sharp AutoUpdated component. Should you version the binary or the source? That was a trick question since it depends. The best answer is to only keep the binary of the library, but this isn’t always possible. One of the awesome things about open source software is that you have access to the source if you need it. So, let’s say you find a bug in the AutoUpdater and for some reason the maintainers aren’t responding quickly enough for your immediate needs. You can’t live with this bug so you have no choice but to fix it yourself. Congratulations, you just took ownership over your own fork of the AutoUpdater component. You now are responsible for maintaining the code – either in your version control library or in a public fork, and merging with the original code base may be more difficult in the future.

What doesn’t go into the source code repository?

Remember the DRY Principle for writing code (Don’t Repeat Yourself)? Well, that applies to your version control system too. Anything that can be derived shouldn’t be held under version control. Since your source code is already in the repository, storing the built binary is a violation of the DRY Principle. The penalty? Confusion, mistakes, and avoidable headaches. Third party libraries in the ext_lib folder don’t violate DRY since you can’t build them – you don’t own the source.

Also, do your fellow developers a favor and keep your personal stuff out of the repository. If you’re testing, nobody else wants to see your test reports. When you run the application, keep your logging messages to yourself. Also keep anything related to how you set up your personal environment in your personal environment. The last thing I want is to open up my IDE and see the last tabs you had open because you committed your personal user settings.

The easiest way to keep these sorts of undesirables out of the repository is by setting up an ignore list. Share it among the team.

How often do I commit?

Generally you should commit your changes anytime you think you’ve finished something useful that doesn’t introduce problems into the system. On the average, you should be committing changes at least once a day.

There’s two parts to this commit rule. “Finished something useful” might mean many things. This is by design. When you’ve finished a logical chunk of code that does something, feel free to commit it. “Doesn’t introduce problems” is a common courtesy to your fellow developers. Make sure, at a minimum, the system builds and passes any automated tests you have. And always update before you commit. Depending on your team size and how important the code is, you might establish a checklist for committing. Google has theirs automated. Every change the system has to build, pass tests, and pass a peer review before it can be committed.

Remember this mantra: Commit early, commit often.

But if everyone is committing all the time, isn’t that going to cause problems?

When you’re working with people and coordinating effort, problems will inevitably arise. Just remember, if you’re going to fail, fail early. It’s better to cause a conflict today through miscommunication while there’s plenty of time to fix it than the day before it’s time to deploy. Why? The conflicts will be smaller since you’re incrementally growing your code base. Also, since you made the changes recently they are fresh in your head and easier to work with. Code more than a week old might as well have been written by someone else.

Taking a risk management approach makes mitigating this easy. The risk: “Developers use a shared repository and commit changes frequently; might cause code conflicts that break the build.” The source of this risk is communication; therefore anything which helps facilitate communication can reduce the likelihood of this risk becoming a problem. Daily stand-up meetings are perfect for getting the word out about what everyone is working on. Automatically generated email updates from the version control system keep folks abreast throughout the day as changes are made. Continuous integration acts as a smoke test for uncovering integration problems while they’re small. Good merge tools can help reduce the impact of the consequence.

Once everyone gets used to the update-then-commit cycle, most of these problems go away. In my experience, big problems with code in the repository are usually a symptom of larger problems such as poor communication or failing processes.

What are some of your version control philosophies? What helps you keep things organized so you can get things done?

Everyone has a chore they hate doing. For me it’s cleaning the dishes. I’m a busy guy so I usually don’t get around to cooking and eating dinner until fairly late. Rather than cleaning anything, I stack the dishes in the sink and maybe soak a pan if something burned to the bottom. If the dishwasher has space, I’ll load it and set it going but most nights I leave a big pile of dirty dishes sitting around. After two or three nights of this, all the pots and pans are dirty and there’s no room for cooking thanks to the piles of dirty dishes. It’s actually kind of disgusting.

My wife takes a slightly different tact. After the meal is finished she immediately cleans all dishes, pots, pans, and utensils used while she was cooking. She has the forethought to run the dishwasher beforehand so there is plenty of room to load dirty dishes after the meal. She even wipes down the counter and stove so everything is ready for the next meal we cook. She’s quite amazing actually and a good cook to boot (especially when she’s following a recipe).

Professional chefs take matters a step further still. They clean as they cook.

I procrastinate doing something I hate and the result is a monumental, exhausting chore which takes an hour or more to finish. My wife spends 10, 15 minutes tops a night “tidying up” and though she hates doing dishes just as much as I, she makes it seem effortless. Professional chefs make miracles in the kitchen minutes at a time.

So it is with software.

Integrating software, even with a small team can be a chore.  Which would you prefer?  Approach A: write a lot of code, get everything working individually, and then do a big bang integration at the end; or Approach B: write a little code and integrate a little. While putting off integrating might satisfy your immediate needs, much like skipping dishes and moving straight to dessert, Approach A is going to cost more than Approach B in the end. Why?

• Integration problems aren’t uncovered until you integrate (profound, I know) so the longer you wait to integrate, the longer it takes to find out if there is a problem. Of course, no one ever plans for problems…
• Conflicts have further reaching consequences the longer you wait to fix them. Modern version control systems usually do a pretty good job merging changes but even magic has its limits.
• The full power of refactoring can’t be realized because the turnaround time on changes is too long.  The side effect is that you don’t refactor which means the code becomes more brittle over time.
• More code changes means more time to bring it all together and a higher likelihood of introducing defects through integration.  Unless you’ve planned knowing that integration will take time, chances are good you’re going to ship late.

Even better than big bang and nightly builds: continuous integration, cleaning as you code. Automated build servers have made huge advancements over the past few years. I highly recommend Hudson. It’s super easy to install and get started and has plug-ins for practically everything. There are even Hudson plug-ins for C# and just about every version control system you could want (and even some you don’t).

Doing chores sucks, but don’t make it worse than it has to be by putting things off. Continuous integration is a no brainer. If I had a dishwashing machine that constantly washed dishes as I finished using them I would dance naked in the streets, celebrating the marvels of modern technology. (You should be thankful such a machine doesn’t exist.)

You really don’t have an excuse for not using continuous integration.

Most software processes are light in two areas: quality assurance and process improvement. Most processes prescribe specific techniques for ensuring the production of quality code. XP for example advocates unit testing with TDD, continuous integration with smoke tests, pair programming, and acceptance tests written by the customer (using something like FIT so you can run automated regressions). Process improvement in XP is accomplished in a ’round the campfire, Kumbaya singing, get in touch with your feelings brainstorming session. Assumptions abound and there is no systematic way of ensuring that either testing or process improvement is handled adequately.  As we know, assumptions are never good enough.

The biggest assumption in XP (and indeed most software processes) is about bug tracking. Common sense dictates that you will create some kind of bug database. Hopefully it will at least be some kind of third party bug tracker such as Bugzilla or Bug Genie. Excel will work in a pinch but quickly becomes unsuitable for teams larger than one developer.  But how does the bug tracking actually work? What bugs get reported? Will you record issues from inspections in your bug tracker or only “true” bugs? What is the process for fixing a bug? What is the process for closing a bug? Who has access to the bug tracker? What information is required in your bug database and what information is optional? How do you determine defect priorities? Or the severity of bugs?

The majority of software processes provide answers for almost none of these questions. You are largely on your own to make up whatever you think makes the most sense for your development environment based on the best practices for your software process and your understanding of “good” software quality assurance practices.

No matter what quality process you follow, you will need a defect control philosophy. Once again, in absence of guidance I turn to the Team Software Process, one of the few processes to define what it means to track defects. In the Team Software Process, defects are treated as blight, a horrific mistake injected through the ineptitude of a developer. To remove these blights, the TSP relies on a series of filters in the form of code reviews, code inspections, unit tests, function tests, and so on. XP has a similar, though somewhat less rigorous set of filters in the form of pair programming, TDD, continuous integration, and acceptance tests. Each filter is meant to remove more and more defects, until finally “all” defects are removed from the system by the time the software has passed through all the filters. Generally each filter is intended to remove different types of defects, though it is conceivable to capture escaped defects from a previous filter in a later filter.

Just as water passing through layers of sand and rock will remove debris, so too will code passing through layers of unit tests and inspections sift out injected defects.

Defect Data

With these ideas in mind, bug tracking has three basic goals.

1. Record defects so they can be analyzed and fixed.
2. Identify the means by which defects are injected.
3. Identify the means by which defects are removed.

We write down bugs so we can go back and fix them later. This is obvious. But bug data can be used to measure process improvement also.

Since many parts of a software process are dedicated to filtering out the defects we’ve injected, understanding how defects are injected is essential to preventing similar defects from being seen in the future. The idea is that we want to learn from our mistakes. To achieve this, record the type of defect and the reason it was injected. The TSP gives us a good starting point for each of these, shown in the tables below. You should feel free to modify the types and reasons so they make sense for you, your team, and your project.

The defect type characterizes what kind of defect is injected and captures the essence of what is needed to fix the defect.

The defect’s reason characterizes why the defect was inject.

Assigning the reason can be a little tricky. I have found it is best if the person who injected the bug assigns the reason. On small teams (or if you’re following the PSP), this is generally pretty easy. It’s not about rubbing their nose in the problem – well, actually it is. Ok, it’s not about embarrassing or punishing the person but creating an opportunity for learning from our mistakes. If you can understand the reason why a defect was injected, it’s possible to prevent the defect from occurring again. For example, if there seems to be a rash of education related defects in a particular module, perhaps some training is in order.

Ideally we’d also like to know when the defect was injected, as in at what phase of development. It is possible to realize this information with additional analysis but I’ve found the return to be rather diminutive. Basically, you’ll learn what we’ve known all along – that the longer a defect is in the system, the harder it is to get out and that the most expensive defects are injected during the earlier phases of development (e.g. design defects are costly). Rather than track when things were injected I think it makes more sense to track when defects are detected. The point is to gain an understanding of how well the quality process filters out defects. To accomplish this, simply write down what you were doing when you found the bug. If you’re using XP, the list might include designing, writing new code (in a pair), writing new code (alone), refactoring, unit testing, integration, and acceptance testing. With this information you should be able to determine how effective each activity filters defects and over time whether the quality process is having issues. For example, I would expect interface defects to be detected during integration. If they are being detected earlier, say during unit testing, or later, say during acceptance testing, then my continuous integration and smoke test suite might not be as robust as it should be.

Better Bug Tracking

The strategies I’ve outlined here are a little more sophisticated than your average bug tracker, but add a lot of punch for very little effort. Tracking defect type, reason injected, and phase detected allow you to get a better handle not only on how defects are being injected into the software, but also how they are being detected. Both these chunks of information are necessary for understanding how defects are making their way into the system and how your process is helping you ferret them out of your system.

Process improvement is a tricky mistress.  Usually it’s sufficient to feel that something can be done better, but often, especially for larger organizations or folks looking for higher CMMI rankings, it’s necessary to quantify improvements in terms of measurable effects.  There is a cost to process improvement; every hour spent beefing up process is an hour less spent doing “real” work.  And if you optimize out the wrong processes it could seriously damper development.  The key is to determine the most cost-effective changes and only execute those process improvements.  Return on investment analysis is one way to understand cost-effectiveness (PDF), but I’ve found this technique conceptually difficult to apply to software.  How much is quality worth?  How do you put a dollar amount on developer happiness?  What’s a line of code worth to you?  It’s enough to make your head spin, but a good friend of mine, Kenny, once gave me some great advice that helps me keep things straight.

You’ve got to know when to hold ‘em

When you’ve got a process that works, keep using it.  It’s a great feeling and there isn’t much better than finding a process that works well for you and your team.  When you finally settle on a process that works it can lead to a boon of productivity.  If you’re disciplined enough to capture data, use it to prove that your processes are actually working.  If you aren’t the data type you’ll have to rely on heuristics and intuition and the best heuristic I’ve found is the KA Index.  The KA Index is a subjective measure of how much your team Kicks Ass.  The best processes are the ones that make you awesome – fewer defects discovered in the field, better predictions of when you’ll finish tasks, fewer fires to fight – all of these are “gut measurable” data points indicating that your process is probably working.

Your gut can get you far but eventually you’ll need data to convince your boss to make the investment.  I can highly recommend the Personal Software Process if you’re interested in learning how to use data to objectively measure your processes.

Know when to fold ‘em

When you’ve got a process that isn’t working, drop it.  Data can be used here too if you’re collecting it.  If you’re not, I’ve found carefully examining process affordances to be helpful in uncovering bum processes but affordances can be extremely subtle.  The trick when giving up a process is to quit strategically – not too early, before you might see benefits, and not too late, after you’re experiencing severe problems.  (The Dip by Seth Godin is an excellent, short read on this subject.)  The other thing to look out for is the cost of dropping the process.  Even if your process is terrible, if worse habits fill the process vacuum you could be worse off.

Know when to walk away

Once you’ve decided to abandon a problem process it usually pays to gently transition to something else.  Changing processes can be expensive between learning curve, tool changes, and the requisite burn-in period when trying out something new.  If your bum process is only killing your team slowly, take a little time to read up on a replacement and avoid setting yourself up for failure.  For example, when my team switched to Scrum, we finished the iteration we were in when we decided to make the switch before making drastic changes to our operating procedures.

Know when to run

If your process is absolutely killing you, drop it.  Immediately.  When considering this route I recommend a thorough impact analysis including some back-of-the-envelope ROI calculations.  Changing a horse mid race will be expensive, I’ve seen it happen enough times in the DoD to know that you will pay dearly in the short term, but if the long term costs justify the change and you’re willing to go through the pain, do not hesitate to make the change.

Never Count your money when you’re sitting at the table

Generally I’ve found that processes out of a book need a few iterations of development before I’m able to understand how they really work and how I’m supposed to use them.  If you’re following an established process (XP, Scrum, TSP, ACDM, and so on), then try to follow it to the letter before tailoring it.  Most processes have built in mechanisms for changing them once you’ve gained some experience and by all means, tailor once you know something doesn’t work.

There’ll be time enough for counting when the dealings done

Postmortems are one of the most powerful tools in your silver toolbox.  No matter how the project goes down, always take the time to reflect on what happened, why, and what you should have done so you can use that knowledge next time.  Personally, I hate the idea “lessons learned” and much prefer recommendations for next time.  “Lessons learned” implies that you’ve learned the lesson and no further action is required but this is hardly the right attitude for effective software process improvement.

Every process is a winner and every process is a loser

So every software engineer knows that the secret to surviving, is knowing what to throw away and knowing what to keep.  And in the right hands, the right process can be a savior while under the wrong circumstances the same process might kill a project.  I can’t offer any concrete advice other than to not be afraid of data, listen to your instincts, and look back on what you’ve done.

The Amsterdam airport was able to reduce the amount of urine “spillage” that hit the men’s room floor by 80% simply by etching a life-like image of a fly near the urinals’ drains. The fly was specifically engineered into the urinals to alter gentlemen’s behavior without their having to think about it. The concept is called nudging and it’s been used in domains other than restroom sanitation to encourage desired behavior. Other examples include the use of uncomfortable chairs in fast food restaurants to encourage people not to linger and real-time gas mileage displays in cars to encourage more economical driving. If you’ve read Donald Norman’s The Design of Everyday Things then you’ll know this as an affordance – a hint given to the user prompting them to take a specific action at a specific time.

Obviously the idea of affordances is directly applicable to devices as well as software usability but it wasn’t until I read about the urinal flies that I realized affordances don’t always have to have a physical representation. For example, a well designed software process should gently nudge a team to do the right thing. Since there is no one-size-fits-all process that works for all teams it is essential that the process complements the team and that the process’s affordances nudge team members to do what’s best for the project and the team.

Using a process that lacks the right affordances could have one of two possible outcomes. In the best case, the team abandons the process because they realize subconsciously that it is telling them to do the wrong things at the wrong times. This is bad because it sacrifices repeatability; you’ve regressed back to an ad hoc, “make it up as we go” state. In the worst case, the team sticks with the process and it leads them astray. This introduces risks into the project and could lead to complete project failure.

Software is already difficult enough to build successfully and processes are supposed to make software development easier. Unfortunately, knowing when something isn’t working is not an exact science, but with a dash of experience and little team reflection (for example from regular postmortems) it is possible to figure out when you are working for your process instead of your process working for you. To demonstrate this I am going to tell you a story.

Our Process

My studio team in the Carnegie Mellon software engineering program is charged with building a web-based requirements elicitation tool that helps users follow the SQUARE process out of the SEI. About halfway through the Elaboration Phase of the project (sometime in the spring semester) the project was going downhill. The warning signs were fairly apparent, we were missing milestones, tasking priorities were confusing, and a lot of work was stalling out at different levels of partial completion. Though we knew there was something wrong we weren’t really sure what was causing it, what we were doing wrong in our planning and tracking process.

The planning process we were using was fairly simple. At the beginning of the phase we looked at all the activities and artifacts that need to be completed by the end of that phase. For each identified milestone we enumerate specific entry criteria, general tasking, validation procedures, and exit criteria. This is a technique known as ETVX (entry, tasking, validation, and exit). Next we used planning poker to estimate how long we thought each milestone would take to complete. Finally, with this information we created a phase timeline which includes known due dates and dependencies between milestones.

Since we’re using an iterative approach to complete work in a phase, iterations follow largely the same planning process on a smaller scale. As a team we identify the milestones on which we will work during the iteration. Each milestone is assigned an owner whose job it is to ensure the milestone is completed by either delegating tasks or working on it themselves. The planning poker estimate is used to determine the approximate workload allocation on the team. This estimate is validated with bottom-up estimates that team members create based on their individual tasking.

There are several good things about this process. First, it’s written down and the team follows it. This is good because it means we can produce repeatable results over time. Second, this process makes use of several practices that are generally considered “good” by software experts. ETVX is a great way to clearly identify project milestones. Planning poker is similar to the wide-band Delphi estimation technique. Third, we’re using two forms of estimation to validate the plan as more information becomes known. Finally, the engineers responsible for the work determine the specific tasking and creating the bottom-up estimates.

You’re Good, but not That Good

In spite of all the good things we were doing, something still wasn’t connecting. The big aha! moment occurred about two weeks into the second iteration. Up to that point I had been working on my tasks that had carried over from the first iteration. The team leader noticed that almost no work had been started on the milestones I owned. [An aside: this, to me, says that at least our tracking process works somewhat well.] During the discussion that followed I became extremely defensive when the team leader asked me to shift priorities for the rest of the iteration. What should have been a simple request turned into a heated debate over tasking. I felt compelled to complete the past due work and here was this jerk trying to stop me. “Sure,” I thought, “I’ll do what you ask, buddy, but when this whole project comes crashing down it’s on your head, not mine.”

Later, as I looked back at the incident, I wondered to myself, “Why was I so defensive in light of such a simple request?” The reality was that the project wouldn’t come crashing down if I shifted priorities and I knew that. So why defend these older tasks when it was obvious that there were more immediate needs?

It turns out that the affordances built into the planning process were encouraging my behavior. There were a few simple things at play that, when combined, decreased our ability to plan effectively.

First, our process encouraged us to plan more work than time allowed. This was due to there being a missing connection between day-to-day progress and the “big picture,” the overall plan. Second, though the new team leader may have believed there was consensus, the team in fact did not wholly agree with the priorities for iterations. This behavior was not specifically discouraged by our planning process and so allowed to persist. Third, leftover work was not addressed during planning. Some tasks might simply expire while others may change priority, becoming more or less important with a new iteration. Since this wasn’t addressed it created a sense of urgency for individuals carrying over work from iteration to iteration. Finally, assigning milestone owners had unanticipated side effects. The goal was to ensure that someone was taking responsibility for coordinating and monitoring milestone work. This worked so effectively that milestone owners exhausted themselves attempting to finish milestones and resisted changes to the plan that prevented them from finishing what was promised.

When it came time to make a necessary modification to the plan, our process encouraged us to fight against the best course of action for the team. We didn’t have the level of flexibility needed due to our process’s affordances nudging us to do the wrong things. Milestones were slipping and people wanted to finish what they started. Project priorities were shifting as the project matured but team members were wearing blinders, ignoring the changing facts around us. To stand a chance at success we had to change the affordances in the planning process. We had to nudge the team in a new direction.

Our Solution

To try to solve this problem we decided to incorporate some of the planning principles from Scrum, specifically the product backlog, sprint backlog, and sprint planning meeting, into our planning process. Scrum takes a more task-oriented approach when planning iterations and correlates the sprint backlog with the product backlog. This better encourages the team to not plan more work than there is time to complete while connecting day-to-day work with the overall plan. Scrum also requires that the team reprioritize work when planning iterations and that we agree on the resulting priorities. This will hopefully eliminate the prioritization conflicts we experienced during iterations. With Scrum, leftover work from iterations is saved in the product backlog. This change decreases the anxiety team members feel when work is left undone (because the work is not forgotten) while simultaneously giving the team more flexibility to change direction as the project progresses. Finally, the team, rather than individuals, takes ownership over the milestones held in the product backlog. With each commitment made during iteration planning, the whole team buys in effectively shifting the passion and dedication individuals held for owned milestones to the commitments we agreed on as a team.

I’m not really sure how Scrum is going to turn out for us. I think the most important thing is that we recognized that something was not working and took action to correct it. I personally would rather see the team fail in a new and spectacular way rather than repeating the same mistakes again and again.

Add This to Your Silver Toolbox

Unfortunately, I don’t think there is a trick for detecting these sorts of process failures. Data and metrics can help but only if the process is repeatable and the team has the knowledge and discipline to collect the data in the first place. Team postmortems can help but if individuals are afraid to raise concerns, you’ll find yourself on a trip to Abilene before you realize it. In many cases, if you think something isn’t going well, others are probably thinking the same thing. Once I spoke up I found out that others thought something wasn’t working also. I was just the first person who was able to articulate it.

Affordances are powerful but subtle mechanisms. In well designed things, we aren’t supposed to be consciously aware of them. But that doesn’t mean they always nudge us to do the right thing.