Reflections on Software Engineering

When I was a kid my brother and I used to play a game called Make Believe. My favorite variant of the game was simple. Together we would build some kind of fortress and then one person gets the fort and the other person tries to invade the fort. In theory, the game ends when the fort has been overtaken by the invader. What made the game fun was that as the invasion began, the rules of the game always changed. The first thing to go was any notion of death. If one of us was “killed” in battle then near instantaneous respawning was created. Shortly after that we skipped respawing and simply became invincible. Soon the fort became invisible which means the invader just has to run around trying to find it. Sometimes someone gained super strength or the ability to force other people to move in slow motion. We almost always created super weapons (such as a hand held Death Star) which for some reason could always be defended against. Nearly every game ended in tragedy, someone crying or upset: “That’s not fair! You can’t do that! I’m invisible! You can’t do that!”

Kid’s stuff right?

A lot of software projects with teams made up of working adults still play this game. The scenario goes something like this. A team is put together to build some software. Neither the clients nor the team talk about the objectives of the project other than building “some software”. After a few months, something goes wrong or someone doesn’t like what’s happening so someone changes the rules. Before too long, one side or the other is upset that they can’t win, somebody throws a fit, and goes home. Instead of summoning invisible armor, software projects change the rules by cutting features, adding more requirements, moving due dates, wasting resources, and things like that.

We make believe that we’re software engineers.

While Make Believe was a fun game as a kid, changing the rules when there’s real money on the line isn’t as fun. My brother and I ran into problems as kids because we got the objectives of the game wrong. Actually, there were no common objects, which is why we could change the rules so easily. The same thing happens on a software project when the objectives aren’t well known.

Defining and committing to a clear picture of success establishes the common ground rules for a project by making the basic project goals explicit. The technique is known as Threshold of Success.

Defining What Success Looks Like

The Threshold of Success for a project is the minimum set of conditions that must be met for the project to be considered successful. If the team fails to meet even one of the conditions then the project is a failure. A good Threshold of Success is made up of about 3-4 SMART goals (no more than a few bullets on a single PowerPoint slide). SMART is a mnemonic which stands for Short/Specific, Measurable, Achievable, Relevant, and Time bound.

Some other pointers for defining a Threshold of Success:

• The Threshold of Success should be built as a team. Since this is the measure by which you will define success or failure, everyone on the team must buy into it. If you can include your client that’s even better.
• Threshold of Success goals should be challenging, but it’s important that they are achievable. If the goals are too easy, victory will be meaningless, too difficult, elusive.
• Once the Threshold is established, don’t change it! The only reason to modify the Threshold of Success is if the project has changed so drastically that the Threshold no longer makes sense (for example if someone leaves the project).
• Revisit the Threshold of Success regularly (a good time is when planning iterations) so everyone remembers what success looks like. Put it on your team wiki so that it’s readily accessible.
• Be sure that the goals in your Threshold are SMART! The point of defining a Threshold of Success is to take away the wiggle room for defining what it means to succeed or fail. The goals you define should make this black and white. The more specific the goal is the better.

Building a Threshold of Success

The easiest way to create a Threshold of Success is to first create a minimum picture of failure, then convert failure into success. Here’s an example:

Failure for my current project might look something like this.

• Essential features are not ready by the end of the second quarter.
• Team members are dissatisfied or bored with their jobs.
• Newly hired team members don’t feel like they’re part of the team by March 31.
• There isn’t enough money to continue development after this fiscal year and we have to fire people.

Now that I know what failure looks like, seeing success is easy. I don’t want any of these things to happen. The threshold of success for my current project might look something like this.

• By the end of the second quarter, all “Must Have” features are implemented and pass acceptance tests with no known critical defects.
• All team members give average score of 5 or better on a job satisfaction survey taken quarterly.
• By March 31, the team has successfully executed at least three team building activities with all team members present.
• Funds of at least $1 million are secured by December 31 to allow for future development without a reduction in team size.

Notice that only 1/4 of the success goals in this example are related to software functionality. While goals might come from anywhere, teams traditionally focus on goals related to people and relationships, process, resources (such as budget or schedule), and product (software functionality and quality).

As this technique originated with the Software Engineering Institute (pdf), nearly every studio team in the Carnegie Mellon Master of Software Engineering program creates a Threshold of Success for their projects. The MSE Studio Archive has extensive examples of both good and bad pictures of success that teams have created. The Square Root Team’s threshold (my team) is a good place to start, but there are plenty of other examples.

There might be many goals for a project. In the Team Software Process you actually identify at least three different kinds! But there is only one threshold of success for a project. Knowing what success looks like gives you a better chance of actually achieving it.  Without it, you’re just pretending that you know what’s going on.

Van Halen may have known more about project management than most program managers. Van Halen’s legendary “No Brown M&Ms Rider” is simultaneously the greatest example of rock star excess and project signaling I’ve ever seen. As David Lee Roth puts it:

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say “Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . .” This kind of thing. And article number 126, in the middle of nowhere, was: “There will be no brown M&M’s in the backstage area, upon pain of forfeiture of the show, with full compensation.”

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

In economics, signals are indicators that convey specific meaning between producers and consumers. For example, when you see THX on the side of a set of speakers, you know the speakers are going to probably be of audiophile quality. The THX logo is the speaker manufacturer’s signal to you, the consumer, that these speakers are really good. To David Lee Roth and the Van Halen road crew, the presence of brown M&Ms indicated that the hosting venue had not understood all details of the contract and had very likely made a mistake in configuring the set. One mistake in this case could cause malfunctions during the show or even the death of a crew member.

As it turns out, signaling software projects isn’t that difficult. The 12 step Joel Test is a reasonable signal for software development companies. While the Joel Test is nice for getting a feel for a company before you work for them, the concept is still useful once you’ve got the job and the project is in full swing.

Ultimately signals, also known as tripwires or triggers, are really just binary metrics for uncovering potential problems your project might be facing before the problems explode in your hands. When some condition is met (the signal), you know it has specific significance and prompts certain actions to prevent a problem from occurring. Triggers are most often used with risk management but their use should not be exclusive to that practice. In fact, if you’re collecting real data, you have even more opportunities for identifying signals outside of risk management.

On past projects I’ve used signals for a variety of issues. Here are some examples.

• During the past 3 iterations the team identified between 15 and 20 defects. I expect a similar number of defects to be detected for this iteration. If more defects are detected, there may be a disconnection in understanding between requirements, design, and implementation. If fewer defects are detected, tests may not have been as rigorously defined as they should have been.
• A Fagan inspection completed in less than one hour with a rate of 400 LOC/hour. Since most inspections have covered only 250 LOC/hour it is likely that this inspection was not effective and the results not reliable since the inspection team sped through the code.
• When evaluating potential open source libraries, Source Forge projects without a website shows a general lack of dedication to the project and indicates that the software is probably of poor quality or ill-maintained; the library is worth neither the time nor effort to use.
• Tasks that have been estimated to require longer than 9 hours have probably not been thoroughly thought through.
• No risks have been identified for this project or risks have not been updated for several iterations. This implies that the team doesn’t have a realistic understanding of what problems the project faces.

In each of these examples, when the signal is heard, I knew there was going to be a problem on the project.

Work with your team to establish signals for your project. The best part is that once you’ve decided on the signals for your team, when triggers are tripped you can throw a Van Halen sized rock star fits in your cubicle! Well, try to resist throwing your monitor out the window anyway.

Software developers are, in their heart of hearts, dataphiles – people who are absolutely in love with data. When was the last time you had a passionate discussion about frame rates, hardware benchmarks, gadget specs, sports statistics, dungeons and dragons, the merits of high def…the list goes on. Face it, you love data. You love comparing things using data. You don’t feel comfortable making decisions without a comprehensive comparison of data.

Why then do most software developers treat software development differently?

Tom DeMarco recently brought his own famous quote into question (pdf), musing that not only is it possible to control what you can’t measure, but the most important stuff you need to control on a software project is impossible to measure. Once again, DeMarco is wrong (in my opinion anyway).

Wikipedia is one of the most controlled projects on the planet

One the surface, Wikipedia is the Wild West of online content. Not only can anyone edit any page, but content from Wikipedia is widely proliferated in the media and (sadly) school reports. Wikipedia is the single greatest success of user generated content in the history of mankind (”The Internet,” as the medium, doesn’t count). What started with a dozen humble articles has evolved into the most comprehensive encyclopedia ever created and includes everything from the fundamentals of science to the definitive source on Babylon 5.

What folks seem to forget is that even in the Wild West, there were laws and there were lawmen. Though we love to think romantically about such brigands and gunslingers as Jesse James, Billy the Kid, and Butch Cassidy, most stories about these historic figures are greatly exaggerated. So too is the case with Wikipedia.

Let’s take a closer look at the Wikipedia entry for Billy the Kid. This article belongs to a number of internal WikiProjects, visible from the top of the article’s talk page. The WikiProject Biography is not unlike most projects in Wikipedia. There defined processes for assessing articles, conducting peer reviews. There are rubrics defined for assessing the quality of articles within the project. People even take on specific roles and responsibilities within the project. The collection of processes and information serves as the main means of coordination for content contributors and helps the group control articles within the scope of the project.

The WikiProject Biography even collects metrics on articles which it then uses to make decisions concerning the articles under the project. The metrics are derived from quantifiable data and help control the project.

As it turns out, Wikipedia is not the lawless territory of the internet it has been made out to be.

You can measure the immeasurable

Wikipedia works because people were able to figure out ways to measure things that usually can’t be measured. The fundamental principle that many people overlook is that binary is a metric too. Yes or no questions can be just as effective a measure as any complex metric. Did everyone fill out their task data today? Yes or no. Did the estimate match the actual? Yes or no. Did the test pass? Yes or no. Is the project done? Yes or no. Have we identified risks? Yes or no. Has this risk become a problem? Yes or no.

At the heart of every complicated metric is really a series of yes or no, binary questions. When considering whether the project is done, you have to define done. One way of defining done is in terms of a checklist. Is feature 1 done? Is feature 2 done? Defining done for a feature could be as simple as checking whether all the tests have passed for the feature, again a binary measure.

For more subjective assessments, you can rely on observation-based, experience-defined rubrics. Does the team get along with one another? In the simplest form, this could be a binary metric (Am I friends with everyone on the team?) but it could also be more complicated relying on gut feelings and a guiding rubric (”we never hang out together and don’t trust one another” might indicate low harmony while “we hang out often and feel comfortable sharing personal stories” could indicate high harmony). Teachers use rubrics and experience to judge subjective assignments everyday. The difference is that they slap a grade on it and send it home as a report card.

While DeMarco is correct that many of most critical things in a project are the most difficult to measure, it is possible to create measurements if you feel it is important enough to do so. How would you assess whether you have a good architecture that solves the problem at hand? Rubrics might play a part but so too might binary gates based on quality attribute scenarios or intricate observations concerning design trends over time. If you think hard enough, you’ll find that it’s extremely easy to find measuring points for nearly every aspect of a software project.

Whatever you do, don’t become a mindless, data-driven robot

I love data and I know you do to. While it’s tempting to inject data collection and derive metrics for every aspect of a project (because it’s fun and informative!) don’t. Collecting data and calculating metrics can be expensive. Not so expensive that you shouldn’t use it, but expensive enough so that you shouldn’t use it on everything. I like to compare using metrics to eating out at restaurants. Once or twice a week isn’t that big a deal, but it’s not something you should do every day if you’re trying to watch your budget.

DeMarco is right about one thing: control is not the end-all-be-all of software engineering. Consider carefully, what are the most risky parts of my project? What are the parts of my project that even require control? What are the parts in which I need more insight or want to improve? Strategically develop metrics for these areas and don’t worry about measuring the rest. Trust me, the world won’t end. If you don’t know what you’re doing, start with a simple binary measure. And above all, if something isn’t working, change it.

The setting is a small library filled with various books on software engineering. Five people are sitting around a single, small table, the room overcrowded with the group. The Square Root team is reviewing the mid-semester presentation I threw together last night before we were to show it to our mentors in a few hours. I was showing the team the last slide. Up to that point everyone had pretty much agreed with the content in the presentation.

“Does anyone have any other risks they’d like to bring up?” I asked, confident that, as the team leader, I had a firm grasp on how the team was doing and where our current problems lay.

I waited a few seconds.

“OK, if nobody’s got anything I’ll go ahead and email this out…”

“Communication,” the quietest member of the team chimed in. “We have problems with our team communication.”

I was in shock. Surely this must just be an isolated problem. “Do you mean team communication or do you mean you don’t understand some aspect of the project?”

“I agree. Team communication seems to be problematic.” Now there were two dissenters.

A few seconds later there were three. Three fifths of the team felt we had team communication problems that were directly impacting the outcome of the project. I sat there shocked for a moment. I had just been blindsided by a team communication problem, therefore, obviously, we have a team communication problem.

Looking back at this incident I have decided that leadership, my leadership, was to blame. From the beginning, I had led the way I liked to be led: hands-off with enough space to make my own decisions and get things done the way I wanted to do them, the polar opposite of micromanagement. Of course, for me this worked out really well. This was the way I liked to work and the way I was used to working. Unfortunately, for a new team, a team lacking in trust, whose members didn’t know one another and had other commitments and priorities (school work) my ideal working environment and preferred leadership style was disastrous.

To prevent utter team destruction I changed perspectives. From what I could tell, team members didn’t know what they were supposed to be doing. I had assumed that this team would operate similarly to my last team where everyone would come together like a well-oiled machine to get work done. Shortly after the incident in the library I remembered that my last team took almost a year to become that awesome. Of course there were going to be problems in my new team.

Since my team seemed not to know how to take initiative in completing tasks and getting work done I thought I’d set an even better example for them. By doing so, maybe they’d get the hint and follow suit. The next day I kicked things into gear. I took on more work. I picked up all the slack I could and then some. I took on tasking outside of work specifically assigned to my role. I set high standards for myself and my team and we were going to meet those standards if I had anything to do with it.

Leadership disaster number two was prevented thanks to a timely assignment in my Managing Software Developers course. Leadership that Get’s Results by Daniel Goleman defines six distinct leadership styles and gives a little advice on when each style is appropriate. Up to that point I had no knowledge of such styles and had been flying on instincts. The following table is taken from Goleman’s paper but I encourage you to read the full paper to gain a better understanding of these ideas in a more complete context. There’s also tons of information on the web, a search away.

As it turns out, I started with the right idea by using an authoritative style of leadership even though I didn’t know the name for it. I failed with this style because the team wasn’t ready for it yet. We didn’t have a clear vision or clear goals. We weren’t yet working well together. There was little buy-in to my leadership or the few goals the team did have. We were a full-fledged dysfunctional team. Switching to a pacesetting style was just about the worst possible thing I could have done. If I had gone on for too long I am fully confident that I would have destroyed the team and we wouldn’t have gotten anything done during our first semester. With my new found knowledge of the different leadership styles I now had new options. I immediately put three styles to use: affiliative, coaching, and coercive.

The team was clearly broken. My hope was that affiliative leadership might help build trust and better bonds among teammates. Being a programmer I tend to be a little more logical than emotional so trying to tune in more to how people felt was tough. I found that affiliative leadership goes hand in hand with vulnerability and trust and that individually thanking someone for their hard work, even if it’s relatively small, and meaning it is one of the most important things you can do as a leader.

In addition to team harmony, it was obvious that some team members were struggling with the tasks they had been given. Rather than taking over those tasks as I had been doing, I decided to try taking on more of a coaching role. In some cases I would help team members directly, other times I encouraged other team members to work together on tasks. The end result was a team better able to work together to accomplish tasks.

In spite of all this, I had the feeling that if we didn’t do something immediately, the entire semester would be a wash. To prevent this from happening I used coercive leadership in an attempt to get us back on track quickly even though the overall impact could be negative in the long term. In a sense, I was willing to be a bad guy so the team had a chance of meeting its goals. The gamble paid off and the way the team operated turned around almost instantly.

About two months after I had been blindsided by unseen communication problems, my team seemed to be working together much better. Problems were being flushed out into the open more quickly and everyone on the team seemed to enjoy working with one another on the project. I am not going to take full credit for the change but I will take credit for being the catalyst that put the change into motion. All because I changed how I led the team.

There is a small downside to the changes, but it’s only really a downside because I enjoy doing technical work. The leadership role on my team has evolved. By the end of the semester I found myself directly responsible for almost no work but rather, I was the go to guy for all problems the team was facing. I had my hands in everything but I wasn’t able to really sink my teeth into anything. If this is what management is like and I ever do become a project manager I will need to find ways to remain technically involved in something or I think I’d eventually go insane.

The biggest lesson I’m taking away from this experiences is that sometimes the team’s needs and my needs aren’t going to line up. Recognizing when this is occurring and finding a balance between these conflicting needs is critical to team success.