[This post is a recap on the second talk I gave at XP2010. This was the big one, the experience report talk, one of 15 experience reports published at XP2010. You can download the slides (pdf) or the full paper (pdf) from this website or from XP2010.org.]

Process improvement is important for nearly all teams but it can sometimes be difficult for a team to know what is working, what isn’t working, and what techniques or methods to try when attempting to improve. Performing a scientific experiment is one way help overcome these problems but as academic research has shown us, while experimentation can yield interesting results, running an experiment is time consuming, expensive, and requires some serious thinking and control to pull off. From a practitioner’s standpoint this means that experimentation is a non-starter.

Of course, that’s only if you run experiments like an academic.

Back Story

Just over a year ago, my MSE studio team at Carnegie Mellon had a problem. We had decided we would use Extreme Programming for the construction phase of our project but some team members had doubts concerning pair programming. We had decided that we would use some kind of peer review, having already seen the many benefits of inspection when reviewing other artifacts. The dispute arose over whether pair programming would give similar enough results. Also, not all team members had experience with pair programming but everyone on the team knew and enjoyed solo programming.

The number one concern was whether pair programming would allow us to meet our very strict deadline. We had just over three months to complete the construction phase of the project. According to our threshold of success this meant implementing all “must have” requirements with a minimum level of quality. Did we really have time to waste by having two people working on the same code at the same time? Wouldn’t working independently and inspecting code on an as needed basis allow us to get more work done faster?

At the time it just so happened that I was taking a reading class with Mary Shaw and in that class we discussed some research findings that might help settle this debate. Research from Laurie Williams, Ward Cunningham, Barry Boehm, and many others showed that pair programming requires more effort (although never double the effort) but is faster than programming alone (pdf). Also pair programming creates code of about the same quality as coding alone with inspection (pdf). Of course, the research may not apply to us since Square Root is closer to a professional team working on a large project with a real client, not undergrads working on short term toy projects.

After an iteration where some teammates used pair programming and others refused, we decided to try an experiment to see which practice actually worked better. The original idea was that we might be able to validate some of the research but decided instead that it was more important just to resolve our own internal conflicts and figure out which processes worked better.

Conducting a Lightweight Experiment

With the scientific method as our guide we planned and executed a lightweight experiment which pitted programming alone against pair programming. The results were amazing (and you can find the raw data in our project archive). In conducting the experiment we used a set of novel techniques which I think can be useful in conducting other lightweight experiments. There’s more background in the experience report so I’m only putting the meaty stuff in this post.

Focus narrowly on a single question – The essential key to keeping an experiment light is to only tackle one thing at a time. In this case we focused on comparing and contrasting a single technique, pair programming, rather than multiple techniques or an entire process (such XP vs. TSP).

Divide work, not teams – If I were comparing pair programming to programming alone in an academic setting, I would put together two teams of about the same experience and have them each build their own version of the same software, one team using pair programming, the other programming alone. In a business setting this is a complete waste and few companies can afford to have two teams duplicating effort. By dividing work instead of teams you may lose some control over variables in the experiment but in most cases isolating more variables doesn’t add any further clarity to helping answer the narrowly focused question. To divide work successfully you need to have some way of estimating work units for division. We used use case points as shown in the figure depicting our modified planning game.

Continue making releases – Since we still needed to make a comparison, rather than dividing into teams and duplicating effort we divided the features that were released each iteration. In this way we built about half the features released during an iteration using each technique. Working on about half the features using pair programming meant that at least some features were being built by individuals. At the time this was a risk reduction decision to make sure that if pair programming completely failed we’d still have something to ship at the end of the iteration. Explicitly managing risks is the only way to know if the lightweight experiment may cause problems for making releases. Also, we had a strictly defined cut-off for stopping the experiment if it ever stopped us from shipping to our client.

Use the data you have – In almost all cases we were able to get the data we needed to evaluate our hypothesis from our current process. When we couldn’t, we only had to make minor modifications to our data collection practices, for example adding a check box to our SharePoint server for indicating whether a task was paired or individual.

One of the more interesting things we did was to create a “tally sheet” for collecting pair programming issue detection statistics in real time, as the issues were discovered. Given the near instantaneous code-inspect-fix cycle when programming in pairs, this was the only way to collect similar data for comparing pair programming to inspection.

Statistical significance is overrated – The whole point of running a lightweight experiment is to collect just enough data to help you make a better decision or validate your gut feeling. This technique is not meant for uncovering universal truths or proving something to the rest of the world. In exchange for keeping the experiment light, the results will only apply to your team. Over the course of an iteration or two, 4-6 weeks, you’ll only get enough data to start to see trends. In our case the results were not statistically significant using individual T-tests but that didn’t matter. The most important thing is that we had data that could be used for comparison, data that everyone felt good about and that helped us gain clarity into what we did and how well it worked.

Retrospectives get immediate value – The whole reason the experiment is light is to reduce cost and decrease the lag time to providing value to the team. Just to give you a little perspective, it took us 6 weeks to run the experiment and had enough data and casual observations to make a decision during the retrospective when the analyzed data was shared. That event occurred in early August of 2009. This experience report required almost nine full months of gestation from the paper proposal to the talk I gave at the conference. The gestation period on “universal truth” research can be even longer. We, as practitioners, don’t have to wait for those universal truths to be born to get value from research. By running your own quick and dirty, lightweight experiments, you can get results in a timely fashion that you know will apply to your team because your team was the subject of the experiment. It’s all about closing the gaps between research and practice and taking the information you need now instead of waiting for academic research to catch up.

Overall Conclusions

For the Square Root team it turned out that pair programming was faster, cheaper, and produced code that had more predictable albeit slightly worse quality. The more important lesson is that we discovered a technique, lightweight experimentation, for learning other interesting things about our team and about software engineering in general.

My paper and this blog post were all about trying to describe the technique, using our experiment as an example. I think it would be awesome if teams around the world conducted lightweight experiments on a variety of topics. If enough folks share what they learn, we might start to see trends emerge across teams that could lead to universal truths, validate research, or at least discover some great rules of thumb.

What else might make for a great experiment? Anything you’ve got a question about on your team!

• What is the clearer way to write requirements, user stories or use cases?
• Which estimation technique is more accurate of X and Y?
• Can we skip unit testing if we use inspection (looking at quality, knowledge sharing)?
• Is UML a better design notation than the one we made up as a team?
• What else…?

If you do a lightweight experiment, let me know! Share what you learn as a blog post or whitepaper. Let others know what you’ve learned! Even if the specific results only apply to your team and the way you’ve executed your project, your experiences help form a baseline, a sort of shared understanding for how software development works, how some of these practices work. And there’s so much about software engineering that we have yet to learn.

Acknowledgements

This paper was my first experience report and it was an awesome journey. Naturally a lot of folks helped me along the way and I would like to take a moment to make sure they know that I appreciate their influences and support. The Square Root team: Marco Len, Yi-Ru Liao, Abin Shahab, and especially my fellow experiment co-champion Sneader Sequeira for having the guts to go along with this idea in the first place. Some of the faculty at Carnegie Mellon: Dave Root and John Robert (my studio mentors) for bringing up the idea of writing a paper, and Jonathan Aldrich for helping review my proposal. Artem Marchenko was my XP2010 paper shepherd after the proposal was accepted, and the quality of each draft only improved because of his inputs. A group of my fellow employees at Net Health Systems sat through an early draft of the presentation I gave and shared valuable feedback for improving it. And finally I thank, Marie, my wife, who was with me from start to finish and read more drafts and sat through more practice talks than anyone else. She’s probably as much an expert on this subject by now as I.

A Final Aside

I wrote the initial draft of this paper as my final reflection paper for my Master of Software Engineering degree (pdf). That draft has a very different tone, approach, conclusion, and direction than what I eventually published for XP2010. This is half due to there not being a hard page limit but also I had a lot more time to think about what was really important when writing for XP2010. There’s some interesting information, mostly in the lessons learned, that might prove interesting to those who are interested. You should check out my Square Root teammates’ reflection papers as well since they are all interesting and well written.

Over the past few weeks I’ve been thinking a lot about risk exposure in the context of managing projects. Exposure is a technique used almost universally when managing risks, yet as I’ve already discussed, exposure can cause major problems because it’s a precise number based on mostly made-up information. At the same time, exposure is used widely and successfully – otherwise there wouldn’t be as much literature throughout the web telling you to calculate risk exposure.

This begs the question: is risk exposure really as meaningless as I’ve made it out to be? I’ve collected some data that helps answer this question.

Data Collection and Context

Risk management is one of the basic subjects covered in the Managing Software Development course, one of the five core courses students of the Carnegie Mellon Master of Software Engineering program take in completing their degree. Students learn about the continuous risk management paradigm from the Software Engineering Institute. Two of the cornerstones of this technique are threshold of success and condition-consequence based risk statements.

Having ready access to risk management experts at the SEI, nearly every team conducts a facilitated small team risk evaluation workshop in which risks are collected with the help of a taxonomy-based questionnaire (pdf), analyzed, and prioritized using group multi-voting. The basic workshop has been conducted the same way for close to a decade and many teams have put their risk data collected during the workshop in the MSE’s project archive.

I’ve gathered data from these small team risk evaluation workshops for 9 MSE Studio teams, a total of 164 identified, analyzed, prioritized risks.

What’s in the Data?

During a risk evaluation workshop, teams identify risks using their threshold of success as a guide. Once identified, risks are briefly analyzed and assigned an impact, probability, and time frame value based on a rough average from the team members’ initial gut feeling on the risk. These values are assigned simply so when a manager asks to see the probability, for example, there is a value to give him. Each of impact, probability, and time frame can only be one of 3-4 values. The idea is that by decreasing the precision we can increase the accuracy. Values are assigned based on a rubric. For the purposes of calculating a risk exposure I assigned each of the analysis categories a number. Time frame is not used in calculating exposure.

Impact

• Catastrophic – The team will be unable to meet threshold of success. (numeric value 4)
• Critical – The team can only meet the threshold of success with significant additional effort and stress. (numeric value 3)
• Marginal – The team can meet the threshold of success with minimal extra effort. (numeric value 2)
• Negligible – There is no real impact on achieve the threshold of success or little increase in effort. (numeric value 1)

Probability

• High – Chance of becoming a problem is above about 80%. (numeric value .8)
• Medium – Chance of becoming a problem is about 50/50. (numeric value .5)
• Low – Chance of becoming a problem is below about 20%. (numeric value .2)

Time Frame

• Short – May occur in about a month or less.
• Medium – May occur in 1 to 3 months.
• Long – May occur in more than 3 months.

Instead of relying on the results from the analysis, teams perform 3 to 4 rounds of multi-voting. The final multi-voting rank is shown. Not all teams ranked all risks since teams generally only deal with the top few risks, usually less than 10. This idea is captured in the priority. A risk is either a high priority, meaning the team is actively addressing it, or a low priority meaning the team is aware of it but it was not ranked high enough to deal with yet. Teams might choose different strategies for determining priority. The two most popular are to only examine the top X or to rely on consensus derived from how the risks clustered as a result of multi-voting. Usually there is strong team consensus for the top 4 to 5 risks and weak consensus after this.

Analysis and Discussion

My hypothesis is that teams’ rankings will generally match exposure, meaning that risks that are ranked highly will also have a high exposure. As the data shows, this is generally the case. On average nearly every team’s high priority risks were also the ones with the highest exposure.

Examining the risks rank and exposure tells a similar story but not convincingly. There is a relatively weak negative correlation (correlation coefficient of -0.22) between exposure and team assigned rank. Basically the best that can be said is that there is a general downward trend in exposure as the rank increases but there is enough variation that I can’t really say anything for certain.

I have two possible explanations for this. First, traditional risk exposure does not take into account time frame while teams evaluating risks in this data set do. So, all things equal from an exposure perspective, a long term risk might be ranked very low while a short term risk will be ranked much higher. If this were the case, we’d see more short-term risks assigned high ranks than long-term risks and this is indeed the case. In fact, the majority of risks identified are short-term risks with nearly three times more short-term risks being identified than long term risks. Mid-term risks are, unsurprisingly in the middle. A better exposure number might be had by taking into account risks’ time frame values.

The second possible explanation I have is that 3 – 4 buckets isn’t sufficient to allow for enough variation to form a strong correlation between rank and exposure. Indeed this is one of the greatest differences between this data set and traditional risk exposure calculations in which impact might take on nearly any number and exposure is usually a percentage from 10 – 100%. That said there still is a general trend which shows that most of the time, multi-vote ranking very roughly corresponds to exposure.

There is one more catch about this data and it’s a subtle but important one. Values for probability, impact, and time frame were determined as a team using a sort of rough average approach where team members vote and the approximate averages are rounded to the nearest bucket. Since all the values and rankings were determined through a group effort, it would make sense that they should roughly correspond.

Conclusions

As it turns out, risk exposure is a rough and somewhat accurate indicator for relative risk priority, at least when calculating exposure or rank using group-driven techniques. Teams relying only on exposure are likely to rank some risks higher than they otherwise might. Part of this is due to exclusion of the concept of time from traditional exposure, part of it might be differences of opinion within the group as far as impact or probability are concerned.

Talking with other MSE alumni, and I mostly agree with them, the most important thing about risk management is bringing up concerns and talking about them. Delphi mutli-voting is an easy way to encourage conversation since differences of opinion are addressed as part of the multi-voting process. No matter what technique you use, exposure (with time somehow included), multi-voting, or some combination, do not reduce risk management to simple numbers. It’s really all about communication. Encourage this communication using whatever techniques work for your team.

Raw data used for analysis in CSV format.

I like the idea of the risk burndown chart. Burndown is an effective and satisfying visual indicator of progress and it’s relatively easy to calculate to boot. But does looking at a project’s risks through the lens of a burndown chart make sense?

I see several problems with thinking about risk in this way.

Numbers can be Misleading

The first key to effective risk management is to value accuracy over precision. This means that it’s better to be right in your predictions than it is to be spot on correct. Remember, risk is about assessing your likelihood for project success. It doesn’t matter if you miss your threshold of success by a little or a lot; either way you still fail the project!

Pop quiz. Say there are two risks in your project. There’s a 25% probability that Risk A will become a problem while Risk B only has a 20% probability. For now, assume the impact is the same for both risks. Which risk is a greater threat to the project?

That one’s easy. Risk A is a greater threat because, impacts aside, Risk A has a 5% greater probability of turning into a problem.  Ok.  What if I told you that I made up probabilities based on my gut feelings so I could easily rank risks? Now which risk is a greater threat to the project?

The real question I’m asking you is this. Are you willing to bet the success of your project on those numbers? Because if my best guess, gut feeling probabilities are off by more than 5%, the project could be in serious trouble depending on the risks’ impacts.

I know, I know. That was a trick question. Nobody on your team would make up numbers on one of your software projects. In all fairness, nobody goes out of their way to fabricate false values. Use your logics. If you were any good at guessing the probability of futures events occurring, you would not be reading this post right now. You would be a multi-millionaire, off enjoying your gambling winnings from the ponies. Too much precision gives folks too much confidence in the correctness of your assessment when the reality is that probability and impact are based on best guesses and gut feelings. Probability and impact numbers just make it easier to calculate exposure so risks can be ranked automatically.  Burndown is a fairly precise metric.

Not all Risks are Created Equal

If you are monitoring project risk with a risk burndown chart, how do you know whether the right risks are being reduced? Let’s take a look at an example.  Which of these sets of risks should be addressed?

Set 1 with a total exposure of 7 days made up of the following risks:

• Risk A has a probability of 20% and an impact of 15 for an exposure of 3 days.
• Risk B has a probability of 25% and an impact of 10 days for an exposure of 2.5 days.
• Risk C has a probability of 30% and an impact of 5 days for an exposure of 1.5 days.

Or Set 2 with a total exposure of 7 days (6.7 rounded up) made of the the following risk:

• Risk D has a probability of 95% and an impact of  7 days for an exposure of 6.7 days.

In the first set, I can mitigate 3 risks, each with very low probability of becoming problems. In the second set I mitigate only 1 risk that is almost certainly going to become a problem. Reducing the imminent risk seems to make the most sense but this choice is not reflected in a risk burndown chart. Simply reducing risk over time is not enough. You have to reduce the right risks.

Impact Isn’t Really About Money or Effort

The only way for a visual chart such as risk burndown to work is if we’re able to quantify risks. This is generally done with exposure. Exposure = probability x impact. Impact is a funny thing. Impact is an assessment of how much the consequence of a risk will affect the project if the risk becomes a problem. Traditionalists like to think about this from a money perspective (which makes sense since software engineers stole most of our risk management practices from the finance world, originally anyway). For small teams, effort is a better measure as in the number of person days a risk that becomes a problem will cost to fix. This is a quantifiable loss.

There’s a problem with thinking about impact in terms days of loss. Since not all risks are created equal, not all loss is truly equal either. Some kinds of loss can’t be measured in terms of effort. It really all depends on your project’s threshold of success. Some example risks (which don’t rely on ye olde life-critical system standby) from which you might never recover if they became problems include:

• We don’t have a reliable backup solution; might lose all of our project data. (Lost yer data? You’re up a creek, son!)
• We don’t have backup power for our data center; data centers might go offline for more than a few hours. (How many days will it take you to get those customers back?)
• The demo has bugs and our contract renewal is based exclusively on how much the client likes our demo; a bug might occur during the demo. (HA! HA! You don’t have a job!)

In all of these cases you would reduce the risk by working on attributes other than impact (e.g. reduce probability, eliminate the condition, extend the time frame). Enough said. When it comes to calculating exposure, each of these risks has a catastrophic impact. That’s catastrophic, short for epic failure. No amount of days can really capture the essence of complete catastrophe.  Impact works best when considered in terms of success, not days or dollars lost.

Forget Risk Burndown

I want risk burndown to make sense, but given the problems I can’t help but think of it as a meaningless metric. Sure, some risks will be reduced and some will go away by converting into problems or being overcome by events. And a chart showing this would be really neat. But you’ll also uncover new risks as the project goes on. And some risks are just not worth caring about while others deserve a lot of attention. Risk management is about identifying the things that are most likely to kill your project so you can deal with them before it becomes too expensive (or impossible).  A burndown chart doesn’t reflect any of these things directly.

Burndown masks project risks too much and gives teams a false sense of confidence. To put it another way, there’s a risk with using risk burndown:

Our new risk management strategy assumes our estimation precision is better than it is; we may not mitigate the right risks.

Exposure is a ruse. And risk burndown is a metric for showing a reduction in exposure over time. To wax poetic, perception is reality and risk burndown provides a false perception.

That said, any risk management is better than none at all.  If a risk burndown chart helps to get your team thinking about risk, then so be it.  But there are other ways (might not be as fancy) to manage risk which are easier and more effective.

I was recently asked by a coworker how software risk management is different from traditional SWOT analysis. SWOT is a technique commonly used for strategic planning where the strengths, weaknesses, opportunities, and threats facing a group are compiled and analyzed to determine an appropriate course of action. Software risk management (as defined using the continuous risk management paradigm from the Software Engineering Institute) is similar in that risk management can be used for strategic planning but risks yield much different information which is applied in a very different way.

The first step when performing a SWOT analysis is to define the business objectives. This is very similar to defining a threshold of success in software risk management. The main difference is a business objective takes the form of the desired end state whereas the threshold of success is the minimum objectives necessary for the project to be successful. For example, a perfectly valid business objective might be to deliver all 100 story points by the end of the year while the threshold of success might be to deliver the core functionality (worth only about 50 story points). Would more stories completed be better? Of course, but what if you end up only completing 75 story points by the end of the year? How did you do? You missed your goal, but you still succeeded right? It’s difficult to tell without understanding the difference between wants and needs.

The main part of a SWOT analysis consists of a group session where strengths and weaknesses internal to the group and opportunities and threats external the group are identified. People like to put SWOTs into a 4×4 grid so it’s easier to look at. While there is some great advice out there for understanding what goes into a SWOT, the analysis is largely subjective, relying on a teams’ gut feelings to know the strengths from the weaknesses, the opportunities from the threats. Software risk management can be a much more systematic approach to understanding the potential dangers that face a project based on known facts when tools such as the SEI’s Taxonomy Based Questionnaire for risks (pdf) are used. Guts still come into play, but there is enough engineering in place to help people make the right decisions.

Risks are specifically actionable – depending on the risk you might be able to mitigate it by manipulating the timeline, impact of the consequence, probability of the risk occurring, or by addressing the condition. You might transfer the risk to someone else or simply accept the risk. SWOT by itself is merely a collection of statements relative to internal or external entities which may or may not actually be true. Are you good at testing? How do you know that? Is Bing really a threat to Google Search? Should you do anything about your weaknesses? Will they prevent you from achieving your business objectives? Without further analysis there really is no way to know and other than prioritizing there really is no way to analyze a SWOT, nor is there any clear direction for next steps.

Look, when planning a project you really need both SWOT analysis and risk management. SWOT is a tool for assessing capabilities while risk management is a tool for assessing the likelihood of success. Each technique serves a very different purpose. SWOT is most useful at the beginning of a project to help you figure out what you’re doing and come up with an overall strategy. Risk management, though is an ongoing activity that makes sure you don’t fall flat on your face in trying to achieve your business objectives.

When I was a kid my brother and I used to play a game called Make Believe. My favorite variant of the game was simple. Together we would build some kind of fortress and then one person gets the fort and the other person tries to invade the fort. In theory, the game ends when the fort has been overtaken by the invader. What made the game fun was that as the invasion began, the rules of the game always changed. The first thing to go was any notion of death. If one of us was “killed” in battle then near instantaneous respawning was created. Shortly after that we skipped respawing and simply became invincible. Soon the fort became invisible which means the invader just has to run around trying to find it. Sometimes someone gained super strength or the ability to force other people to move in slow motion. We almost always created super weapons (such as a hand held Death Star) which for some reason could always be defended against. Nearly every game ended in tragedy, someone crying or upset: “That’s not fair! You can’t do that! I’m invisible! You can’t do that!”

Kid’s stuff right?

A lot of software projects with teams made up of working adults still play this game. The scenario goes something like this. A team is put together to build some software. Neither the clients nor the team talk about the objectives of the project other than building “some software”. After a few months, something goes wrong or someone doesn’t like what’s happening so someone changes the rules. Before too long, one side or the other is upset that they can’t win, somebody throws a fit, and goes home. Instead of summoning invisible armor, software projects change the rules by cutting features, adding more requirements, moving due dates, wasting resources, and things like that.

We make believe that we’re software engineers.

While Make Believe was a fun game as a kid, changing the rules when there’s real money on the line isn’t as fun. My brother and I ran into problems as kids because we got the objectives of the game wrong. Actually, there were no common objects, which is why we could change the rules so easily. The same thing happens on a software project when the objectives aren’t well known.

Defining and committing to a clear picture of success establishes the common ground rules for a project by making the basic project goals explicit. The technique is known as Threshold of Success.

Defining What Success Looks Like

The Threshold of Success for a project is the minimum set of conditions that must be met for the project to be considered successful. If the team fails to meet even one of the conditions then the project is a failure. A good Threshold of Success is made up of about 3-4 SMART goals (no more than a few bullets on a single PowerPoint slide). SMART is a mnemonic which stands for Short/Specific, Measurable, Achievable, Relevant, and Time bound.

Some other pointers for defining a Threshold of Success:

• The Threshold of Success should be built as a team. Since this is the measure by which you will define success or failure, everyone on the team must buy into it. If you can include your client that’s even better.
• Threshold of Success goals should be challenging, but it’s important that they are achievable. If the goals are too easy, victory will be meaningless, too difficult, elusive.
• Once the Threshold is established, don’t change it! The only reason to modify the Threshold of Success is if the project has changed so drastically that the Threshold no longer makes sense (for example if someone leaves the project).
• Revisit the Threshold of Success regularly (a good time is when planning iterations) so everyone remembers what success looks like. Put it on your team wiki so that it’s readily accessible.
• Be sure that the goals in your Threshold are SMART! The point of defining a Threshold of Success is to take away the wiggle room for defining what it means to succeed or fail. The goals you define should make this black and white. The more specific the goal is the better.

Building a Threshold of Success

The easiest way to create a Threshold of Success is to first create a minimum picture of failure, then convert failure into success. Here’s an example:

Failure for my current project might look something like this.

• Essential features are not ready by the end of the second quarter.
• Team members are dissatisfied or bored with their jobs.
• Newly hired team members don’t feel like they’re part of the team by March 31.
• There isn’t enough money to continue development after this fiscal year and we have to fire people.

Now that I know what failure looks like, seeing success is easy. I don’t want any of these things to happen. The threshold of success for my current project might look something like this.

• By the end of the second quarter, all “Must Have” features are implemented and pass acceptance tests with no known critical defects.
• All team members give average score of 5 or better on a job satisfaction survey taken quarterly.
• By March 31, the team has successfully executed at least three team building activities with all team members present.
• Funds of at least $1 million are secured by December 31 to allow for future development without a reduction in team size.

Notice that only 1/4 of the success goals in this example are related to software functionality. While goals might come from anywhere, teams traditionally focus on goals related to people and relationships, process, resources (such as budget or schedule), and product (software functionality and quality).

As this technique originated with the Software Engineering Institute (pdf), nearly every studio team in the Carnegie Mellon Master of Software Engineering program creates a Threshold of Success for their projects. The MSE Studio Archive has extensive examples of both good and bad pictures of success that teams have created. The Square Root Team’s threshold (my team) is a good place to start, but there are plenty of other examples.

There might be many goals for a project. In the Team Software Process you actually identify at least three different kinds! But there is only one threshold of success for a project. Knowing what success looks like gives you a better chance of actually achieving it.  Without it, you’re just pretending that you know what’s going on.