Reflections on Software Engineering

Software developers are, in their heart of hearts, dataphiles – people who are absolutely in love with data. When was the last time you had a passionate discussion about frame rates, hardware benchmarks, gadget specs, sports statistics, dungeons and dragons, the merits of high def…the list goes on. Face it, you love data. You love comparing things using data. You don’t feel comfortable making decisions without a comprehensive comparison of data.

Why then do most software developers treat software development differently?

Tom DeMarco recently brought his own famous quote into question (pdf), musing that not only is it possible to control what you can’t measure, but the most important stuff you need to control on a software project is impossible to measure. Once again, DeMarco is wrong (in my opinion anyway).

Wikipedia is one of the most controlled projects on the planet

One the surface, Wikipedia is the Wild West of online content. Not only can anyone edit any page, but content from Wikipedia is widely proliferated in the media and (sadly) school reports. Wikipedia is the single greatest success of user generated content in the history of mankind (”The Internet,” as the medium, doesn’t count). What started with a dozen humble articles has evolved into the most comprehensive encyclopedia ever created and includes everything from the fundamentals of science to the definitive source on Babylon 5.

What folks seem to forget is that even in the Wild West, there were laws and there were lawmen. Though we love to think romantically about such brigands and gunslingers as Jesse James, Billy the Kid, and Butch Cassidy, most stories about these historic figures are greatly exaggerated. So too is the case with Wikipedia.

Let’s take a closer look at the Wikipedia entry for Billy the Kid. This article belongs to a number of internal WikiProjects, visible from the top of the article’s talk page. The WikiProject Biography is not unlike most projects in Wikipedia. There defined processes for assessing articles, conducting peer reviews. There are rubrics defined for assessing the quality of articles within the project. People even take on specific roles and responsibilities within the project. The collection of processes and information serves as the main means of coordination for content contributors and helps the group control articles within the scope of the project.

The WikiProject Biography even collects metrics on articles which it then uses to make decisions concerning the articles under the project. The metrics are derived from quantifiable data and help control the project.

As it turns out, Wikipedia is not the lawless territory of the internet it has been made out to be.

You can measure the immeasurable

Wikipedia works because people were able to figure out ways to measure things that usually can’t be measured. The fundamental principle that many people overlook is that binary is a metric too. Yes or no questions can be just as effective a measure as any complex metric. Did everyone fill out their task data today? Yes or no. Did the estimate match the actual? Yes or no. Did the test pass? Yes or no. Is the project done? Yes or no. Have we identified risks? Yes or no. Has this risk become a problem? Yes or no.

At the heart of every complicated metric is really a series of yes or no, binary questions. When considering whether the project is done, you have to define done. One way of defining done is in terms of a checklist. Is feature 1 done? Is feature 2 done? Defining done for a feature could be as simple as checking whether all the tests have passed for the feature, again a binary measure.

For more subjective assessments, you can rely on observation-based, experience-defined rubrics. Does the team get along with one another? In the simplest form, this could be a binary metric (Am I friends with everyone on the team?) but it could also be more complicated relying on gut feelings and a guiding rubric (”we never hang out together and don’t trust one another” might indicate low harmony while “we hang out often and feel comfortable sharing personal stories” could indicate high harmony). Teachers use rubrics and experience to judge subjective assignments everyday. The difference is that they slap a grade on it and send it home as a report card.

While DeMarco is correct that many of most critical things in a project are the most difficult to measure, it is possible to create measurements if you feel it is important enough to do so. How would you assess whether you have a good architecture that solves the problem at hand? Rubrics might play a part but so too might binary gates based on quality attribute scenarios or intricate observations concerning design trends over time. If you think hard enough, you’ll find that it’s extremely easy to find measuring points for nearly every aspect of a software project.

Whatever you do, don’t become a mindless, data-driven robot

I love data and I know you do to. While it’s tempting to inject data collection and derive metrics for every aspect of a project (because it’s fun and informative!) don’t. Collecting data and calculating metrics can be expensive. Not so expensive that you shouldn’t use it, but expensive enough so that you shouldn’t use it on everything. I like to compare using metrics to eating out at restaurants. Once or twice a week isn’t that big a deal, but it’s not something you should do every day if you’re trying to watch your budget.

DeMarco is right about one thing: control is not the end-all-be-all of software engineering. Consider carefully, what are the most risky parts of my project? What are the parts of my project that even require control? What are the parts in which I need more insight or want to improve? Strategically develop metrics for these areas and don’t worry about measuring the rest. Trust me, the world won’t end. If you don’t know what you’re doing, start with a simple binary measure. And above all, if something isn’t working, change it.

I was extremely disappointed to read a recent article on Coding Horror reflecting on an IEEE editorial written by Tom DeMarco. If you have not already, please read Tom DeMarco’s article now. It’s only two pages and it’s well written.

With all due respect, Tom DeMarco is wrong.

And Jeff Atwood made things worse.

According to Atwood’s interpretation of DeMarco, since we can’t control software projects, there is no sense in trying to engineer software.

What DeMarco seems to be saying — and, at least, what I am definitely saying — is that control is ultimately illusory on software development projects. If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.

Atwood’s conclusion simply is not supported by DeMarco’s article. DeMarco made two points in his piece.

1. We don’t have as much control over software as we think we do — even when we can measure the software on which we work.
2. We should be focusing more on the upfront “conception” activities than the areas that currently receive the most attention, construction.

My interpretation of “conception” activities are things like requirements, architecture, and design — details that ultimately help you figure out whether it makes sense to build the thing you think you want to build. By framing DeMarco’s argument as “craftsmanship” vs. “engineering” Atwood misses the whole point and reopens the tired art or engineering debate.

Overlooked by Atwood, DeMarco never questioned the idea that software should be engineered.

I’m gradually coming to the conclusion that software engineering is an idea whose time has come and gone. I still believe it makes excellent sense to engineer software. But that isn’t exactly what software engineering has come to mean. The term encompasses a specific set of disciplines including defined process, inspections and walkthroughs, requirements engineering, traceability matrices, metrics, precise quality control, rigorous planning and tracking, and coding and documentation standards. All these strive for consistency of practice and predictability.

DeMarco is really saying that the engineering part of software engineering has become overshadowed by a collection of best practices for building software. In my mind this isn’t necessarily a bad thing. All it means is that what has become known as “software engineering” is different than the original definition intended by the NATO Conference on Software Engineering.

But by discounting current software engineering practices, DeMarco dismisses the real engineering that went into advancing the field to where it is today.

DeMarco seems to imply that what we really want software engineering to be–the application of systematic, disciplined, quantifiable approaches to the development of software–and what software engineering has become cannot coexist. Essentially, to reach a state where metrics and measures, quantifiable approaches, are used correctly and consistently by the software development community we must stop using the term “engineering” to describe the current set of practices.

This is backwards thinking.

Engineering is more than something you do; it’s also a way of thinking about problems and solutions. Reaching the point in software engineering that we are today required systematic, disciplined, and quantifiable thinking. Over time results of this thinking have been codified into the set of best practices that most developers now take for granted.

For example, we know that there is a 100x or more difference in costs between defects discovered later in the software lifecycle than earlier. We know that certain practices can effectively remove defects at different costs and at different times throughout the lifecycle (for example, inspection vs. prototyping vs. unit testing vs. system testing). We also know that historical data is an excellent indicator of future performance on software projects.

Systematic, disciplined, quantifiable thinking was required to make these discoveries.

Because of these codified best practices, it is not always necessary to conduct experiments on a project to trust that they are working. I know unit testing combined with regular system integrations will flush certain defects from my software before those defects become a problem during system testing. I know that statistical analysis of collected task tracking data will help me better predict how long future tasks of a similar size will take. It doesn’t matter whether I completely understand the engineering behind the practice or whether I simply follow the process or use the tool. The benefits will be the same.

Does that make me less of an engineer? I don’t think so.

Using best practices codified as processes, methods, or tools on a software project means you are engineering software whether you like it or not. With many of these practices, the control mechanisms are already built in so you don’t realize that you’re already controlling your project. As DeMarco points out, it simply isn’t necessary that every engineering detail be painstakingly scrutinized for a project to be successful. For many projects, the essence of the project is sufficient to overcome the accidents encountered when engineering by coincidence.

But just because you engineer by coincidence it doesn’t make you a software craftsman. To prove it, I’m calling Jeff Atwood out. Jeff, I dare you and the Stack Overflow team to take the PSP Challenge. Take a course on the Personal Software Process, honestly give it a try — use actual software engineering for a few weeks — then tell me that software engineering is dead. But don’t knock it until you’ve tried it.

For some reason software engineers and others who deal with software love sitting around debating whether software engineering is more art or engineering. Part of the problem comes from how software engineering came to be, the term assigned as more a challenge by the NATO Science Committee for the software industry to get its act together; one great big label to talk about many things. Software engineering covers all aspects of software development including requirements, formal modeling, estimation, planning, tracking, managing people, processes, designing, coding, testing, deploying, and maintaining just to mention the most obvious required skill sets. Some aspects of software engineering are not heavily engineering oriented and require no mathematical or scientific reasoning such as requirements and managing people. Many aspects can benefit greatly from a mathematical background but math is not required to “get by” such as when writing code, testing, or planning a project. This is unfortunate. Confusion and arguments abound among both the academic and professional software engineering communities creating holes in the professional software world where no talent amateurs defend abominable code and runaway projects under the guise of “artistic expression.” Software engineering is an engineering discipline.

I’m not saying there isn’t a creative component to software engineering. In fact, there is a huge creative component and creative problem solving is essential for building great software. I’m also not saying that code and the resultant software can’t be aesthetically pleasing. It absolutely can and should have good aesthetics. I enjoy reading well written, elegant code and always do the best I can when writing software myself. What I am saying is that art is more than just creativity and elegance and in almost all cases, software falls well short of art. Since the vast majority of software is not art, why should the people who wrote it be considered artists?

My neighbor upstairs is an artist in the traditional sense. He writes and plays music, he paints, he makes stained glass, and he’s a professor of eurhythmics (more specifically Dalcroze Eurhythmics) at Carnegie Mellon University. As you can imagine, he’s a really cool guy. He’s actually one of a handful of eurhythmics grand masters in the world and attended the world eurhythmics conference in Tokyo this past summer. From what I gathered from talking with him, eurhythmics is a way of studying art through the body’s motions. The most obvious application is music and dance but it’s not too difficult to imagine a painter creating a masterpiece; moving with the flow of the painting as it is created.

Some have even tried applying eurhythmics to study other, non-traditional art forms such as architecture and even mathematics. He explained it to me something like this. A brilliant mathematician working on an elegant proof might be considered more of an artist. But only someone truly brilliant who is in the zone, able to see all the connections of the work with unbridled clarity, tuning out the outside world (much like flow when programming), with body and mind working as one to solve a proof might be considered to be making art rather than doing math. It’s an extreme extension of eurhythmics theory, but it seems plausible.

The argument for treating software engineering as an art form is similar. Writing software could be considered art but only in the hands of truly brilliant software developers (Paul Graham calls them hackers) and only when those truly brilliant software developers are in the zone, building something amazing. Everyone else is just writing code. Much like how no one would consider multiplication tables artistic, anyone who considers churning out code without intention or understanding of what is written as art is kidding themselves. Looking at software engineering from the perspective of eurhythmics lets us state this more as a fact than an opinion.

Making this distinction is important because treating software like art and software engineers like artists is damaging to the industry. Only the greatest, most brilliant developers might ever have the ability to build software as artists. Everyone else is just trying to get by, much like the many thousands of starving artists who are never discovered, never hit it big, or whose muse never makes it around for a visit. Treating software like art allows developers who don’t have the artistic chops to have the excuse of creating poor quality software that is not fit for the purposes of its creation.

Fitness for purpose is another key distinction. Art exists to evoke emotion but software has a utility. In many cases, software is too important for it to not work correctly. Approaching software development from an engineering perspective creates an environment where it is possible for the rest of us to create working software that is still fit for purpose. Again, I’m not saying that software can’t be have good aesthetics, it should, I’m simply saying that a systematic, critical, methodical, mathematical, scientific approach to building software is required for most people to succeed – much like how architects need a solid foundation in physics and engineering to build beautiful buildings.

I like the idea of software as art. Artists are cool and thinking of myself as an artist allows me to justify extravagant behavior that, as a creative knowledge worker, I know I need. On the other hand, I rarely care about using software as a medium for creating an emotional experience for my users. I only really care about creating something great that helps users kick ass while accomplishing whatever it is that they want to do. Following sound engineering principles helps me meet this goal consistently.

Building software is a creative endeavor and the engineering part of the field is still very young compared to other engineering disciplines. But treating software engineering like an art form is a copout, an excuse for ignoring the sound engineering practices that do exist because they’re difficult or boring or unappealing. There are important creative components to building software but that shouldn’t hide the fact that proven engineering practices can and should be used too.