Controlling Development Chaos

A lot of software developers really hate any comparison or analogy that involves building construction. I think the primary reason behind this dislike is the sense that construction workers are viewed as passive players in the whole development project, in the same way that factory workers are seen as nearly-mindless cogs in a production line.

There is some reason to fear this type of comparison, intellectual pursuits are clearly not the same as physical. The rules are different. For whatever reason, we have a great deal of knowledge on how to refine and maximize the efforts of factory workers, but virtually none on how to do the same for white-collar jobs. Trying to take what we've learned from the physical and apply it to intellectual hasn't been particularly effective.

Still, in this post, I start with looking back at a construction analogy and then moving forward to a more sophisticated, and probably more popular analogy. Even if construction is not a perfect fit, building things is always a lot like building things, and we can always learn a lot from the way other people build things.


THE UNLOVED CONSTRUCTION ANALOGY

While I agree that we shouldn't draw two many parallels between software and construction, there are still things about a construction analogy that are 'truish' enough to be useful.

For instance, a large sky-scraper project contains a huge number of specialists, including carpenters, plumbers, electricians and of course general-labourers. At the same time as well are engineers. managers, inspectors and a whole host of other people with interesting roles in keeping the process, and the building from collapsing. Software projects are similar, in that they often need graphic designers, editors, GUI expects, database programmers, system administrators, managers, tool-smiths and all manner of specialists for handling any sophisticated technologies or domain issues. They are multi-disciplinary projects.

Both software development and construction involve building things and both, to some degree or another, change radically depending on the size of the thing you are building. A shed out back is not the same as a house, or an apartment building. A sky-scraper is a completely different deal. Scale has a huge influence on the size and techniques required to bring it all together. One does not easily jump from building sheds in a backyard to building apartment buildings, it is a different set of skills.

The construction of a modern sky-scraper is an amazing project that is pulled together with a degree of smoothness that software developers can only drool at. If we could build as large and sophisticated things, with the same degree of precision and timing, then we would have been able to get beyond our current hit-or-miss random guessing style of development process.

It is worth mentioning too, that while design and code are fixed to some degree, the job of an electrician on a huge site still requires a significant amount of thinking and problem solving. It is not a clear-cut mindless day at the office. While they may be working with their hands, electricians still need to think about what they are doing, the various plans and codes, and how they are going to make it work in the current context. The same is true for all of the other specialists. It takes more mental effort than just being a body, that after all is the specific job assigned to general labourers.


WHY IT FAILS

But still, even if you give all of that its due, construction just doesn't map entirely onto software development. We know this, it is a frequent discussion. Sometimes, I wonder if it really is a case of us just not wanting to accept it; we have a great deal of freedom, and it becomes hard to surrender that, even if it is for the good of the project. Still, there are strong differences.

The biggest single problem with construction is that it is a one-phase deal. You build the building, and then move in and repair it as you go. When we tried this with our waterfall ideas, we found it doesn't match reality. Code is seems, never gets done and is always rusting. Projects just don't end, and when they do, the code goes out of circulation. For software, there are a nearly infinite number of phases, ending only with the lifetime of the code.

The state of the art of building design has progressed over centuries, and mostly it is only slightly tweaked between buildings. In a sense, construction companies redo the same projects over and over again, they have no choice. This repetitiveness is the root of mastering the process. If you do it enough, eventually you'll get good at it. Contrast that with software, where each design for each system is unique and rarely learns from its predecessors. Because we can clone code, we don't want to keep reinventing it (even where in some cases, that might actually produce significantly cheaper or better code). Code builds up, but our experiences do not.

Buildings stay around for a long time. That aspect, as well as safety considerations bends the choices made towards the less risky long-term options. Saving time or money by not following the process, the code or using inferior parts may help in the short run, but because the life span is so long it opens up a lot of risk of getting caught in the future. Most buildings are built correctly. Software, on the other hand, being mostly invisible and uncontrolled doesn't provide much incentive for programmers to make long-term choices, despite the fact that software projects are always long-term projects.

Another significant different is that workers on a construction site have less degrees of freedom then computer programmers. Their roles and jobs are far more rigid. Right or wrong I don't know, but this is a key issue that I want to get back to later. We've always had our freedoms, and they have always been a problem.

For all of the differences, there is also a lot that matches, but not enough that we should try to emulate construction companies, although we should admire their skill and organization. But it is exactly that line of thinking that sent me in search of some other type of better suited analogy.


GOING BEYOND CONSTRUCTION

As often happens with me, I get a little down the road of a thought, and then it gets left behind for a while. In particular, I dropped my pondering of construction analogies so that I could do my usual Friday night routine, which was watching a film with friends. The film -- which I can't remember -- had finished, and we had moved on to the DVD special features.

As the director and actors mutually patted each other on the backs and proclaimed their love of working together, a little thought was brewing in my head. A film, you see is a large complex project, that also brings a large number of multi-disciplinary professions together in order to create something. Most people don't quite see it as the same thing, but if you look at the effort and money that was poured in a mega production like Lord of the Rings, you start to see some similarity to these huge buildings we keep throwing up in our cities. But, it is hugely different.

We deem a film as art, and the director, as an artist. We see most people involved with film as artisans. Even thought, in the end, there are also carpenters, electricians, plumbers, casting agents, cooks, boom operators, grips, computer programmers, and a huge huge host of other professionals involved.

What's interesting with a film, is that even thought the script is often written by one or more writers, and a huge number of producers and managers are involved, the films themselves have always held the particular 'stamp' of the director. In a very real sense, even with all of the 'creatives' vying their wares, the director of a modern major feature film, gets and sets the final 'vision' under which the film will be created. You can tell a lot more about a film by its director, then you can by its writers or actors. In many ways, the director of a film, is far more significant than the architect of a building. An architect shapes the design, but the engineers ensure it is built correctly. A film director often has no such constraints, at least not ones that are 'that' objective (budget excluded). They control the vision.


ANOTHER ANALOGY

But if we go back to the special features, even with all of the influence, the staring actors still often talk about how much artistic freedom they were allowed in playing their parts and contributing to the film. In that way, a good director stays true to their vision, but not necessary at the expense of making all the actors just mindless pawns on a film assembly line. Most films are collaborations of many artists, yet remain true to their director's goals.

So if I am looking for a more realistic role-model, I probably want to be more like a director of a modern film when I am leading a big development project. In that way, I want to have my vision implemented, and I want the final product to clearly bear my stamp, yet I want my fellow developers to be able to contribute to the overall project in a way that makes them feel honored and proud to have been on that particular development team. Like the actors, the coders should feel that they were able to make significant contributions, but without disrupting the overall product.

I do at this point, want to warn programmers about thinking that I am saying they should have some infinite degree of freedom while developing software. Not even the directors have total freedom. There is a difference between creative contributions and just plain old chaos. The requirements of the users, the limits of the technology, and the consistency of the conventions all combine to remove a great deal of the degrees of freedom from coding.

For tools to be useful, they must have certain specific functionality that clearly matches a problem domain. You can't creatively work your way around the actual problems, they are what they are. Our existing technological base is awful, but it is the only one we have. If you have to bend your design for a rigid relational database schema in order to allow for reporting, then there is little choice in that matter. And if every other similar application uses widgets in some annoying brain-dead, but extremely common way, you too must follow that or pay the consequences of annoying your users instead of impressing them. A good software product, one that solves a specific problem is heavily constrained long before it is even sketched on the back of a napkin.

In all, if you are implementing a targeted solution for a specific industry, there isn't an infinite array of possibilities for the development. Like a 'genre' in film making, if you break the rules, you had better understand them first, and given a solid reason for breaking them, or people just take you for someone that doesn't get it.

A film like Lord of the Rings, for example, while intensely creative is also highly restricted. Frodo can't be green, and you can't redefine elfs back into funny little forest creatures. The film would flop if it violated the spirit of the books. In the same way, software is constrained by its intended targets.


FINAL CREDITS

In many ways software is unique, but in many ways it is just another method of building things. Our biggest difference is the immaturity which lingers in our development process. We clearly do not want to grow up and start producing better software. But the issue might have been one about what our newer 'maturer' self would really look like. With terms like architect and and processes like waterfall, we've been quietly, but agonizingly following along behind various construction analogies, but with little effect toward reducing the 'software crisis' first identified in the late sixties. Is it any wonder we are resistant to change? Is there actually anything to change to that will really work?

I think if we really don't want to be factory or construction workers, then we need to look carefully at other groups of skill artisans, and start trying to fit their processes into ours. A director for a major motion picture, and his teams of 'creatives' is such a great parallel because it involves a large group of diverse people putting together a big complex work. Its main weakness is that a film is a short-term project, once completed it is over. Because of that, most choices of how to get the work done are not made for the long term. If we did that, it would be a disaster in software. Still, mixing this type of analogy with a construction based one, which is a much longer-term vision, produces something in the middle. We are neither, but we could be both.

I'm probably late with this analogy, as I suspect given the credits on the back of most video games, that at least one segment of the software industry has been quietly following this analogy all along. Still, a unified single vision is the key to bring out high-quality products, and a film director does it better than any other industry. Major motion pictures, at least according to the DVD special features, are fine examples of creative and organizational projects. We should marvel at how they work, and how they manage to bring together so many diverse disciplines into one final amazing product.

Next time someone asks me about what title I want for leading a project, it may be something like: commercial software director. Can an Oscar be that far behind?