The Engineering of Software

The last thing I ever wanted to be -- twenty-six years ago -- was an engineer. When I started university I had heard that 70% of all engineers hate their jobs and that they were often bored at work. Boredom sounded horrible.

What I wanted, was to master the black art of coding. To follow in the foot steps of those early magicians I had seen in the computer magazines. The ones that crafted weird cryptic concoctions to do really neat things to their machines. I wanted to be a hacker.

It’s funny how time and experience temper one’s thoughts; how your actions change once you really understand the consequences; how much you grow once you see the larger picture.

Development shops span the range between the cowboys, who are just hacking at the code furiously, and the bureaucrats, that are so ground down in organizational paperwork that little actually gets accomplished.

In my younger days the cowboys where fun. It’s a dynamic environment. Things are happening. And they are happening fast. Well, OK, there are a lot of needless problems cropping up, but hey, that just adds to the excitement, doesn’t it? You could leave work feeling like you got something accomplished.

Well, that lasts for a while. Until you start to get caught in the same hollow accomplishments, over and over again. That is, you’ve fixed the same bug before, wrote the same code before or stuck a band aid on the same gaping wound before. Doing it once or twice is fun. Doing it repetitively, while giving up your life to endless overtime, starts to become depressing after a while. Then suddenly one day you find yourself looking down at the growing pile of band-aids before you and you start thinking “isn’t this all just a waste of my time and my abilities?”

Early on I moved away from the cowboys. They were driving me nuts. I landed deep in the heart of a reasonable engineering culture. Some people on the outside might have confused that development process with a classic waterfall one, but it wasn’t really. Although there was a significant design process up front, the last of the details were always worked out in the various iterations. It was actually a healthy mix. Some thinking, some paperwork, then a hard run at the code. Some testing, a release and then back to thinking again.

Later when I set up my own shop, it naturally started as chaos. Startups are sensitive to cash-flow problems so sometimes the only way to deal with that is an ugly short-term hack. But as time progressed -- every time I got the chance -- I applied long-term fixes. These gradually paid off and when I finally moved on, the place was humming along smoothly releasing a steady stream of mostly high quality code and the occasional patch within a day or two of finding a bug. Each development cycle was a bit easier than its predecessor. Everything was tracked. We had managed to built a big application. One that was far larger, and far more sophisticated than you would have expected from such a tiny shop. I learned a lot in those years.

I found that it isn’t all that complicated. If you boil it down again and again, software development keeps coming back to its roots in engineering. Not in the classic sense that I was afraid of when I was younger -- the large, slow, boring, mindless process -- but rather in the sense that as the work progresses, developers like myself needed to be hacking less, guessing less, and understanding more. The project may start with a lot of unknowns, but at some point it always gets down to routine engineering. That is, we should be doing the ‘right’ things that work because we know that they work, and fixing the little problems -- permanently -- before they become big serious ones.

My earlier fears were misplaced. Engineering itself is not boring. I’m not even sure where I heard that statistic about unhappiness. Engineering, particularly when it comes to something like software, is simply knowing what works and applying it correctly so that the project is successful. It isn’t hack and hope, it isn’t wild guesses, and it isn’t a spin job to cover up the fact that the system is horribly broken. No, quite simply, it is the path to dreaming up cool solutions and then implementing them as cool solutions.

Yes, there are boring times. There is a lot of digging to see what is already out there. No sense reinventing the wheel especially if it’s already decades old. Note that that doesn’t mean you can’t recode the wheel -- making a better one -- but rather that you don’t try to rethink the wheel, that work is already done and accessible.

There is sometimes paperwork. But if one focuses on documentation that’s useful, rather than on the documentation a management consultant would want, then it’s not so bad either. It helps to refine ideas and highlight missing corner-cases long before they become too serious to fix. A little design goes a long way and any organizational chaos, at any level, is always going to grow into a serious problem someday. What you’re missing, or not thinking about, often becomes really obvious when you try to write it down. It also helps you to understand the vocabulary of the problem and to empathize with your users (if you can’t easily explain it in a document then the users aren’t going to be able to understand it either).

There is a lot of thinking. And ironically, although programmers preach heavily about their need for freedom, many choose to turn off their minds when they code, resorting to as much brute force as possible. But at the end of the day, code is just the by-product of thinking about the solution, so it is only as good as the thinking that went into it. To build sophisticated things you have to have a deep understanding of the problem and you have to spend a lot of time working through it. Thoughtless code is dangerous code because of it lacks intelligence. It just causes stupid problems.

And of course, if you’ve really set a firm direction and you know where you are going with the development then the rest of it is just hard work. Little issues have to be sorted out, details need gathering, but the main flow of the development can be worked out in advance. The project may drift but either the long-term perspective still holds, or it is rethought again to insure that it is still valid.

I remember reading decades ago in Scientific American about how the pharmaceutical industry moved from art, to science and then on to engineering. The pills we pop these days are highly likely to contain exactly what they say they will. That wasn’t always the case. It took them a long time to go from alchemy to engineering. The article said that software also started as an art, and has some scientific basis, but it will eventually follow the same path to engineering. I remember the article because I figured I’d bail when that happened. Sounded boring. But these days, as I am surrounded by more software with increasing complexity and decreasing quality I realize that what I want most right now is to get a whole lot more of the engineering mindset into the software development industry. It is time, I think, that we started taking our profession seriously.

There is still room out there for the hackers and the cowboys. Sometimes a band-aid is the best choice, or the solution is so small it doesn’t matter how much of a mess it is. For quick one-offs I think hacking  will always be common, but when I go out and rely on someone else’s code, on their thinking and on their intelligence, I would like to know that they weren’t just mindlessly flailing at the keyboard or that they skipped their homework and just tossed it together. It makes a difference. I don’t want to build on a house of cards. I don’t want to find out later that I’m dependent on somebody who didn’t really ‘get it’ at the end of the day. I don’t want that degree of instability in what I am doing. I can’t build great software on top of a mess; dealing with other people’s mistakes eats through my resources like a knife cuts butter.

So these days I am all in favor of us seriously moving into engineering. Not doing so is a major obstacle that has prevented software from living up to its true potential. I think we can do it, and do it in a way that it is not boring and it isn’t just paperwork. We can still build quickly, but we can do so from a place of understanding, not one of panic. It’s about time that software development grew up and became a real profession, one that is taken seriously. Even in our deplorable state we are changing the world, so can you imagine what we could achieve if our collective works didn’t suck?