Return to the Dreaded M...

You’re being spontaneous. You buy a huge collection of tools. Everything from hammers and saws all the way up to clamps, shovels and drills. You order in tonnes of concrete, wood and drywall. Perhaps you even throw in a bit of re-bar -- those steel rods used to reinforce concrete -- just for extra strength. With all of your energy and a few of your closest friends you hit the dirt and start digging like mad. Dig, dig dig, and then you pour the concrete. Creatively, of course; in various spots around the excavation. A few truck loads at least if not more. A bit here, a bit there. Your not sure yet were it really belongs, but you'll figure that out as you go. You pound in some boards: big and small. Then it is time to whack up the drywall. So you add a wall here, maybe a window or a doorway over there. A bit more concrete; then some more wood; you keep going. It looks like some stairs might help at this point. Towards the end you toss up some "roofing" areas to keep the rain out. Add in a few shingles and "presto", you have finished house. No fuss, no muss and definitely no plan. Sound's a little crazy doesn't it?

So honestly now: how messed up do you think this house is going to be? I've been fixing parts of my house for decades, but I can't imagine building a whole house from scratch. I'd definitely never try it without having a well thought out plan in advance. It would be insane to spend tens of thousands of dollars on supplies, equipment, and all of that time, if you suspected from day one that the house was probably going to fall down. But you'd expect that, wouldn't you, if you started working without a plan. Even if you were a professional house builder with years under your belt, unless you were following a design, detail-by-detail from memory, things will go wrong. Horribly wrong. A house is a complicated thing, you just can't build one without some sort of coherent design. All of the pieces have to come together for it to work. Built randomly, the building will have an inordinately large amount of problems. Any structure built under the same circumstance would, in fact anything at all built under the same circumstance would have lots of problems, unless of course it was trivial.

If you wouldn't do it with thousands of dollars worth of housing supplies, you certainly wouldn't do it with millions of dollars worth of software development, would you? Or would you?

Virtual or not, building software without a design will result in an edifice that won't stand up to the rigors of time. Other than a simple project, there must be a plan and it must contain a design. Arguing otherwise is not rational, and certainly not a position you want to dig in and defend for any length of time. A design is necessary to avoid wasting time and resources, and in the case of a house, in keeping the structure from collapsing on your head. Designing is a good thing. No one should seriously suggest otherwise.

You need a design, but how much of it do you really need? What about going out there and using a Polaroid camera to capture all of the good bits that you see in other houses. Polaroids are useful because you can almost instantly get a physical hard-copy of the image right after taking the picture. Snap, snap, snap, and you capture this corner, that wall, this part of the window frame, and so on. If you visit enough houses, and take enough pictures of all of the cool details, you should be able to put it all together to form an uber-design for the perfect house, right?

Is it that surprising that your stack of Polaroids holds no more value then just winging it?

An incoherent design, one that doesn't present a complete model of the thing to be constructed can't be validated. If it is not small enough, or not abstract enough to be manipulated inside of someone's head, it can't be validated. As such, there will be problems. The only question is how fatal will they be? The problem with a stack of Polaroids as a design is that the details may all be there but the big picture is missing and the organization is awful. The stack might be enough input to produce a design, but it is no substitute for a well-drafted blueprint of a house. Blueprints are a representation that can be worked from. Small, abstract, but highly detailed in a manner that is quickly readable and easily validated. Content is important, but how it is presented is equally as vital.

So why, then do software developers gather buckets of requirements; the little details, and oodles of screen shots; more little details, and then expect to build a coherent system from that mess. Tossing in more details in the form of a bunch of disconnected use cases isn't going to fix the problem.

Capturing the bits may be crucial, but if they aren't integrated into a design they are useless. Or even worse: misleading.

The bulk of most designs is pretty much 'routine engineering' anyways. It is repetitive, and has been done before. Lots of times, actually. It doesn't serve anyone to iterate out all of the same basic details over and over again. The real design quickly gets lost in the mush. You certainly don't see that with blueprints for a house; the actual design is projected onto a 2D surface where the true spacial locations for things are not explicitly specified, yet there is enough there to correctly build it. That is a key point. A critical aspect.

A design needs to encompass most of the details -- it shouldn't get lost in that 80% of redundant stuff -- but the other 20% must be clear. To be useful, the developers need quick access to the right information at the right time. Should you break out each requirement into mind-numbingly trivial detail, it only slows down the process. By being pedantic we often obscure the details with noise. A massive list of all of the possible requirements for a system fails to work as a design. It it too big to validate, too redundant to pay attention to, and too noisy to fully understand.

As well, I think that fully listing out all fifty screens in the system is not a good design either. It makes more sense to design a couple of 'types' of screens such as a details view and a list view, then match them back to the fifty. In that way, there is a concept (screen type), a high-level view (different types of screens) and then the remaining details; a list of fifty screens and their resulting mappings.

Going at the design in this manner serves several useful purposes. First, it is smaller and more likely that a single human can validate it correctness. A feat that should not be underestimated in usefulness.

Second, it helps to provide the much needed consistency to the overall design. The screens flow, because they were designed to flow, from detail to list and back again. Users love that type of consistency, and getting it into the design early means a better likelihood of keeping it there for the implementation.

The most important reason for handling design like this, is that any and all of the problems at the design stage directly influence the implementation; no design for instance causes chaos. While having too many little bits, such as requirements and screen shots blurs the overall picture with too much noise. We are told to avoid compound requirements, as if they are bad somehow. But if the designers define a huge number of independent requirements, that "mass" of detail is exactly what will be coded, and that coding style will be "brute force". The results will be a big ugly mess of code. So, not only will all of the requirements make the design larger and more cumbersome than necessary, but the implementation itself will follow suit; a problem that threatens the design of the system and the ability for the development to finish on time.

Compacting the design compacts the implementation.

Getting back to house building, it is done with considerable care from a working design. Generally one that is thought out in advance. Designing a house is straightforward and well understood, particularly if you gloss over some of the weirder local conventions. Software development needs an equivalent mechanism for design.

We live in our houses, and renovate them over time. They get extended and modified to suit the various different occupants. The designers could not have or would never have conceived of what would happen down the road, it is not possible. My own house is over 100 years old and has been renovated so many times I could not even guess. The original builders would barely recognize it today, and it looks quite different from the other similar houses in our neighborhood that all started life as the same design.

Knowing that about houses, brings up two corollaries for software: one is that the design is going to change over time, as things change around it. People live in their software systems the same way we live in our houses. It changes over time but that should never be an excuse not to design. You might not know exactly where it is headed, but you do know that it is a house you are building.

That leads to the second point: it is not possible to renovate my house, or any other small structure into an apartment building. Won't happen. If someday that's what ends up covering our land, it is because it came from tearing down our house (and several others) and rebuilding it. A house is a house, and you cannot enhance it to what it is not. You need to design and build an apartment building from the ground up to bear the types of stresses and strains that are unique to it. Often that point is missed in software. It is just that we can tear it down in bits and convince ourselves that we haven't made major changes. Or sometimes, we can choose not to tear it down, even though it would have resulted in better code and less work. Virtual monstrosities may not collapse on your head, but they do come tumbling down.

Still there? Good, the M from the title comes rolling back into this blog entry in the form of stating that a methodology defines the steps necessary to design and build software. The aspects of the design -- what goes into its makeup -- are key to the methodology. The way the needs of the users are collected, processed and carefully laid out in a design should be explicit steps in the process. How the design is packed together, should also spring from the process. If you were working with an architect to build or enhance a house, the "thing" that you are working on is the blueprints for the house. That much is defined by the process of building a house.

A good software methodology needs the same. It should tell you how to produce a design in a way that it will be as simple and compact as possible. The blueprints for the system should be precise, but simplified abstractions that contain the overall view embedded with enough detail. What that actually looks like on paper, is a problem for the methodology, not the software developer.