Layering, Architecture and Convergence

The central problem in software development is managing complexity. This complexity comes from the sea of details and decisions, large and small, that need to converge into a stable, working system in order for it to be successfully used for its intended purpose. That not only includes the code, but also the analysis, management, documentation, packaging, distribution, training, support, configuration and operations that inevitably precede a user being able to utilize some functionality on a computer.

Unchecked, rampant complexity growth in any area can swamp a project, making it impossible to continue substantial progress. About the only way I know to contain complexity is by breaking it off into manageable chunks and getting it encapsulated into well-defined sub-problems. This works at a base level, but new problems emerge. Overlapping sub-problems create redundancies which add significantly to the complexity. This can be managed typing and categorizing the sub-problems carefully, so that overlap is rare or negligible, which gets one past the first hurdle. But as the number of sub-problems grows, their inherent complexity needs to checked as well. At this upper level, they just form a slightly higher, more abstract version of the overall problem, but all of the same issues apply.

Thus, at some point in order to manage the complexity another higher layer becomes necessary and the same issues repeat over again. As the complexity grows, the number of layers to manage it grows as well. Thus, in any attempt to manage growing complexity, a steadily increasing number of layers will need to be created. This is the inevitable consequence of our existence, we can’t handle unlimited sized problems. Smart people or well-running teams may have higher limits, but there is always some limit, and it is always reached earlier than expected.

Layering occurs in all aspects of software development, but for this post I’d like to just focus on how it effects the code, data and operations of software.

The first and most obvious layer in software is the underlying code itself. Programmers get presented with a number of domain or technical problems for which they assemble a large number of instructions for a computer to follow. It is well understood that failure to organize these instructions results in ‘spaghetti code’. That is code with a logical structure so intertwined it resembles the noodles in a plate of spaghetti. Of course this is bad, because any attempt to alter the code is hampered by a significant wall of complexity. From this, within the context of coding, we can draw the conclusion that dis-organization increases complexity.

As the code base begins to grow, it becomes necessary to break it off into more and more sub-problems to manage it. This layer is most often referred to as system’s architecture. It introduces conceptual groups like modules, sub-systems, components, etc. Each of these pieces encapsulates a larger subset of the functionality, and the interactions between them gets formalized. Although I’ve never seen it mentioned, dis-organization at this level can produce a spaghetti architecture, one that impedes both changes and overall stability. Experience coding in the base layer does not easily translate into understanding the organizational issues at this layer. They appear similar, but they are fundamentally different problems.

Once a system has grown out of its base components, it is often desirable to widen its functionality to a larger group of developers. This creates essentially a platform layer where the core abilities are exposed, and controlled, allowing for the existing work to be leveraged for more divergent purposes. Once again, failure to organize this layer results in more pasta, and of course understanding the dynamics of the necessary organization is not directly related to the layers below.

Continuing upwards we get to the product or system layer. This layer generally introduces a new set of problems because it is driven more often by the desires or needs of non-technical people. Organization at this level comes more as ‘paths’ that are followed serially or in parallel, but they still need the same generalized principles of organization. Experience at this level generally involves far more people and negotiating skills, than technical ones. A product manager for a commercial system for instance is balancing the technical issues against the direction and current of the market place.

The final layer that I’m going to discuss is the one that is completely ignored by most technical people. For users to access the system, it has to be running somewhere. In an effort to contain the complexity of design and work most systems form vertical silos. That is they handle a very specific problem and don't easily inter-operate with the other systems being run in their environment. There are some interconnections, but standards are weak because they interfere with individual goals or profit margins. As such, most modern large organizations have a large number of these silos. Like every other layer, redundancies cause significant problems. Thus, at an operational level, all large organizations need an organizational scheme to coordinate the building or purchasing of these silos into a manageable affair. This drives concepts such as the common enterprise software ‘categories’ like CRM and CM, which implicitly contain the primitive building blocks of the operational environment. Again, dis-organization at this level results in an enterprise wide plate of pasta, which diminishes that usefulness of the tools.

No doubt there are many more layers, since they are dynamic and unlimited in number. For any given layer, as it gets better organized we can say it is ‘converging’. By this we mean that in some overall way, its base complexity matches closely to the underlying ‘necessary’ complexity of the layer. If it’s not managed then it diverges, generally as a result of disorganization, over-engineering, over-simplification or any other amplifier that isn’t fundamentally inherent to solving the problem.

It should be noted that there is a subjective (human-based) component to minimal complexity. That is, for reasons based on the way individuals think, the lines between over, under and minimal complexity tend to move around. One person’s perfect trade-offs can appear overly complex to another person. Thus rather than a fixed point, a layer will generally converge within a range, particularly if there are many people involved.

Thus in software development, in order to manage complexity, we want to see the overall system converge on a reasonable number of layers, while each layer is converging on a reasonable system of organization. Development is an ongoing progress, and thus so is converging. We simply need to track all of these ‘paths towards organization’ and continually insure that they are headed for the correct goal. At the bottom layer that is accomplished by programmers pushing themselves for elegance, while at the upper layers is it often architects or managers trying to place some organizational constraints on the chaos below. Every layer needs to get close in order to win.

Contradictions

Programming is easy. You just assemble large lists of instructions for a computer to follow. And these days there is an abundance of information out there about which combinations of instructions will produce the best effects. If you can’t figure it out, there are plenty of people to ask.

But programming is hard, because there is never enough time to assemble all of the instructions yourself so you end up relying on other people’s lists. And often, the behavior of their collections doesn’t match your expectations. They skipped functionality or built it weird, or just hacked it together without thinking.

But it is easy because there are plenty of the stable ‘components’ that have been around for years or even decades. There are loosely-followed standards and although some are quirky, once you understand how to use them properly they’ll do what you expect. It may take a while to find the ‘grain’, but once you know how to utilize something, you can make it work for you.

But then it’s hard because the more stuff you build on, the farther away you get from what is happening underneath. This detachment often causes people to over-simplify their understanding, leading to invalid assumptions about the way things really work.

But it’s easy again because there are plenty of conventions, knowledge and advice out there to follow. Sometimes you have to dig a bit, but it’s out there somewhere. At some point, someone has delved into the details and has spent time explaining them. It just might take a while to find and assemble that information, but if you take the time to learn how to do it properly, you can skip a lot of the frustration and pain.

But then programming is hard, because it often attracts people who want to utilize it, but they don’t have any real experience or patience to learn. They over-simplify the amount of effort involved, and think that short-cuts will get them there faster.  Looking at a screen, many people think that the work involved is close to what they see. But it’s in what lies beneath that all the details and complexity brew. The screens are the easy part, particularly if you follow the existing user interface conventions.

But it can be easy. If you spend the time to learn it, then do it properly, and gradually work through all of the issues one-by-one until they are complete. Eventually you’ll get there. A good solid organized process with no cheap short-cuts will always result in something that is usable. It may just take a while.

But then we’re back to hard, because to overcome other’s lack of experience and impatience we need to point them to an authoritative reference. Somewhere that explains the basis for getting the work completed properly. There are little clues spread every where; some correct, some wrong. But no ‘building code’. Without that, new people coming in just ignore what was known for decades, without realizing that it’s been worked out already.

So is programming easy or hard? When the people around you accept it as hard, then although it may take a while, it doesn’t have to be painful or a mess. But when the people around you think it is easy, they get impatient and take really bad short-cuts to avoid the necessary work, so it becomes increasingly difficult to get anything finished properly within the chaos. Thus it all depends on whether the people around you get that in order for it to be easy, they really have approach it like it is hard. Or slightly restated: it is easy right up until you fall into the trap of thinking it is easy ...

Know your Limits

I’ve taken this week off from my day job to work around the house. It’s a yearly ritual -- my house is over a 100 years old and in constant need of repairs. This time my main focus is the front deck. It’s 16 feet in length and the top is about 8 feet deep. There are four deep steps running the whole length of the deck. It was probably built at least 20 years ago, but I suspect that it might be way older, perhaps as much as fifty.

Four years ago I rebuilt the inside of the deck because it was starting to rot in places. I put back the original covering boards since most of them were in reasonable shape. This year we decided that we wanted to get it painted, but many of the original boards we’re starting to go at their ends, so I figured it was time to recover much of the deck and the stairs with new boards.

Sometimes, when I am working on a project like this, people look at me strangely. I’m a bit geeky looking and their initial reaction is to get concerned when they see me playing with power tools. I guess they expect me to end up injuring myself, but I’ve actually been doing this sort of work since I was young and have plenty of experience with big tools of all shapes and sizes. So far I haven’t managed to nail my feet to floor. I guess people think that software guys are supposed to stick to virtual bits. But for me “building is building” it is only the medium that has changed; the work remains the same.

Of course natural materials like wood take a bit of getting used to. Wood warps, bends, and comes in weird sizes (a 2x4 is actually 1.5x3.5). Sometimes it does what you want, other times it takes a bit of ingenuity to get it to fit properly. In that way, it resembles software when you are building on top of quirky libraries. Sort of standardish, but not enough to make the work easy and plenty of unexpected surprises.

So far my project has gone well. As usual I made up a plan for what I was going to do and ordered a huge amount of wood to get shipped to the house. Once I opened up the stairs I found a bunch of problems. The bottom stair for instance, essentially had its cross-beams completely rotted out. A number of other steps needed new, but rather awkwardly sized little support beams.

Pretty typical for this type of project. An initial plan is great, but you have to keep re-adjusting it as you find more issues along the way. The issues are never quite predictable and you learn to not get easily frustrated. Sometimes it’s smoother then you expect, but rarely.

The stairs were a fairly simple job, but because of a few technical setbacks -- my power cord died, although at first I though it was the mitre saw -- it took longer than expected. When I finally got to do the main part of the deck -- it’s essentially a separate unit from the stairs -- I found that my earlier reconstruction efforts hadn’t worked as well as I had hoped.

The main deck was sagging horribly in the middle, which turned out to be caused by my fixing the cross-beams that had rotted. I cut off the bad parts, then extended the remaining pieces. Those extensions weren’t the same height as the original cross-beams (which are 8 inches) and I foolishly didn’t brace them on anything. With time they started to shift downwards.

Not a huge problem, but it meant taking them all off again (there are twelve), then cutting some new pieces to prop them up underneath, and then finally putting each one back in place. It took around three hours, but at least this time it was braced properly.

During my enthusiasm to fix these initial deficiencies, I knew that the new height would no longer be sagging, and that it was now somewhat higher than the original height. Way nicer and a lot more solid, but it left me with a new problem. The very last board on the top of the deck, which is also the first board as one steps up from the stairs, sits on a pair of 6x2s. The inside 6x2 helps to hold up the cross-beams, while the outside one is mostly decorative (and had to be replaced due to carpenter ants, another minor setback). With the cross-beams now raised and fixed, these two boards were sitting nearly an inch below the main deck height. A gap large enough that I couldn’t leave it that way, but awkward enough that I would never be able to find a pre-cut board that would fill it properly.

I’ve got lots of tools, but nothing that could cut such a small piece for that long distance. I basically needed a 1x4x16. Sixteen feet is a hard length to keep straight with a hand-held saw.

As I was sitting there pondering my problem, my friend Tom walked by. He’s constantly building or fixing things and he’s dropped in a few times to take a look at how the work is going. It’s a hugely valuable contribution, since I’ve found that although I have experience with this type of work, there are plenty of situations were I hit a novel issue that I’ve never encountered before. To be able to stop for a bit, chat about what I’ve done and what I’m about to do really helps in both clarifying the issues and getting someone else’s perspective. And given that Tom has way more experience than I do, he often has really helpful suggestions or points out things that I have missed.

In this instance I was really happy to see Tom. His timing was perfect. He took a quick look at the problem and told me that he could cut the boards that I needed. He disappeared for a bit, and came back with a specialized circular saw. The cutting was a bit tricky because of the length (and because he was holding the boards in the air with only one end touching the ground), but he cut six strips of nearly straight 1x1.5. Layered side-by-side, across the length, they fit in perfectly and my problem was resolved. He popped off for somewhere else -- it was nearly dinner time -- and I continued to finish up the top of the deck, thankful that I had been saved from a rather tricky and no doubt, time consuming problem.

I can relate this back to software development because it is often when we are building complex systems that we hit upon a thorny problem. Over the years, I’ve seen programmer after programmer hide away in their cubicles attempting to solves these issues in isolation. Programming is about solving problems, but there are always times when the problem is beyond any one person’s ability or experience to fix it properly. Sometimes it is best to go out and seek advice, help or at least a sounding board. Just trying to explain a solution to someone else often shows that it is not particularly viable or just too fragile. Sometimes there is an overlooked trivial solution. However, some programmers go out of their way to avoid seeking advice. I think it’s a culture problem with programmers at the deepest levels. Perhaps the fear of sounding stupid because of a lack of solution. Or possibly just a need to horde the more interesting bits. It’s different for different people, but all too frequent.

From the outside however it’s usually appears as a case of someone taking on work that is too big for them to chew. They go beyond their limits and as a result their solution causes serious side-effects. A badly designed or poorly architected piece of software won’t ever fix itself and will always get worse over time. Stopping the problem early saves a huge amount of wasted effort. Admitting that the problem is beyond their limits may be difficult for a lot of programmers, but it is far better than the alternatives. Seeking help is a good thing.

Had I stubbornly refused to get help with my deck project, I might have labored for hours to find a solution. My initial instinct was to cut a huge number of 2x4 shims with my mitre saw. But as Tom pointed out that would have looked incredibly ugly. Although I have the skills do to most of the work necessary, this one sticky problem was just beyond my abilities. I neither had the equipment nor the experience to be able to cut the required piece (at the time I was seriously wishing that I had bought a huge bandsaw for the basement, it would have worked fine but I’m sure that my wife wouldn’t be too impressed with such a large machine sitting around idle most of the time).

For me, bouncing ideas off other people or letting someone else do a piece that they are better at is normal at work. When I am doing complex modeling or schemas for relational databases I usually spend some time discussing the design with Gavin, he’s got a lot more experience in this area then I do. Igor is a wiz at algorithmic stuff and in fiddling with complex technical bits. It’s always worth checking in with him first (he also is great at finding excellent parts for custom PCs; both of my home machines came from his recommendations). I tend to specialize in architectural or product related issues, which seems to be my latest focus (I used to prefer performance optimizations, but I guess I am getting older now...).  I often consult with other programmers both internally and on the web whenever I need deeper advice. After twenty years, while I know lots of stuff, it’s just a drop in a rapidly increasing bucket. Getting things right the first time is far better than just guessing, and you’re just guessing if you don’t already have the knowledge.

A while ago I saw a blog post on how everyone should be doing code reviews. Formalizing that step is a good idea in places where the programmers are not communicating with each other, but certainly at my current work place it’s not necessary right now. As we are often bouncing ideas off each other and we do look at each other code (and point out problems), most of the code written by our rather small group was been reviewed at some point or another. That’s just a nice side-effect of our interactions. That, and we are able to gain indirect experience from each other’s efforts. If that culture doesn’t already exist naturally, then processes like code reviews are a great idea. Programming has gradually become a collaborative occupation where communication plays an important role in achieving success. The projects are too large and there are too many little details floating about for individuals to track properly.  

Building software is always a complex endeavor and the external pressures such as time or management don’t make it any easier. You can learn by experience, but that often means doing the same work a few times before getting it right. That’s not a particularly efficient process. Software is complex enough that there are always a huge amount of things people don’t understand. Knowing where your strengths and experience really lie makes it easier to seek outside advice when the problems get beyond your abilities. Hiding in a cubicle, unwilling to talk to anyone, and struggling alone generally doesn’t turn out well (and is never the best solution).