Layering, Architecture and Convergence

The central problem in software development is managing complexity. This complexity comes from the sea of details and decisions, large and small, that need to converge into a stable, working system in order for it to be successfully used for its intended purpose. That not only includes the code, but also the analysis, management, documentation, packaging, distribution, training, support, configuration and operations that inevitably precede a user being able to utilize some functionality on a computer.

Unchecked, rampant complexity growth in any area can swamp a project, making it impossible to continue substantial progress. About the only way I know to contain complexity is by breaking it off into manageable chunks and getting it encapsulated into well-defined sub-problems. This works at a base level, but new problems emerge. Overlapping sub-problems create redundancies which add significantly to the complexity. This can be managed typing and categorizing the sub-problems carefully, so that overlap is rare or negligible, which gets one past the first hurdle. But as the number of sub-problems grows, their inherent complexity needs to checked as well. At this upper level, they just form a slightly higher, more abstract version of the overall problem, but all of the same issues apply.

Thus, at some point in order to manage the complexity another higher layer becomes necessary and the same issues repeat over again. As the complexity grows, the number of layers to manage it grows as well. Thus, in any attempt to manage growing complexity, a steadily increasing number of layers will need to be created. This is the inevitable consequence of our existence, we can’t handle unlimited sized problems. Smart people or well-running teams may have higher limits, but there is always some limit, and it is always reached earlier than expected.

Layering occurs in all aspects of software development, but for this post I’d like to just focus on how it effects the code, data and operations of software.

The first and most obvious layer in software is the underlying code itself. Programmers get presented with a number of domain or technical problems for which they assemble a large number of instructions for a computer to follow. It is well understood that failure to organize these instructions results in ‘spaghetti code’. That is code with a logical structure so intertwined it resembles the noodles in a plate of spaghetti. Of course this is bad, because any attempt to alter the code is hampered by a significant wall of complexity. From this, within the context of coding, we can draw the conclusion that dis-organization increases complexity.

As the code base begins to grow, it becomes necessary to break it off into more and more sub-problems to manage it. This layer is most often referred to as system’s architecture. It introduces conceptual groups like modules, sub-systems, components, etc. Each of these pieces encapsulates a larger subset of the functionality, and the interactions between them gets formalized. Although I’ve never seen it mentioned, dis-organization at this level can produce a spaghetti architecture, one that impedes both changes and overall stability. Experience coding in the base layer does not easily translate into understanding the organizational issues at this layer. They appear similar, but they are fundamentally different problems.

Once a system has grown out of its base components, it is often desirable to widen its functionality to a larger group of developers. This creates essentially a platform layer where the core abilities are exposed, and controlled, allowing for the existing work to be leveraged for more divergent purposes. Once again, failure to organize this layer results in more pasta, and of course understanding the dynamics of the necessary organization is not directly related to the layers below.

Continuing upwards we get to the product or system layer. This layer generally introduces a new set of problems because it is driven more often by the desires or needs of non-technical people. Organization at this level comes more as ‘paths’ that are followed serially or in parallel, but they still need the same generalized principles of organization. Experience at this level generally involves far more people and negotiating skills, than technical ones. A product manager for a commercial system for instance is balancing the technical issues against the direction and current of the market place.

The final layer that I’m going to discuss is the one that is completely ignored by most technical people. For users to access the system, it has to be running somewhere. In an effort to contain the complexity of design and work most systems form vertical silos. That is they handle a very specific problem and don't easily inter-operate with the other systems being run in their environment. There are some interconnections, but standards are weak because they interfere with individual goals or profit margins. As such, most modern large organizations have a large number of these silos. Like every other layer, redundancies cause significant problems. Thus, at an operational level, all large organizations need an organizational scheme to coordinate the building or purchasing of these silos into a manageable affair. This drives concepts such as the common enterprise software ‘categories’ like CRM and CM, which implicitly contain the primitive building blocks of the operational environment. Again, dis-organization at this level results in an enterprise wide plate of pasta, which diminishes that usefulness of the tools.

No doubt there are many more layers, since they are dynamic and unlimited in number. For any given layer, as it gets better organized we can say it is ‘converging’. By this we mean that in some overall way, its base complexity matches closely to the underlying ‘necessary’ complexity of the layer. If it’s not managed then it diverges, generally as a result of disorganization, over-engineering, over-simplification or any other amplifier that isn’t fundamentally inherent to solving the problem.

It should be noted that there is a subjective (human-based) component to minimal complexity. That is, for reasons based on the way individuals think, the lines between over, under and minimal complexity tend to move around. One person’s perfect trade-offs can appear overly complex to another person. Thus rather than a fixed point, a layer will generally converge within a range, particularly if there are many people involved.

Thus in software development, in order to manage complexity, we want to see the overall system converge on a reasonable number of layers, while each layer is converging on a reasonable system of organization. Development is an ongoing progress, and thus so is converging. We simply need to track all of these ‘paths towards organization’ and continually insure that they are headed for the correct goal. At the bottom layer that is accomplished by programmers pushing themselves for elegance, while at the upper layers is it often architects or managers trying to place some organizational constraints on the chaos below. Every layer needs to get close in order to win.