The
central problem in software development is managing complexity. This
complexity comes from the sea of details and decisions, large and small,
that need to converge into a stable, working system in order for it to
be successfully used for its intended purpose. That not only includes
the code, but also the analysis, management, documentation, packaging,
distribution, training, support, configuration and operations that
inevitably precede a user being able to utilize some functionality on a
computer.
Unchecked,
rampant complexity growth in any area can swamp a project, making it
impossible to continue substantial progress. About the only way I know
to contain complexity is by breaking it off into manageable chunks and
getting it encapsulated into well-defined sub-problems. This works at a
base level, but new problems emerge. Overlapping sub-problems create
redundancies which add significantly to the complexity. This can be
managed typing and categorizing the sub-problems carefully, so that
overlap is rare or negligible, which gets one past the first hurdle. But
as the number of sub-problems grows, their inherent complexity needs to
checked as well. At this upper level, they just form a slightly higher,
more abstract version of the overall problem, but all of the same
issues apply.
Thus,
at some point in order to manage the complexity another higher layer
becomes necessary and the same issues repeat over again. As the
complexity grows, the number of layers to manage it grows as well. Thus,
in any attempt to manage growing complexity, a steadily increasing
number of layers will need to be created. This is the inevitable
consequence of our existence, we can’t handle unlimited sized problems.
Smart people or well-running teams may have higher limits, but there is
always some limit, and it is always reached earlier than expected.
Layering
occurs in all aspects of software development, but for this post I’d
like to just focus on how it effects the code, data and operations of
software.
The
first and most obvious layer in software is the underlying code itself.
Programmers get presented with a number of domain or technical problems
for which they assemble a large number of instructions for a computer
to follow. It is well understood that failure to organize these
instructions results in ‘spaghetti code’. That is code with a logical
structure so intertwined it resembles the noodles in a plate of
spaghetti. Of course this is bad, because any attempt to alter the code
is hampered by a significant wall of complexity. From this, within the
context of coding, we can draw the conclusion that dis-organization
increases complexity.
As
the code base begins to grow, it becomes necessary to break it off into
more and more sub-problems to manage it. This layer is most often
referred to as system’s architecture. It introduces conceptual groups
like modules, sub-systems, components, etc. Each of these pieces
encapsulates a larger subset of the functionality, and the interactions
between them gets formalized. Although I’ve never seen it mentioned,
dis-organization at this level can produce a spaghetti architecture, one
that impedes both changes and overall stability. Experience coding in
the base layer does not easily translate into understanding the
organizational issues at this layer. They appear similar, but they are
fundamentally different problems.
Once
a system has grown out of its base components, it is often desirable to
widen its functionality to a larger group of developers. This creates
essentially a platform layer where the core abilities are exposed, and
controlled, allowing for the existing work to be leveraged for more
divergent purposes. Once again, failure to organize this layer results
in more pasta, and of course understanding the dynamics of the necessary
organization is not directly related to the layers below.
Continuing
upwards we get to the product or system layer. This layer generally
introduces a new set of problems because it is driven more often by the
desires or needs of non-technical people. Organization at this level
comes more as ‘paths’ that are followed serially or in parallel, but
they still need the same generalized principles of organization.
Experience at this level generally involves far more people and
negotiating skills, than technical ones. A product manager for a
commercial system for instance is balancing the technical issues against
the direction and current of the market place.
The
final layer that I’m going to discuss is the one that is completely
ignored by most technical people. For users to access the system, it has
to be running somewhere. In an effort to contain the complexity of
design and work most systems form vertical silos. That is they handle a
very specific problem and don't easily inter-operate with the other
systems being run in their environment. There are some interconnections,
but standards are weak because they interfere with individual goals or
profit margins. As such, most modern large organizations have a large
number of these silos. Like every other layer, redundancies cause
significant problems. Thus, at an operational level, all large
organizations need an organizational scheme to coordinate the building
or purchasing of these silos into a manageable affair. This drives
concepts such as the common enterprise software ‘categories’ like CRM
and CM, which implicitly contain the primitive building blocks of the
operational environment. Again, dis-organization at this level results
in an enterprise wide plate of pasta, which diminishes that usefulness
of the tools.
No
doubt there are many more layers, since they are dynamic and unlimited
in number. For any given layer, as it gets better organized we can say
it is ‘converging’. By this we mean that in some overall way, its base
complexity matches closely to the underlying ‘necessary’ complexity of
the layer. If it’s not managed then it diverges, generally as a result
of disorganization, over-engineering, over-simplification or any other
amplifier that isn’t fundamentally inherent to solving the problem.
It
should be noted that there is a subjective (human-based) component to
minimal complexity. That is, for reasons based on the way individuals
think, the lines between over, under and minimal complexity tend to move
around. One person’s perfect trade-offs can appear overly complex to
another person. Thus rather than a fixed point, a layer will generally
converge within a range, particularly if there are many people involved.
Thus
in software development, in order to manage complexity, we want to see
the overall system converge on a reasonable number of layers, while each
layer is converging on a reasonable system of organization. Development
is an ongoing progress, and thus so is converging. We simply need to
track all of these ‘paths towards organization’ and continually insure
that they are headed for the correct goal. At the bottom layer that is
accomplished by programmers pushing themselves for elegance, while at
the upper layers is it often architects or managers trying to place some
organizational constraints on the chaos below. Every layer needs to get
close in order to win.
