Architecture

I was reading an online discussion about architecture. It’s always been a rather odd topic in that it has actually been extremely well codified, at least a few times, but that knowledge always seems to get lost. So more often, people just define it as anything they want it to be, as long as it benefits their personal agenda.

Pretty much all that is wrong, but rather than point to a bunch of better, but old references, I’ll continue a long and painful tradition of throwing around my own definition.

A software system has an architecture if and only if it is organized. That is, if it is just a mess of stuff thrown together, then it has no architecture. It may have had an architecture in its early years, but if subsequent work ignored that, then it is just a pile of stuff now.

The process of establishing an architecture is part structural and part political. That is, for any non-trivial system, it is both so slow and so expensive to build that the act of arranging for the significant costs of the work to be covered is pure politics. All sorts of people have their fingers in this pie, so the technologies and stacks that will be picked as the foundations are just artifacts of the act of raising money. It’s true for startups, but it is also true for any large organization.

Each foundational dependency has an obvious set of strengths and weaknesses. You could rationally choose between them based on those properties to arrive at the best fitting pieces for any given solution. That almost never happens, politics drives people to choose irrationally, often based on prejudice, personality or limited experiences. So, generally the technologies and tech stacks are picked way in advance, long before the real work begins. Often they are poorly matched to the requirements. A lot of work goes into sloppy patches to cover over these weaknesses.

Once you get past the effects of politics, the rest is structural. There are two primary areas.

The first is decomposing related pieces together. That is, all of the code related to a behaviour of the system like reporting should be placed together. But it’s not actually that simple, in that there are always at least two dominant dimensions at play. The problem domain, too often called the business, imposes vertical constraints on the system, while the technical domain, as a foundational layer, imposes horizontal constraints. These are at odds with each other, although in modern times it has become more popular to just pay lip service to any large scale technical issues and blindly follow the business.

Still, you can organize the codebase, even if it is huge, into layers of nicely fitting boxes that clearly delineate the subcomponents at both a high and medium level. The strength of doing this early is that if the pieces are all independent from each other, the work can effectively be scaled (parallized) and the overall quality will be far better.

You can go further though. Instead of breaking down the mechanics by clumping together similar bits of code, you can organize it by fundamental composite data types. When this is correctly applied, it tends to mitigate the vertical and horizontal mismatches, but also simplifies both the code and any visualizations of it. You do see this happen in practice, but because it is a very advanced technique, it is more often applied to high quality low-level commercial products, than to the massive amounts of domain specific application code that is far more frequently written these days.

The second primary area is performance related. All systems have to fit into a larger environment. They share a lot of data, they are constantly getting data feeds and sending out data to other systems. At this level, there are standardized enterprise ways of interconnecting the different systems.

The core problem is usually frequency. One system may be able to generate or collect data far faster than another can import it, so there are common solutions like queues that get placed in between to prevent synchronization problems.

As there are literally 50 years of different data formats, communications tools and protocols and a nearly unlimited number of different right and wrong ways to model the data itself, so a great deal of time is spent on these data transfers. In some cases there are well established communications patterns like having a common data bus for the whole organization, but oddly, impatience more often means that these are ignored and each and every feed is badly home rolled. Often even if the internals of the system is tightly organized, its imports and exports are not, usually because the different teams responsible don’t like to agree on any standardized ways of moving data around. They think it takes too long to implement (while ignoring the fact that they are just reinventing the same wheels, over and over again).

There are a few other key architectural issues, like centralization, avoiding massive duplication, stale data and choosing between interface types like cli, native, web, mobile, etc. All of these have been heavily explored and resolved in the past, but again much of that knowledge has been forgotten too.

That’s pretty much architecture in a nutshell.

At least across parts of a large company, we could really be a lot more organized and save ourselves huge amounts of redundant work, but strangely we are unable to do that. It’s possible that the initial political component ends up overshadowing the technical issues, so that the initial loss of time spent there forces all of the following work to get horribly rushed, and then done badly. Not sure, but over the decades we’ve learned more than enough to be able to avoid this fate, yet it still seems to befall most large internal projects.

Our attempts at, instead, just trying to be fully reactive and then letting the work magically evolve on its own tend to court pure disorganization as the “architecture”, so they are usually far, far worse.

Either way we keep building things badly, then realizing they are bad, then restarting from scratch again. By now there are crappy big systems out there that are in their 7th or 8th generation. Maybe we should just do a better job of getting things organized first before we rush off to repeat the same mistakes again. Not making a mess on purpose is really the aim of architecture. You can tell how good people are at it, by the results that they leave behind.