Architectural Ramblings

While there is a huge range in opinions, I think most software architects would agree that their position is primarily about defining broad strokes for the development of computer systems. Laying down a master plan, or an overview of some type. The development then happens by one or more teams of programmers.

The fastest way to pound out a big software system would be to lay out the instructions in the largest possible sets. Reusing highly repetitive sections helps in speed, but beyond that, adding structure is actually more work. Coding a big system into thousands of small functions is a significant effort, regardless of what's actually being built.

However, we break up the code into smaller pieces because it makes it easier to extend the system later. Most software development has shifted away from the idea that it is a huge one-shot deal, and accepted the fact that software is continuous iterations for the life-time of the code. In this case, the big problem is not writing the first version, it is all of the work happening over years and years to move that version forward into various different incarnations of itself.


VERTICAL AND HORIZONTAL LINES

There are two big things that effect the development of software: the technology and the problem domain (business logic). Philosophically, the two lay themselves out perpendicular to one another.

Domain logic problems are vertical. The user needs some functionality which cuts through the system, and ultimately results in some changes to some underlying data. It's a thin line from the user to the data, and then back to the user again. The system implements an update feature, for example, or a report or some other specific set of functions constrained by a specific set of data that the user (or system) trigger.

Technological problems are horizontal. The same problems repeat themselves over and over again across the whole system, no matter which functionality is being used. All Web applications, for example have similar problems. All distributed systems need the same type of care and feeding. Relational databases, work in in a similar manner. The problems are all unrelated to the functionality (or at least unattached to it) . For example all systems that use transactions in relational databases have the same basic consistency problems.


CONSISTENCY AND ONIONS

Easily, the greatest problem that most big systems have is consistency. It's not uncommon to see large software products where each sub-section in the system bares little resemblance to the others around it.

Aside from just looking messy, it makes for a poor user experience because it is harder to guess or anticipate how to utilize the system. If everything is different, then you have to learn it all in detail, rather then just being able to grok the basics and navigate around the system easily.

In small systems, if there was only one programmer, often the functionality, interface, data, etc. is consistent as a consequence. It's one of those early skills that good programmers learn. Picking a dozen different ways to do things simply makes it harder to maintain the code, and the user's hate it. A messy system is an unpopular one. A pretty, consistent one, even if it has bugs, is always appreciated.

Ideally, if you were deploying programming resources, the best approach would be to assign individual programmers to each section that needs to be consistent. An easy way to do this is by arranging them in an onion-like structure. A series of containing layers, each one fully enclosing the others.

In this approach you would have a database programmer create one big consistent schema, and all of the database triggers and logic. Another programmer would be responsible for getting that model out of it's persistent state and into a suitable form usable by the application, and perhaps augmenting that with some bigger calculations. The application/interface programmer would then find consistent ways to tie user elements to functionality. In this scenario, the inconsistencies between the styles of the different programmers would mostly go unnoticed by the users.

It's probably because of the consistency that most of the really big systems out there, actually started as smaller projects. Growing something into a bigger size, while mostly maintaining the consistency, is far easier than trying to enforce it initially. Big projects tend to fail.


STRUCTURE AND TEAMS

Deploying big teams for large software projects is a huge problem.

The domain logic runs through the system vertically. If you're building a large system, and you partition the work out to various groups based on which parts of the subsystem they'll develop, it is a vertical partition. Of course, we know that arrangement will generally result in massive inconsistencies along the functionality lines. It practically guarantees inconsistencies.

If you orient the developers horizontally, there are two main problems. Generally technology is piled high for most systems, so it means that each horizontal group is dependent on the one below, and effects the one above. The work cannot start all at the same time, it has to be staggered, or excess useless work gets done. In general, you tend to have teams standing around, waiting for one another, or building the wrong things.

The other big problem is that the developers are experts on, and focus on the technology, so they tend not to see the obvious problems with the domain data. That can lead to significant effort going off into useless directions that have to be scrapped. Developers that don't understand the domain always write the most convenient code from their perspective. That is, unless they really have a great grip on real domain problems -- which makes for some brutally complex coding issues -- they will choose to interpret the problem in the fashion that is the most convenient for making the code simple, not the functionality. This leads to functional, but highly convoluted systems.

The choice for dividing up the teams, along either set of lines, is a choice that heavily effects the architecture of the system. One is tied to the other.


MANAGEMENT, TECHNOLOGY AND BUSINESS

An architect's main job is to lay down a set of broad strokes for a group of developers or an enterprise. Horizontally or vertically, these strokes must match existing technological or domain logic lines. That is, the architects don't really create the lines, they just choose to highlight and follow a specific subset of them. If they don't and choose to make their own lines, it is assured that the cross-over points will contain significant errors and often become sinks for wasted effort.

Architects, then, have to both have strong understanding of the underlying technologies and strong understanding of the user domains. You cannot architect a system for a bank, for example, if you do not have both a strong understanding of finance, and most of the technologies used in the system.

I've certainly seem millions of dollars get flushed away because the architects didn't understand the technology, or because they just didn't get the real underlying problems in the business domain. Guessing in either corner is a serious flaw.

Often, in practice, older architects will have great and vast business knowledge, but are weak on the the more modern technologies. That's not uncommon, and it doesn't have to be fatal. Two architects can combine forces, one vertical, the other horizontal, if they are both wise enough to know not to step on each other's turf. Nether issues really takes precedence, they both are critical.

The biggest thing to understand is that the arrangement, and partition of the work into various teams, is fundamentally an architectural issue. Management of the teams, and how they interact, which is reflected in the code, is also an architectural issue. So, an architect inheriting four or five cranky, unrelated development teams, needs to be very wary of how they are deployed.


APPROACHING CONSISTENCY

The simplest way to build a big system is to start small and grow it. Careful extensions, backed by lots of refactoring and cleanup, can keep the code neat along both the horizontal and vertical lines, but it is a lot of work to keep up with it.

In time, when it needs more effort, or if the original project is time-dependent, organizing big development teams will directly effect the code-base. In a very large system, it is just not possible or practical to have one and only one programmer spanning all of the things that are needed for consistency. But consistency is still critical to success.

Like water or electricity, people always flow towards the easiest path. It is just that different people have widely different ideas about what easy really means.

In general, for most of humanity, it is easier to be inconsistent, than not. That's why, for example, building construction has a large number of different standards that have to be followed. But just standardizing something itself is not enough.

Not reading or following a standard is far easier than doing it right. So, there needs to be some extra level of incentive given to push people into abiding by the rules. The only way that really works is by separating the concerns, and giving space for someone to be an expert in the consistency of a very small domain. I.e. if you want people to follow some consistent subset of rules, you need to make an inspector (with the power of enforcement) to look after and enforce that consistency.

This, then, brings the issue back into the domain of a single person, although their actual depth in the work is very light.

So, what I am getting at, is that in a very large project, one with a couple of architects, and several teams of developers: to enforce consistency you also need teams of inspectors. Each individual is responsible for a small subset of rules, but one in which they can apply their efforts consistently. Where the architects lay down the broad strokes, the inspectors examine various pieces at appropriate times in the development, and point out infringements. The software is ready when it has reached a suitably low infringement state.

If you just have an architect laying out a grand scheme, it's unlikely to get followed closely enough to be able to justify the work. Normally, no matter how it starts, over time the meta-team structure disintegrates, and the various groups shift their focus away from commonality and towards just doing what needs to be done. From each of their individual perspectives, it seems like a reasonable approach, but for the project as a whole, it is a fatal step towards a death march. The breakdown in development structure, mirrored by a similar one in the architecture, opens up gaping holes that become endless time sinks.

Besides consistency, the inspectors also become the force that tries to stop each team from solving their own little time sinks by throwing them at the other teams. If the lines are clear, then there shouldn't be any political issues about where the problems lay. If the lines are blurry, as things get worse, everything becomes political.


AND FINALLY

There are a lot of software architects out there that grew up in the ranks of programmers. From this view, they want architecture to be primarily focus on either technology or domain issues, depending on their own personal background. The big problem with that desire is that as just a meta-programmer (a coder at a higher level), without taking into account the environment, a huge chunk of the possibility of success or failure rests on someone else's shoulders. If you leave all of the personal arrangement, politics and other development dynamics to an unrelated set of managers, you are unable to control the huge effect on the overall work.

The environment will build things in, effects how we build them. It can also be the biggest factor between success and failure. If these things represent big issues that can sway the results of an architecture, then the architect needs to be in control of them. If you can't break up and restructure the development teams as needed, then you're not really in control of a major aspect of the work are you?

Although the architects level is higher and more generalized, they do need the possibility of exerting total control over any and all aspects that could derail their efforts. That includes management and politics.