Complexity and Principles

In the midst of overwhelming complexity, the best option is to pull up slightly.

As we are human, things can get far too messy and complicated, so even if we are keeping up with it, the sheer size of the scope will cause us to be rampantly inconsistent. Those inconsistencies themselves are more extra complexity that is getting added as a byproduct of our not being able to cope with the intrinsic complexity. So it is getting worse.

While the ‘devil’ is in the details, trying to deal with the details at their lowest level is a losing game. So, instead, we want to go a little higher and look for patterns that lay out across similar but different cases.

Instead of getting caught up in the abstract nature of patterns though, we can think of these overarching structural constraints as ‘principles’. Then the higher-level principles should constrain the lower-level details.

Then you might have a great sea of details, but you have a fairly small set of principles above that keep the details in place. Then it becomes easy to fix things. If some specific special case violates the higher principle, you adjust it accordingly. Then to be consistent, you don’t have to remember all of the different cases, just the binding principles. With both, you have a way of moving forward and improving the work, without having to exceed your cognitive limits.

If at some point the principles themselves become too much, then we just do the same thing again and go up another level to meta-principles, to ensure that our principles are aligned and consistent.

Now this is not the same as trying to find top-down abstractions.

This is a bottom-up approach, where you start with the details first. When you see enough of them, you try to align their similarities into principles. The key part though is that you recognize and accept that the different groups of details should share common patterns, and so it is worthwhile looking for those patterns.

But it is worth noting that if the principles are not obvious, then creatively overlying fake patterns on top is not good. If there are no obvious patterns, then there are no principles that bind things together. We don’t want principles for the sake of having principles, that would just be extra complexity, we want them for the sake of simplifying stuff and making it consistent to allow us to reduce complexity. So insanely complicated principles are not actually principles, they are just artificial meta-details, that in themselves should be dropped.

In that way, along with ‘encapsulation’ we can tackle overwhelming complexity without getting completely overwhelmed. You cannot fix complicated things by ignoring their nature with over-simplified approaches, that will usually make it worse. If something is complex, you first have to accept that it is complex. You have to deal with it as a complex entity and work within those boundaries.