Complexity Blowout

Despite any good intentions, sometimes the underlying complexity of any large problem, either domain or technical, drastically exceeds the abilities of the people involved. It’s just too complex for them to deal with.

What happens then, is that everybody retreats into tunnel vision. They all clamp down their own expected context, but in different ways. Mostly the odds of any of these sub-contexts being viable are extremely low. There will usually be some variability that was deliberately ignored, that should not have been ignored. The idea may work in the sub-context but will fail miserably in the real, larger, context.

So, what you see in practice is that everybody starts to argue with each other. That is a primary symptom that may indicate that we’ve hit a complexity blowout.

People get quite angry and frustrated. Everyone can see that the other suggested contexts will not work as expected but they feel that their own context is correct. Obviously, it is unlikely that any of the suggested subcontexts are sufficient, so it is a basically impossible problem at this point.

Really the only solution is to find somebody that can “see it all” and take direction from them. But people always misestimate their own abilities, as well as others, so it would first involve admitting that they have no idea about how to properly solve the issue. That is a hard sell. Then start finding some way of verifying someone else’s understanding. We don’t normally see this type of introspection in software development circles. We’re taught to fake it until we make it.

What happens instead is that one or more bad paths eventually get chosen. And as they fail, the people responsible for the failures leave. Given the severity of this type of problem, this failure state can persist for decades. The various attempts to solve things fail, the arguments continue, and the people fracture. It’s politics as usual.

It’s a rather classic and common version of biting off more than you can chew.

If it has gone into this impossible state and you don’t have full and total authority over it, there is really nothing you can do about it. It is systemic at that point. Even if you think you know how to solve it, you are probably wrong too, so you’d just be another incorrect alternative in the endless debate.

If you did know how to solve it, you would also know enough to suspect that you didn’t know how to solve it. That is, the more confident you are in your solution, the less confident you should be in your solution, which is obviously a paradox.

The best you can do is go around to everyone else, see what they propose, understand their flaws, and see if they are also present in what you are proposing. But a lack of flaws does not imply full understanding or correctness. Just that the now obvious reasons for not proceeding are not there.

So, oddly the best choices are the ones that are crystal clear, but not pushed heavily, in that we accept that overconfidence is often driven by insecurity.

As for actually solving the problem in a reliable way, that usually doesn’t happen. Rather it is just an endless parade of broken solutions and turnover. At best, some of the solutions will kinda work but even then their lifespans will be prematurely cut short.