Over Complicated

I’ve seen many, many variations of programmers reacting to what they believe is over-complexity.

A common one is if they are working on a massive system, with a tonne of rules and administration. They feel like a little cog. They can’t do what they want, the way they want to do it. If they went rogue their work would harm the others.

Having a little place in a large project isn’t always fun. So people rail about complexity, but they mean the whole overall complexity of the project, not just specific parts of the code. That is the standards, conventions, and processes are complex. Sometimes they single out little pieces, but usually really it's the whole thing that is bugging them.

The key problem here isn’t complexity. It is that a lot of people working together need serious coordination. If it's a single-person project or even a team of three, then sure the standards can be dynamic. And inconsistencies, while annoying, aren’t often fatal in small codebases. But when it’s hundreds of people who all have to be in sync, that takes effort. Complexity. It’s overhead, but absolutely necessary. Even a small deviation from the right path costs a lot of time and money. Coding for one-person throw-away projects is way different than coding for huge multi-team efforts. It’s a rather wide spectrum.

I’ve also seen programmers upset by layering. When some programmers read code, they really want to see everything all the way down to the lowest level. They find that reading code that has lots of underlying function calls annoys them, I guess because they feel they have to read all of those functions first. The irony is that most code interacts with frameworks or calls lots of libraries, so it is all heavily layered these days one way or the other.

Good layering picks primitives and self-descriptive names so that you don’t have to look underneath. That it is hiding code, i.e. encapsulating complexity, is actually its strength. When you read higher-level code, you can just trust that the functions do what they say they do. If they are used all over the system, then the reuse means they are even more reliable.

But still, you’ll have a pretty nicely layered piece of work and there will always be somebody that complains that it is too complicated. Too many functions; too many layers. They want to mix everything together into a giant, mostly unreadable, mega-function that is optimized for single-stepping with a debugger. Write once, read never. Then they might code super fast but only because they keep writing the same code over and over again. Not really mastery, just speed.

I’ve seen a lot of programmers choke on the enormous complexity of the problem domain itself. I guess they are intimidated enough by learning all of the technical parts, that they really don’t want to understand how the system itself is being used as a solution in the domain. This leads to a noticeable lack of empathy for the users and stuff that is awkward. The features are there, but essentially unusable.

Sometimes they ignore reality and completely drop it out of the underlying data model. Then they throw patches everywhere on top to fake it. Sometimes they ignore the state of the art and craft crude algorithms that don’t work very well. There are lots of variations on this.

The complexity that they are upset about is the problem domain itself. It is what it is, and often for any sort of domain if you look inside of it there are all sorts of crazy historical and counter-intuitive hiccups. It is messy. But it is also reality, and any solution that doesn’t accept that will likely create more problems than it fixes. Overly simple solutions are often worse than no solution.

You sometimes see application programmers reacting to systems programming like this too. They don’t want to refactor their code to put in an appropriate write-through cache for example, instead, they just fill up a local hash table (map, dictionary) with a lot of junk and hope for the best. Coordination, locking, and any sort of synchronization is glossed over as it is just too slow or hard to understand. The very worst case is when their stuff mostly works, except for the occasional Heisenbug that never, ever gets fixed. Integrity isn’t a well-understood concept either. Sometimes the system crashes nicely, but sometimes it gets corrupted. Opps.

Pretty much any time a programmer doesn’t want to investigate or dig deeper, the reason they give is over-complexity. It’s the one-size-fits-all answer for everything, including burnout.

Sometimes over-complexity is real. Horrifically scrambled spaghetti code written by someone who was completely lost, or crazy obfuscated names written by someone who just didn’t care. A scrambled heavy architecture that goes way too far. But sometimes, the problem is that the code is far too simple to solve stuff correctly and it is just spinning off grief all over the place; it needs to get replaced with something that is actually more complicated but that better matches the real complexity of the problems.

You can usually tell the difference. If a programmer says something is over-complicated, but cannot list out any specifics about why, then it is probably a feeling, not an observation. If they understand why it is too complex, then they also understand how to remove that complexity. You would see it tangled there caught between the other necessary stuff. So, they would be able to fix the issue and have a precise sense of the time difference between refactoring and rewriting. If they don’t have that clarity then it is just a feeling that things might be made simpler, which is often incorrect. On the outside everything seems simpler than on the inside. The complexity we have trouble wrangling is always that inside complexity.

Mangled Complexity

There is something hard to do.

Some of the people involved are having trouble wrapping their heads around the problem.

They get some parts of their understanding wrong. In small, subtle ways, but still wrong.

Then they base the solution on their understanding.

Their misunderstanding causes a clump of complexity. It is not accidental, they deliberately choose to solve the problem in a specific way. It is not really artificial, as the solution itself isn’t piling on complexity, instead it comes from a misunderstanding of the problem space, thus in a way the problem itself.

This is mangled complexity. The misunderstanding causes a hiccup, and some of the complexity on top is mangled.

Mangled complexity is extraordinarily hard to get rid of. It is usually tied to a person, their agenda, and the way they are going about performing their role. Often one person gets it wrong, then ropes in a lot of others who share the same mistake, so it starts to become institutionalized. Everybody insists that the mistake is correct, and everybody is incentivized to continue to insist that the mistake is correct.

Sometimes even when you can finally dispel the mistake, people don’t want to fix the issue as they fear it is too much effort. So, it gets locked into the bottom of all sorts of other issues.

We are building a house of cards when we choose to ignore things we find are wrong. A delay caused by unmangling complexity is a massive amount of time saved.

Software Development Decisions

A good decision in a software development project is one that moves you at least one step closer to getting the work completed with the necessary quality.

A bad decision is one where you don’t get a step forward or you trade off a half step forward for one or more steps backward.

Even a fairly small software development project includes millions and millions of decisions. Some decisions are technical, dealing with the creation or running of the work. Some are usability, impacting how the system will solve real problems for real people.

A well running software development project mostly makes good decisions. You would look at the output of the project and have few complaints about their choices.

A poor software development project has long strings of very poor choices, usually compounding into rather substandard output. The code is a mess, the config is fragmented, the interfaces are awkward, the data is broken, etc. It is a whole lot of choices that make you ask ‘Why?’

If you look at the project and cannot tell if the choices were good or bad then you are not qualified to rate the work. If you cannot rate it, you have no idea whether the project is going well or not. If you don't know, then any sort of decision you make about the work and inject into the project is more likely to be harmful than helpful.

Which is to say if you do not immediately know if a decision is right or wrong, then you should push that decision to someone who definitely does know and then live with their choices. They may not be good, depending on the person you choose, but your chances of doing any better are far less.

In a project where nobody knows enough to make good decisions, it is highly unlikely that it will end well. So, at bare minimum, you can't rush the project. People will have to be allowed to make bad decisions, then figure out the consequences of those mistakes and then undo the previous effort and replace it all with a better choice. It will slow down a project by 10x or worse. If you try to compress that, the bad decisions will become frozen into the effort, start to pile up, and then it will take even longer.

That is, if you do not have anybody to make good decisions and you are still in a rush, the circumstances will always get way worse. It’s like trying to run to the store, but you don’t know where the store is, so you keep erratically changing directions, hoping to get lucky. You probably won’t make it to the store and if you do it will certainly have taken way longer than necessary.

If there is a string of poor choices, you have to address why they happened. Insanity is doing the same things over and over again, expecting the results to change. They will not change on their own.

Ratcheting

You know the final version will be very complicated. But you need to get going. It is way too long to lay out a full and complete low or medium level design. You’ll just have to wing it.

The best idea is to rough-in the structure and layers first.

Take the simplest case that is reflective of the others. Not a “trivial” special case, but something fairly common, but not too ugly. Skip the messy bits.

Code the overall structure. End to end, but not fully fleshed out.

Then take something it is not yet doing and fill in more details. Not all of them, just more. If there are little bugs, fix them immediately but do it correctly. If it means refactoring stuff underneath, do it now. Do not cheat the game, as it will hurt later if you do.

Then just keep that up, moving around, making it all a little more detailed, a little more complicated. Keep making sure that what’s there always works really well. Build on that.

Ratchet up step by step. Small focus changes, fix any bugs large or small. Make sure the core is always strong. Sprinkle in more and more complexity.

This is not the fastest way to code. It causes a lot of refactoring. It requires consistency. You need to be diligent and picky. You might cycle dozens of times, depending on the final complexity, but that gives you lots of chances to edit it carefully. The code has to be neat and tidy. This is the opposite of throw away code.

Although it takes longer, I usually find that since the quality is far better, the testing and bugs get hugely reduced, usually saving more time than lost.