Oversimplication

For a long time, the software industry turned against sophisticated power techniques like abstraction.

Partly because getting a little abstract is difficult for some people, but also because it slows down the number of lines they can grind. As an added bonus, it is a bit harder to debug.

People felt that rapidly churning out super simple, near-trivial, brute force code was the best approach.

Strangely, though, a great deal of the actual foundation of software rests on some pretty complex abstractions. Little of what is now possible with computers would be there or work properly if it didn’t. A lot of that code came from an age when people still cared about getting it correct. It’s not ‘legacy’, it’s ‘classic’.

As I watched these oversimplification trends take hold and gain strength, I was getting increasingly worried about the frailty of all of the software that we depend on. Everything is so buggy these days that it’s a wonder that anything works. A few decades back, people would dispute that claim and say it was crazy, but now it’s way worse and so obvious the tables have turned.

There are plenty of known problems out there where any and all attempts to brute-force some crude logic will fail. The scary part is that it may not ‘completely fail’. It may just look like it’s working, work some of the time, but then fail right in the middle of consequential moments.

Software has many essentially ‘physical’ boundaries. You can’t do anything to get past them; they are hard and fast limits. It’s also not particularly malleable, and certainly gets rapidly less malleable as the codebase grows. The crap at the bottom gets frozen there as more stuff gets piled on top. A digital form of gravity, I guess.

Any sort of ‘just jump in and pound it out’ philosophy is a recipe for failure. By the time you’ve clued into your most obvious mistakes, and there will be obvious mistakes, it is too late to go back and fix them. Then building on top of that shaky mess is doomed.

If the problem you are trying to solve is complicated, then any viable real solution to that problem is at least equally complicated. We’ll call this the equality rule.

Techniques like abstraction and generalization are the only means to take that unmanageable complexity and bring it down to a reasonable level where you can implement it correctly. All your other attempts to ‘hack through’ the issues are just a waste of time.

But some people are often so focused on trying to create the simplest solution possible that they end up ignoring the problem they are trying to solve. So, it’s disconnected. The solution is wonderful, but useless.

Instead, it’s way better to go grab a sheet of actual paper and start writing out any proposed solution first. Sketch some mechanics. Think of it as modelling clay, like what they use for car design. Then take that proposed solution and go back to properly explore the problem. If they don’t match up, go back to the design board. A sketch in time saves buckets of wasted coding. It’s orders of magnitude faster than coding up a bad solution, throwing it away, coding up another, throwing that away, ad nauseam.

Grungy coding habits like that are oddly okay for demos and quick prototypes, but it only works because the code is tiny. Scale screws with everything, and any code that is good enough will always grow. Maybe we should call that the destiny rule. If the code kinda works, then it is fated for continued expansion until it doesn’t work anymore.

The oddest part, though, is that the motivation to keep it simple is great. The thing should be as simple as possible; it’s just that any distance way over or way under that bar is exponentially worse. You need to hit the bar, and that bar is positioned by the equality rule.

If there is any sort of method to the madness, it is to start simple. But know immediately that it is too oversimplified. Bad design. Keep going back to the problem to figure out why it won’t work in specific situations. Put those complexities carefully back into the design. Iterate, over and over again.

If you think that might take too long, and you really don’t have decades to catch up with the state of the art, the shortcut is not to ignore the problem, but rather to jump out and do some reading on the state of the art. Leverage what other people have already figured out. Read a bit, learn a lot. Revise what you actually think simple really means. Know that if a kink in the problem space stumbles too close to a boundary, then find the best fit possible within your timeframe to actually address it, or at least carve out some space for it to be corrected later.

What you don’t want to do is hobble something together out of components you barely understand. While that keeps your code simple, what rests below explodes with overcomplexity.

It’s worth noting that the crazy balancing act necessary for modern software design is not something you can learn instantaneously. Because of this, the fastest way to gain knowledge is always mentoring. Get on a good project, work with good teams, and get someone who knows a bit to explain their rationale. Don’t be skeptical, at least not at first. Everyone is wrong, but some people are less wrong, thus closer to the knowledge you are seeking.

While it's tempting to believe that there is always a simple software solution for everything, it’s been proven incorrect so many times that it has become boring and forgotten, again and again. If you want to build software that really solves people’s problems for them, then that software has to really solve people’s problems for them. If their problems have become crazy complicated, then it’s obvious that anything simple will not even come close to fitting properly. I really should call ‘the equality rule’ a law or something...

Ordering

With complex work, the order in which you do it hugely affects the quality, speed, and accuracy.

If there are ten subtasks, you do not really have the option to do them in any order. You need to start with some tasks first, before you do the others. They are dependent.

That’s pretty obvious if you are building a house. You can’t just do the roof first if you don’t have the walls up. And you can’t get the walls up until you have completed the foundations.

The crazy part about software is that although it is exactly like that too, most people incorrectly believe that because it is “digital”, it is somehow different. Somehow it magically escapes any dependencies, so starting in any order is fine. It is not.

As a consequence, you get situations that are equivalent to the stakeholders getting obsessed over the paint colour for a penthouse, when the foundations for the apartment building haven’t even been finalized in design yet, totally out of order.

\While we would not expect non-technical people to understand or be concerned about ordering the work, it is critical for developers to do this correctly. Maybe the second most important thing after organization.

The stakeholders only see the outside of the top; the rest of the box is invisible to them. They always assume it is trivial.

But internally, software is closest to a house of cards. It is far from trivial, where everything rests on what is below it. That orientation sets its dependencies. An interface that calls a backend, for example, is way up there. The persistence that the backend uses is far below. With that in mind, you can assess the “height” of all components.

Some architectures may flip certain parts of the game, but there is still some sort of relative height there. Figure out those heights, and you get a quick rough sketch of the ordering.

The biggest notion is that everything is built on something else. Those times when a few independent components can be built in parallel, in isolation from each other, are rare anomalies. Take advantage when you find them, but don’t expect to find them and definitely don’t assume they exist. Go the other way: assume there are far more dependencies for anything than you are currently aware of, since that is most likely to be true.

Set a height, plan as best as you can, start at the bottom, then work upwards.

When you get it right, development will go smoothly along.

When you get it wrong, there will be lots of frustration and drama. Simple changes will become huge, oversized issues. The schedule will keep getting pushed back. Way too much stress, and the quality will bottom out, causing downstream operational problems which just keep it all boiling.

Not always, but frequently, if there is a lot of drama in a development project, a key ingredient is bad ordering. That is an issue that developers can and should control. It is one of the places where they are directly responsible for the outcomes.

It’s a grey zone, though, in that plenty of management tries to control it too. It is contested ground, mostly because of timelines.

What experience teaches is that the less you say about the tasks and their order, the more control you will have over them, which you need. It’s better to frame any project scheduling around outside events, not internal ones. That’s why along the way I learned to separate ‘features’ from ‘functionality’, where one is a set of things the user needs, while the other is the specific way each one is implemented. That disconnect is not there to over-complicate things, but instead to disconnect the requirements from the work in order to retain control over the ordering of the work.

When and where some non-technical management has stuck their fingers too deeply into the pie, because they don’t care about dependencies, their ordering choices have been disastrous. Erratic and prone to burning crazy amounts of time on wasted efforts that were started far too early. And although they are the problem, they keep complaining that it's developers that are too slow, which is often beyond frustrating.

Ordering matters. It matters a lot. Once you’ve worked on a well-ordered project, it's difficult to deal with a chaotic one again. The shortest path to getting the work done is oddly very strict. A few other paths are similar, but most others are far worse. Stay on a good path; keep people from distracting you.

Yak Shaving

The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.

We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.

Yak shaving is actually a symptom of disorganization and/or ordering problems.

If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.

So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.

Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.

First, get your workspace in order. Do you have all the tools you need? Are they configured properly?

Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.

For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.

When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.

The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.

But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.

It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.

Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.

There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.

In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.

For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.

The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.

Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.

It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.