Ordering

With complex work, the order in which you do it hugely affects the quality, speed, and accuracy.

If there are ten subtasks, you do not really have the option to do them in any order. You need to start with some tasks first, before you do the others. They are dependent.

That’s pretty obvious if you are building a house. You can’t just do the roof first if you don’t have the walls up. And you can’t get the walls up until you have completed the foundations.

The crazy part about software is that although it is exactly like that too, most people incorrectly believe that because it is “digital”, it is somehow different. Somehow it magically escapes any dependencies, so starting in any order is fine. It is not.

As a consequence, you get situations that are equivalent to the stakeholders getting obsessed over the paint colour for a penthouse, when the foundations for the apartment building haven’t even been finalized in design yet, totally out of order.

\While we would not expect non-technical people to understand or be concerned about ordering the work, it is critical for developers to do this correctly. Maybe the second most important thing after organization.

The stakeholders only see the outside of the top; the rest of the box is invisible to them. They always assume it is trivial.

But internally, software is closest to a house of cards. It is far from trivial, where everything rests on what is below it. That orientation sets its dependencies. An interface that calls a backend, for example, is way up there. The persistence that the backend uses is far below. With that in mind, you can assess the “height” of all components.

Some architectures may flip certain parts of the game, but there is still some sort of relative height there. Figure out those heights, and you get a quick rough sketch of the ordering.

The biggest notion is that everything is built on something else. Those times when a few independent components can be built in parallel, in isolation from each other, are rare anomalies. Take advantage when you find them, but don’t expect to find them and definitely don’t assume they exist. Go the other way: assume there are far more dependencies for anything than you are currently aware of, since that is most likely to be true.

Set a height, plan as best as you can, start at the bottom, then work upwards.

When you get it right, development will go smoothly along.

When you get it wrong, there will be lots of frustration and drama. Simple changes will become huge, oversized issues. The schedule will keep getting pushed back. Way too much stress, and the quality will bottom out, causing downstream operational problems which just keep it all boiling.

Not always, but frequently, if there is a lot of drama in a development project, a key ingredient is bad ordering. That is an issue that developers can and should control. It is one of the places where they are directly responsible for the outcomes.

It’s a grey zone, though, in that plenty of management tries to control it too. It is contested ground, mostly because of timelines.

What experience teaches is that the less you say about the tasks and their order, the more control you will have over them, which you need. It’s better to frame any project scheduling around outside events, not internal ones. That’s why along the way I learned to separate ‘features’ from ‘functionality’, where one is a set of things the user needs, while the other is the specific way each one is implemented. That disconnect is not there to over-complicate things, but instead to disconnect the requirements from the work in order to retain control over the ordering of the work.

When and where some non-technical management has stuck their fingers too deeply into the pie, because they don’t care about dependencies, their ordering choices have been disastrous. Erratic and prone to burning crazy amounts of time on wasted efforts that were started far too early. And although they are the problem, they keep complaining that it's developers that are too slow, which is often beyond frustrating.

Ordering matters. It matters a lot. Once you’ve worked on a well-ordered project, it's difficult to deal with a chaotic one again. The shortest path to getting the work done is oddly very strict. A few other paths are similar, but most others are far worse. Stay on a good path; keep people from distracting you.

Yak Shaving

The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.

We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.

Yak shaving is actually a symptom of disorganization and/or ordering problems.

If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.

So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.

Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.

First, get your workspace in order. Do you have all the tools you need? Are they configured properly?

Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.

For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.

When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.

The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.

But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.

It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.

Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.

There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.

In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.

For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.

The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.

Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.

It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.

Setbacks

In the mid-eighties, I was very excited about the potential of computers. Little personal computers were disrupting those massive, expensive beasts lurking in basements. It seemed like computers could integrate into our daily lives, change the world, and make it better.

Computer Science and Informatics were having a heyday; innovation was overflowing. There seemed to be monthly progress.

But over the decades, that progress was plagued by some pretty bad setbacks.

The first one I really saw was the rise of Microsoft.

They had been the David that took down the IBM Goliath, but the moment that they got traction, they got nasty. Over and over again we’d watch as some small group of people would labour to prove some new innovative technology was viable. But just before they could stabilize, Microsoft would swoop in with an army of kids and eat their business.

I was horrified, often quite angry. Instead of software progressing, any new buds were quickly snuffed out by products of far lesser quality, but way more senseless features. Microsoft then applied the rest of its growing monopoly to crush these small competitors.

Just before this era, the heart of innovation had shifted from academics into a small group of energetic people trying to get established. They had good intentions. Mostly these were wiped out. Innovation quickly died as a result; it was too risky, and after all that pain you’d just lose anyway.

That only changed when the next setback took hold.

The Internet had been around for a while, but the Web on top exploded into a huge bubble.

On the plus side, it brought most consumers to software, but it also embraced chaos. If Microsoft killed innovation, the Web killed quality. Whacking out crude software at high speeds was favoured. The ideas shifted from helping people to just exploiting them. The product was no longer software, but a cut of the money, control over people and surveillance. Software was created as fast and crude.

In a quick tumble, we switched from empathy to pure snake oil. If you could trick a million or more people into your broken product, you were famous.

Almost as a counter to the corruption of the Web, mobile apps nearly reversed that trend. For a while, but then everyone else took a page from the Microsoft warbook and quickly bought or crushed anything even mildly innovative or decent quality.

As the mobile dust settled, another setback had been quietly brewing since the dotcom bomb took hold.

Forget quality and innovation, Agile preached mindless coding at high speed.

If you just grind out enough code this week for whatever reactive drama was dominating, the belief was that your chaos would somehow converge into something useful. That lack of direction and foresight was misguided. You can’t randomly build an apartment building; it would obviously collapse, so why would people think that might work for any large technology structure?

Instead of learning from the mistakes of the past, the software industry doubled down on its madness.

Innovative sites vanished, replaced by grungy interfaces that only captured sellable data. People seem to just adjust to a steady flow of disruptive bugs. Using computers got insanely complicated, buttressed by a flood of questionable how-to pages.

Instead of software becoming more intuitive, people just followed opaque, incoherent recipes as if they were magic spells. This produced a mountain of incomprehensible code piled high on mountains of incomprehensible code. It kinda worked, but only if you stayed very close to the magic recipes. Deviate even a little bit at your own risk. Underlying knowledge of how things worked got decimated.

For a while there, it seemed like we’d finally get enough breathing room to be able to fix the industry’s problems. Maybe this time we’d be able to figure out how to build stuff that isn’t crap, but then LLMs appeared, and it started all over again.

Now we live in an era where we are totally dependent on the messes we’ve created but no longer understand. No longer just a house of cards, we suffer from skyscrapers of cards.

And a new generation of people thinks, again, that doubling down on our lack of understanding and handling that over to random traversals of illegally compiled huge lumps of questionable data will fix our problems. But it’s really just another repeat of the same mistakes.

We keep getting distracted from trying to actually figure out how to engineer massive systems that are both reliable and trustworthy. So long as we keep avoiding that, we’ll keep tumbling over the cliffs.