Yak Shaving

The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.

We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.

Yak shaving is actually a symptom of disorganization and/or ordering problems.

If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.

So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.

Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.

First, get your workspace in order. Do you have all the tools you need? Are they configured properly?

Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.

For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.

When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.

The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.

But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.

It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.

Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.

There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.

In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.

For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.

The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.

Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.

It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.

Setbacks

In the mid-eighties, I was very excited about the potential of computers. Little personal computers were disrupting those massive, expensive beasts lurking in basements. It seemed like computers could integrate into our daily lives, change the world, and make it better.

Computer Science and Informatics were having a heyday; innovation was overflowing. There seemed to be monthly progress.

But over the decades, that progress was plagued by some pretty bad setbacks.

The first one I really saw was the rise of Microsoft.

They had been the David that took down the IBM Goliath, but the moment that they got traction, they got nasty. Over and over again we’d watch as some small group of people would labour to prove some new innovative technology was viable. But just before they could stabilize, Microsoft would swoop in with an army of kids and eat their business.

I was horrified, often quite angry. Instead of software progressing, any new buds were quickly snuffed out by products of far lesser quality, but way more senseless features. Microsoft then applied the rest of its growing monopoly to crush these small competitors.

Just before this era, the heart of innovation had shifted from academics into a small group of energetic people trying to get established. They had good intentions. Mostly these were wiped out. Innovation quickly died as a result; it was too risky, and after all that pain you’d just lose anyway.

That only changed when the next setback took hold.

The Internet had been around for a while, but the Web on top exploded into a huge bubble.

On the plus side, it brought most consumers to software, but it also embraced chaos. If Microsoft killed innovation, the Web killed quality. Whacking out crude software at high speeds was favoured. The ideas shifted from helping people to just exploiting them. The product was no longer software, but a cut of the money, control over people and surveillance. Software was created as fast and crude.

In a quick tumble, we switched from empathy to pure snake oil. If you could trick a million or more people into your broken product, you were famous.

Almost as a counter to the corruption of the Web, mobile apps nearly reversed that trend. For a while, but then everyone else took a page from the Microsoft warbook and quickly bought or crushed anything even mildly innovative or decent quality.

As the mobile dust settled, another setback had been quietly brewing since the dotcom bomb took hold.

Forget quality and innovation, Agile preached mindless coding at high speed.

If you just grind out enough code this week for whatever reactive drama was dominating, the belief was that your chaos would somehow converge into something useful. That lack of direction and foresight was misguided. You can’t randomly build an apartment building; it would obviously collapse, so why would people think that might work for any large technology structure?

Instead of learning from the mistakes of the past, the software industry doubled down on its madness.

Innovative sites vanished, replaced by grungy interfaces that only captured sellable data. People seem to just adjust to a steady flow of disruptive bugs. Using computers got insanely complicated, buttressed by a flood of questionable how-to pages.

Instead of software becoming more intuitive, people just followed opaque, incoherent recipes as if they were magic spells. This produced a mountain of incomprehensible code piled high on mountains of incomprehensible code. It kinda worked, but only if you stayed very close to the magic recipes. Deviate even a little bit at your own risk. Underlying knowledge of how things worked got decimated.

For a while there, it seemed like we’d finally get enough breathing room to be able to fix the industry’s problems. Maybe this time we’d be able to figure out how to build stuff that isn’t crap, but then LLMs appeared, and it started all over again.

Now we live in an era where we are totally dependent on the messes we’ve created but no longer understand. No longer just a house of cards, we suffer from skyscrapers of cards.

And a new generation of people thinks, again, that doubling down on our lack of understanding and handling that over to random traversals of illegally compiled huge lumps of questionable data will fix our problems. But it’s really just another repeat of the same mistakes.

We keep getting distracted from trying to actually figure out how to engineer massive systems that are both reliable and trustworthy. So long as we keep avoiding that, we’ll keep tumbling over the cliffs.

Flow Design

One of the pitfalls in software design is for people to only concentrate on the narrow case where everything works as desired, but ignore all of the other cases where it has gone partially or totally wrong.

Data flow analysis can help avoid this.

The idea is that given some complex scenario for software mechanics, we can fully iterate all of the issues in advance, so we know what we need to solve.

To do this, we can examine the flow of the data.

In a simple example, the data is something a user knows. They enter that into a screen. On completion, the data is sent to the back end, which then persists it.

So the flow is:

User -> screen -> request -> backend -> persistence -> response -> user

Two key problems:

• Data is invalid
• One of the two backend resources is down

If the data is invalid, it is a computational problem. But it shows up in three places:

• screen
• request
• persistence

We’d like all three to be identical. If they do differ, then the messages need to be specific about at which level.

We’d like to give the user a chance to correct the problem without having to start all over again. They really appreciate not having their time wasted.

There are at least three computers involved in this flow. If the first one is down, the flow is blocked, so we don’t need to worry about it.

If either of the other two is down, most of the time we want to wait for a little bit, just in case they become available again, so we sleep for a bit and retry.

If it's a long disruption at either computer, we want to let the user know. Since they may communicate it to support personnel, we want to be clear about which of the two resources is down. So there are two similar error messages for the user, but distinguishable for diagnostic purposes.

If the attempt to persist is successful, we also want to let the user know. But the backend is an intermediary, so it could fail just after the persistence worked and accidentally hide the success message. It was persisted, but the user gets informed about a failure instead. We need to avoid that somehow.

So, there are six arrows, four of which are problems that should be solved in the software.

One way of getting around this is to collapse the arrows as much as possible. Distributed programming is fraught with complex error handling and transaction integrity problems. Avoiding it is the best way of solving it.

Keep in mind that this is a tiny, common, simple example. But it does include all of the spinning pieces and issues. If you code all of the issues to work correctly, then you can extend that approach to much more complex scenarios. If you get this working, then filling in the missing deterministic computations is easier.

Uncertaincy

While each line of code is pretty simple, and a few of them are not much worse, assembling big collections of code that behaves as expected is crazy difficult.

I used to think it was just scale that was the problem. Remembering a list of five things isn’t too bad; remembering a list of fifty things is beyond most people’s ability.

That can be solved with organization. Write them down, make it easily accessible. But even when the work is organized, it is still courting disaster.

I realized that any means of organizing stuff falls apart with growth. You neatly organize fifty things; it doesn’t work anymore long before you get up to five hundred. Constantly reorganizing stuff is always necessary, but people try to avoid it.

Still, even in the best huge codebase that’s been meticulously cleaned, it is still difficult. I think now that all of these smaller problems are tied together by one larger one: uncertainty. No matter how experienced you are as a programmer, lots of aspects of the work are uncertain.

The foundations and environment are too complex to fully understand. The technologies all have issues. They are wrapped up in histories and complex dynamics. The domain is messy, and understanding is constrained by the slow reveal. Each little step only shows the next step down.

Together, the norm is that you don’t know way more than you actually know. It’s worse when you are a junior, but it never, ever goes away. The explosive growth of the software industry guarantees it.

The difference between success and failure often seems to be how the developers deal with uncertainty. Do they accept it, or just pretend like it doesn’t exist?

For me, I know that I don’t know. But there are a bunch of things that I currently don’t know, but I really, really need to know them. These are the focal points. I can not complete and release the code until I know these things. Every other problem is immaterial. Resolve the key uncertainties and the rest will fall into place.

These uncertainties are often the properties implied by the way the solution is specified.

A trivial example is any sort of feature that lets a user save something and get it back later.

That obviously implies persistence, since there is no way to guarantee that an instance of the program, a thread, or a session or whatever is guaranteed to be live throughout that arbitrary time period. You have to take the data ‘now’, then return it ‘later’.

But it can get worse. If the data spans different resources during persistence, it requires transaction integrity. If it is dependent on external data, it involves synchronization. If it's a lot of data, it eats through storage and requires optimizations. To implement one simple feature may require resolving dozens of uncertainties or worse.

So it comes down to knowing that there are a massive number of things you don’t know or don’t really understand, then deciding which of those you must understand in order to get out the next release. Once you fill in those details, the others either reveal themselves or end up as some risk that you’ve decided is acceptable.

We’re forced to gamble sometimes; the alternative is a potentially bottomless pit of personalities and politics. Some details you can’t pull from the environment; some decisions get hobbled by personalities. For these uncertainties, you randomly pick an answer, shove it out into the world to see what happens, then pay attention to the feedback in order to really resolve it. That is the shortest path.

Dealing with the tornado of uncertainties whirling around you is what makes software development difficult. If you try to pretend that you are not caught up in this, the denial will only make it worse. If you accept that, unfortunately, this is the way it is right now, then at least you can adopt strategies that are more likely to produce better outcomes. You might not know, but that is fine; you still have to get the work done as well as possible.

Properties

Code is a set of instructions for a computer to follow.

But code also has additional properties, beyond its runtime behaviour. There are lots of them.

A simple one is readability. Some code is cryptic; it takes a lot of investigation to figure out what it will do. Some code is self-explanatory. It’s written in a clear style so that most programmers can quickly figure out what it will do. It has extra comments added to fill in any missing explanations.

Another one is idempotent. The first time you call that code, something changes. All subsequent calls with the same inputs won't change it again. It’s a useful property to protect against unintentional repeated calls.

Another one is stateless. There are no internal variables that persist between calls. All the outputs are the results of pure computations on the inputs.

Optimization is also a property. Given the same inputs, some variations on the code will produce the expected outputs with less resources. Common resources include memory, CPU, GPU, disk, and network. This code usually exploits deeper dependencies within the data or the environment to get to the results faster.

Secure is a property too. Some code can be triggered by anyone; some code only by a strictly controlled small set of fully identified people.

Generalization is another interesting property. Within the code is considerable flexibility to accept a full range of different inputs and to produce a larger range of outputs. Instead of having dozens or hundreds of versions of nearly identical code, there is one version with a more complex flow that works its way through the different sets of computations.

Its cousin is abstraction. The code isn’t specific in its instructions. It pulls back from the actual logic to solve the problem at a higher conceptual level. Then there is often a way to bind that output back to the specifics, so the results end up the same. Its strength is that it is far less code than just grinding it out by brute force.

Safety is a property too. The author expected problems like bugs or bad data and added additional logic to detect, prevent, or correct the issues. The code has been explicitly constructed to prevent these bad outcomes. This is often called self-defensive as well.

Extensible is another less common one. The code is structured in a way to help ensure that future changes are less cognitively demanding. This often employs organizational techniques like symmetry or table driving. Adding new logic is usually just duplicating a few lines and making fairly safe little changes. Future changes take a few minutes.

While these properties don’t change the outputs of the code when it runs, they do strengthen its usability. They make deploying and using the code far less risky. They head off common problems, and make the development and operations of the code less stressful.

Most of them don’t require a lot of extra work, and all of them save a lot of time later in diagnosing and dealing with the usual problems. The fewer of these properties the code has, the more fragile it is when used. It’s important to add these in where they will enhance the development and usage of the code. Just calculating outputs with overly fragile code is not fully solving the problems.