The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.
We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.
Yak shaving is actually a symptom of disorganization and/or ordering problems.
If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.
So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.
Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.
First, get your workspace in order. Do you have all the tools you need? Are they configured properly?
Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.
For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.
When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.
The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.
But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.
It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.
Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.
There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.
In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.
For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.
The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.
Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.
It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.