Uncertaincy

While each line of code is pretty simple, and a few of them are not much worse, assembling big collections of code that behaves as expected is crazy difficult.

I used to think it was just scale that was the problem. Remembering a list of five things isn’t too bad; remembering a list of fifty things is beyond most people’s ability.

That can be solved with organization. Write them down, make it easily accessible. But even when the work is organized, it is still courting disaster.

I realized that any means of organizing stuff falls apart with growth. You neatly organize fifty things; it doesn’t work anymore long before you get up to five hundred. Constantly reorganizing stuff is always necessary, but people try to avoid it.

Still, even in the best huge codebase that’s been meticulously cleaned, it is still difficult. I think now that all of these smaller problems are tied together by one larger one: uncertainty. No matter how experienced you are as a programmer, lots of aspects of the work are uncertain.

The foundations and environment are too complex to fully understand. The technologies all have issues. They are wrapped up in histories and complex dynamics. The domain is messy, and understanding is constrained by the slow reveal. Each little step only shows the next step down.

Together, the norm is that you don’t know way more than you actually know. It’s worse when you are a junior, but it never, ever goes away. The explosive growth of the software industry guarantees it.

The difference between success and failure often seems to be how the developers deal with uncertainty. Do they accept it, or just pretend like it doesn’t exist?

For me, I know that I don’t know. But there are a bunch of things that I currently don’t know, but I really, really need to know them. These are the focal points. I can not complete and release the code until I know these things. Every other problem is immaterial. Resolve the key uncertainties and the rest will fall into place.

These uncertainties are often the properties implied by the way the solution is specified.

A trivial example is any sort of feature that lets a user save something and get it back later.

That obviously implies persistence, since there is no way to guarantee that an instance of the program, a thread, or a session or whatever is guaranteed to be live throughout that arbitrary time period. You have to take the data ‘now’, then return it ‘later’.

But it can get worse. If the data spans different resources during persistence, it requires transaction integrity. If it is dependent on external data, it involves synchronization. If it's a lot of data, it eats through storage and requires optimizations. To implement one simple feature may require resolving dozens of uncertainties or worse.

So it comes down to knowing that there are a massive number of things you don’t know or don’t really understand, then deciding which of those you must understand in order to get out the next release. Once you fill in those details, the others either reveal themselves or end up as some risk that you’ve decided is acceptable.

We’re forced to gamble sometimes; the alternative is a potentially bottomless pit of personalities and politics. Some details you can’t pull from the environment; some decisions get hobbled by personalities. For these uncertainties, you randomly pick an answer, shove it out into the world to see what happens, then pay attention to the feedback in order to really resolve it. That is the shortest path.

Dealing with the tornado of uncertainties whirling around you is what makes software development difficult. If you try to pretend that you are not caught up in this, the denial will only make it worse. If you accept that, unfortunately, this is the way it is right now, then at least you can adopt strategies that are more likely to produce better outcomes. You might not know, but that is fine; you still have to get the work done as well as possible.