Two Passes

I have been reminded quite recently of a very common software operations problem.

You have to do a big upgrade to an existing system.

For this, you should have an incredibly thought-out plan, down to the littlest details, and the ability to roll back if there is trouble.

That is, you are 142% certain that it will work, and if it doesn’t, you are 220% certain you can abort the attempt safely.

“We’re only eighty percent sure, and it is too much work to plan it all out.”

The thing you should never, ever do is just go ahead anyway.

First, you’re less than a hundred percent certain, which is bad. Very bad.

But the key problem is that if you actually knew what you were doing, the creation of the plan would be trivial. That you think it would take a long time is the prime indicator that you don’t really understand what you are doing. The two are the same.

If you don’t know, admit it. It is way less painful.

And then do a 2-pass.

Do it once to make sure it works, and if that goes “perfectly,” do it again for real.

Odds are that you’ll screw it up and have to repeat that first pass a bunch of times. But understand that each time you repeat it, you just saved a crazy amount of both time and pain. You just avoided a disaster.

Yes, it is longer. Yes, it is more work. But to avoid it, you have to be able to produce an incredibly accurate plan and its rollback. You didn’t, you can’t, so you’re pretty much in the 100% fail territory right now, unless you get ‘win millions in a lottery’ lucky. If you’ve got that type of horseshoe, there are more effective things for you to do with your time than upgrades.

It’s really just a variation of ‘measure twice, cut once’.

You walk through all of it, in full, as if for real, but then don’t do the very last thing to commit it. Then with that knowledge, you can obviously draft a plan, so now you can move forward.

Note that if the first pass in a 2-pass works, you probably can’t just flip the switch right then and there. Mostly, you have to let other people around you know in advance, and unless you lied to them about what you're doing the first time, that notification gets done only for the second, actual pass. But that is fine. Doing it twice is a necessity, not a waste of time.