Setbacks

In the mid-eighties, I was very excited about the potential of computers. Little personal computers were disrupting those massive, expensive beasts lurking in basements. It seemed like computers could integrate into our daily lives, change the world, and make it better.

Computer Science and Informatics were having a heyday; innovation was overflowing. There seemed to be monthly progress.

But over the decades, that progress was plagued by some pretty bad setbacks.

The first one I really saw was the rise of Microsoft.

They had been the David that took down the IBM Goliath, but the moment that they got traction, they got nasty. Over and over again we’d watch as some small group of people would labour to prove some new innovative technology was viable. But just before they could stabilize, Microsoft would swoop in with an army of kids and eat their business.

I was horrified, often quite angry. Instead of software progressing, any new buds were quickly snuffed out by products of far lesser quality, but way more senseless features. Microsoft then applied the rest of its growing monopoly to crush these small competitors.

Just before this era, the heart of innovation had shifted from academics into a small group of energetic people trying to get established. They had good intentions. Mostly these were wiped out. Innovation quickly died as a result; it was too risky, and after all that pain you’d just lose anyway.

That only changed when the next setback took hold.

The Internet had been around for a while, but the Web on top exploded into a huge bubble.

On the plus side, it brought most consumers to software, but it also embraced chaos. If Microsoft killed innovation, the Web killed quality. Whacking out crude software at high speeds was favoured. The ideas shifted from helping people to just exploiting them. The product was no longer software, but a cut of the money, control over people and surveillance. Software was created as fast and crude.

In a quick tumble, we switched from empathy to pure snake oil. If you could trick a million or more people into your broken product, you were famous.

Almost as a counter to the corruption of the Web, mobile apps nearly reversed that trend. For a while, but then everyone else took a page from the Microsoft warbook and quickly bought or crushed anything even mildly innovative or decent quality.

As the mobile dust settled, another setback had been quietly brewing since the dotcom bomb took hold.

Forget quality and innovation, Agile preached mindless coding at high speed.

If you just grind out enough code this week for whatever reactive drama was dominating, the belief was that your chaos would somehow converge into something useful. That lack of direction and foresight was misguided. You can’t randomly build an apartment building; it would obviously collapse, so why would people think that might work for any large technology structure?

Instead of learning from the mistakes of the past, the software industry doubled down on its madness.

Innovative sites vanished, replaced by grungy interfaces that only captured sellable data. People seem to just adjust to a steady flow of disruptive bugs. Using computers got insanely complicated, buttressed by a flood of questionable how-to pages.

Instead of software becoming more intuitive, people just followed opaque, incoherent recipes as if they were magic spells. This produced a mountain of incomprehensible code piled high on mountains of incomprehensible code. It kinda worked, but only if you stayed very close to the magic recipes. Deviate even a little bit at your own risk. Underlying knowledge of how things worked got decimated.

For a while there, it seemed like we’d finally get enough breathing room to be able to fix the industry’s problems. Maybe this time we’d be able to figure out how to build stuff that isn’t crap, but then LLMs appeared, and it started all over again.

Now we live in an era where we are totally dependent on the messes we’ve created but no longer understand. No longer just a house of cards, we suffer from skyscrapers of cards.

And a new generation of people thinks, again, that doubling down on our lack of understanding and handling that over to random traversals of illegally compiled huge lumps of questionable data will fix our problems. But it’s really just another repeat of the same mistakes.

We keep getting distracted from trying to actually figure out how to engineer massive systems that are both reliable and trustworthy. So long as we keep avoiding that, we’ll keep tumbling over the cliffs.

Flow Design

One of the pitfalls in software design is for people to only concentrate on the narrow case where everything works as desired, but ignore all of the other cases where it has gone partially or totally wrong.

Data flow analysis can help avoid this.

The idea is that given some complex scenario for software mechanics, we can fully iterate all of the issues in advance, so we know what we need to solve.

To do this, we can examine the flow of the data.

In a simple example, the data is something a user knows. They enter that into a screen. On completion, the data is sent to the back end, which then persists it.

So the flow is:

User -> screen -> request -> backend -> persistence -> response -> user

Two key problems:

• Data is invalid
• One of the two backend resources is down

If the data is invalid, it is a computational problem. But it shows up in three places:

• screen
• request
• persistence

We’d like all three to be identical. If they do differ, then the messages need to be specific about at which level.

We’d like to give the user a chance to correct the problem without having to start all over again. They really appreciate not having their time wasted.

There are at least three computers involved in this flow. If the first one is down, the flow is blocked, so we don’t need to worry about it.

If either of the other two is down, most of the time we want to wait for a little bit, just in case they become available again, so we sleep for a bit and retry.

If it's a long disruption at either computer, we want to let the user know. Since they may communicate it to support personnel, we want to be clear about which of the two resources is down. So there are two similar error messages for the user, but distinguishable for diagnostic purposes.

If the attempt to persist is successful, we also want to let the user know. But the backend is an intermediary, so it could fail just after the persistence worked and accidentally hide the success message. It was persisted, but the user gets informed about a failure instead. We need to avoid that somehow.

So, there are six arrows, four of which are problems that should be solved in the software.

One way of getting around this is to collapse the arrows as much as possible. Distributed programming is fraught with complex error handling and transaction integrity problems. Avoiding it is the best way of solving it.

Keep in mind that this is a tiny, common, simple example. But it does include all of the spinning pieces and issues. If you code all of the issues to work correctly, then you can extend that approach to much more complex scenarios. If you get this working, then filling in the missing deterministic computations is easier.

Uncertaincy

While each line of code is pretty simple, and a few of them are not much worse, assembling big collections of code that behaves as expected is crazy difficult.

I used to think it was just scale that was the problem. Remembering a list of five things isn’t too bad; remembering a list of fifty things is beyond most people’s ability.

That can be solved with organization. Write them down, make it easily accessible. But even when the work is organized, it is still courting disaster.

I realized that any means of organizing stuff falls apart with growth. You neatly organize fifty things; it doesn’t work anymore long before you get up to five hundred. Constantly reorganizing stuff is always necessary, but people try to avoid it.

Still, even in the best huge codebase that’s been meticulously cleaned, it is still difficult. I think now that all of these smaller problems are tied together by one larger one: uncertainty. No matter how experienced you are as a programmer, lots of aspects of the work are uncertain.

The foundations and environment are too complex to fully understand. The technologies all have issues. They are wrapped up in histories and complex dynamics. The domain is messy, and understanding is constrained by the slow reveal. Each little step only shows the next step down.

Together, the norm is that you don’t know way more than you actually know. It’s worse when you are a junior, but it never, ever goes away. The explosive growth of the software industry guarantees it.

The difference between success and failure often seems to be how the developers deal with uncertainty. Do they accept it, or just pretend like it doesn’t exist?

For me, I know that I don’t know. But there are a bunch of things that I currently don’t know, but I really, really need to know them. These are the focal points. I can not complete and release the code until I know these things. Every other problem is immaterial. Resolve the key uncertainties and the rest will fall into place.

These uncertainties are often the properties implied by the way the solution is specified.

A trivial example is any sort of feature that lets a user save something and get it back later.

That obviously implies persistence, since there is no way to guarantee that an instance of the program, a thread, or a session or whatever is guaranteed to be live throughout that arbitrary time period. You have to take the data ‘now’, then return it ‘later’.

But it can get worse. If the data spans different resources during persistence, it requires transaction integrity. If it is dependent on external data, it involves synchronization. If it's a lot of data, it eats through storage and requires optimizations. To implement one simple feature may require resolving dozens of uncertainties or worse.

So it comes down to knowing that there are a massive number of things you don’t know or don’t really understand, then deciding which of those you must understand in order to get out the next release. Once you fill in those details, the others either reveal themselves or end up as some risk that you’ve decided is acceptable.

We’re forced to gamble sometimes; the alternative is a potentially bottomless pit of personalities and politics. Some details you can’t pull from the environment; some decisions get hobbled by personalities. For these uncertainties, you randomly pick an answer, shove it out into the world to see what happens, then pay attention to the feedback in order to really resolve it. That is the shortest path.

Dealing with the tornado of uncertainties whirling around you is what makes software development difficult. If you try to pretend that you are not caught up in this, the denial will only make it worse. If you accept that, unfortunately, this is the way it is right now, then at least you can adopt strategies that are more likely to produce better outcomes. You might not know, but that is fine; you still have to get the work done as well as possible.

Properties

Code is a set of instructions for a computer to follow.

But code also has additional properties, beyond its runtime behaviour. There are lots of them.

A simple one is readability. Some code is cryptic; it takes a lot of investigation to figure out what it will do. Some code is self-explanatory. It’s written in a clear style so that most programmers can quickly figure out what it will do. It has extra comments added to fill in any missing explanations.

Another one is idempotent. The first time you call that code, something changes. All subsequent calls with the same inputs won't change it again. It’s a useful property to protect against unintentional repeated calls.

Another one is stateless. There are no internal variables that persist between calls. All the outputs are the results of pure computations on the inputs.

Optimization is also a property. Given the same inputs, some variations on the code will produce the expected outputs with less resources. Common resources include memory, CPU, GPU, disk, and network. This code usually exploits deeper dependencies within the data or the environment to get to the results faster.

Secure is a property too. Some code can be triggered by anyone; some code only by a strictly controlled small set of fully identified people.

Generalization is another interesting property. Within the code is considerable flexibility to accept a full range of different inputs and to produce a larger range of outputs. Instead of having dozens or hundreds of versions of nearly identical code, there is one version with a more complex flow that works its way through the different sets of computations.

Its cousin is abstraction. The code isn’t specific in its instructions. It pulls back from the actual logic to solve the problem at a higher conceptual level. Then there is often a way to bind that output back to the specifics, so the results end up the same. Its strength is that it is far less code than just grinding it out by brute force.

Safety is a property too. The author expected problems like bugs or bad data and added additional logic to detect, prevent, or correct the issues. The code has been explicitly constructed to prevent these bad outcomes. This is often called self-defensive as well.

Extensible is another less common one. The code is structured in a way to help ensure that future changes are less cognitively demanding. This often employs organizational techniques like symmetry or table driving. Adding new logic is usually just duplicating a few lines and making fairly safe little changes. Future changes take a few minutes.

While these properties don’t change the outputs of the code when it runs, they do strengthen its usability. They make deploying and using the code far less risky. They head off common problems, and make the development and operations of the code less stressful.

Most of them don’t require a lot of extra work, and all of them save a lot of time later in diagnosing and dealing with the usual problems. The fewer of these properties the code has, the more fragile it is when used. It’s important to add these in where they will enhance the development and usage of the code. Just calculating outputs with overly fragile code is not fully solving the problems.