Navigation

The primary goal of software is to be useful. In order to do that, someone with a problem needs to use it to build up a ‘context’ and then execute some functionality. This context can be seen as all of the inputs necessary for their computation, whether it is persistent data, current configuration or the results of navigating from a broader context down to a specific one.

As an example of that latter input, for a GUI application, a user might start the program and select a number of menu items and buttons that gradually get them closer and closer to being able to run the desired code. They might open a file, move the cursor to a specific line, then enter some text. When satisfied, they will finally save the document, as a file, for long-term persistence on a disk or flash drive somewhere, achieving their goal.

The functionality that they require is really to modify an existing persistent file of a given format. Everything else they have to do, starting the app, picking file, changing it, etc. is just building up a context of inputs so that the save happens in the way they expect it to. Consequently, if the editor dies before the file is saved then they have to go back and navigate through the context again. It’s not over until the final functionality is completed.

For a command line environment it is the same. Most often the user works their way into a directory, might change the file with something like sed or awk redirected into a temp file, and then copies the modified file on top of the old one. Their navigation through the various commands is nearly equivalent to utilizing a series of widgets in a GUI.

In that sense we can frame any user action with a computer as a long series of navigating down to specific contexts to fire off some code. Programmers often focus on what that code is, and how it works, but from a user’s perspective the ease or pain of navigating is what they really see.

When programs are small, the amount of functionality is small enough that it can be pushed up to the top level. It really doesn’t need any serious organization. But as development continues and more and more functionality becomes available, knowing some functionality exists and navigating to it become larger and larger problems. In many active but ancient products, a great deal of the features just aren’t used because they are nearly impossible to find when needed. They exist, but they are buried in unexpected places.

To counter this, the focus should change as the code grows, first as needing new functionality, then it switches over to finding ways to reorganize the interface so that all of that existing functionality is still usable. This of course involves heavy refactoring, which is hugely expensive as the underlying layers of code have become locked into position, so it is rarely done.

It is also extremely difficult, in that the designers need to re-imagine every aspect of the program from the top down, from the perspective of the user. And then they need to overlay an organizational philosophy on top of everything so that it is clearly intuitive to most people on where to navigate to achieve their goals. There isn’t, that I know of, some magical means of applying ‘normalization’ such that it makes this process mechanical, and whatever arrangement they come up with will eventually degrade as the functionality continues to grow. It’s a perpetual problem.

What seems to occur in practice is that the programs just gradually decay until they are essentially stuck. The navigation becomes so bad, and arbitrary, and the work to fix it is so large, that almost no more forward progress is made. Generally this is followed by another program from someone else, with a different organizational perspective getting written and taking away most of the users. Eventually that one stalls too, and the cycle repeats itself. In that way it may seem like we are progressing, but often the case is that the replacements simply ignore old, but required functionality, which when added hastens the inevitable outcome. These days it seems more like we are just going round and round in circles.

It is far easier to ignore the navigation and just focus on grinding out functionality, which is why it is common practice. When software was young, this approach made a lot of sense, in that there was a real need for new features. But most of the industry has matured now, and there is a plethora of functionality available for just about every imaginable computation. It’s no longer an issue of having to create the functionality, rather it has become one of trying to find it. This change does not seem to be reflected in our development cultures. Yet.

Partial Knowledge

We live in an amazing time. Collectively our species knows more about the world around us than any one individual could ever learn in their lifetime. Some of that knowledge is the underlying natural complexity of our physical existence. Some of it is the dynamic complexity of life. On top of this, over a long history, we have built up hugely complex social interactions, that have been evolving to maintain our growing technological innovations. Altogether, the details are staggering, and like swiss cheese, our understanding of it all is filled with countless holes.

Full knowledge is rather obviously impossible for any individual. The best we can do is focus in on a few areas and try to absorb as many details as possible. We all have partial knowledge. To make matters worse, often the details are non-intuitive. An overview is a simplification, so it is easy on the outside to misunderstand, to think that the issues that matter are different from what really matters on the inside. We see this often, with people misdirecting resources because their knowledge is too shallow.

Given our out-of-control complexity, it is not surprising that there are frequent and long-running problems with our societies. Each new generation hyper-focuses on their own partial understanding, pursuing changes that rather than fix the problems, just shifts them elsewhere and slowly amplifies them. This gets continually repeated, as the underlying general knowledge gets watered down. In time most people start knowing less about the specifics because they get swapped with the volume, their understandings become increasingly destructive. Their changes converge on making things worse.

What we need as a species is to be able to merge our partial knowledge. To bring it all back together, generation after generation, so that we are moving forward, rather than just cycling between calamities.

Knowledge was initially trapped in an individual. We eventually learned to verbally communicate it to our peers, then to preserve it for generations in physical form, but other than the gradual simplifications brought on by our repetitive communication we really don’t know how to extract the essence of underlying truth from it. If we could pull out the truth from the things we say, and we could combine it across large numbers of people, we could reassemble it in ways that enlighten our perspective on our world, our lives, our future.

What lies under the heart of all communications is a cloudy mix of aspects of our existence. Never really correct, but often not obviously wrong. As it flows through people over time it maintains that anchor to its origins but reshapes itself from other exposure. It is slowly merging, but what we really want is to be able to quickly identify it, strip it down, collect it and build it back up again. We want to accelerate this, but also to ensure that we improve in quality, rather than ebb and flow. We have some means of moving forward, like the scientific method and the ideas underpinning engineering, but daily life is a constant reminder that these approaches fail often. They are not perfected and they really come to rely on individuals for success. That makes them subject to being overwhelmed by the growing tsunami of misinformation.

What we need are tools to help the individuals work together, to help them merge their partial knowledge and to allow the merging to reliably converge on a quality of understanding that allows us to fix our past mistakes without just shifting them elsewhere. We have such a tool: computers. They can amass large amounts of information and they are patient enough to sift through it, using the underlying structural information to measure fitness. We can build tools to hold massive contexts and to gradually merge these into larger and larger ones. We can build tools to help leverage these huge structures to correct and fix grand problems.

We do some of this now, but we do it on an ad hoc basis with very specific modeling. The next generation of tools needs to do it across everything we know, rather than just for tiny sub-disciplines. It’s a model and merge problem, but one that spans the perspectives and understandings of billions of people, not just a small group of researchers.

If we can’t utilize our partial knowledge to improve things then we know that complexity will continue to grow, now at an increasing rate, up to some threshold where it in itself will become the most significant problem for our species. That is, technology, being neither good nor bad, is just an amplifier. And at the point that it expands things to be large enough, the core gets so watered down that the entire edifice will stall, or even worse it will implode. It is a recurring pattern in history, it also seems to be a physical limitation of our universe, and it is this path that we are currently traveling on.

We see that we can not go on like this forever, but our reliance on partial knowledge also means that without better tools we will. That no single individual can lead us away from our fate, that now not even a small number of people will suffice anymore. We need to harness a gigantic understanding in order to correct our path and we haven’t even begun to accept this as ‘the problem’ let alone started working on it yet. In a sense, technology got us into this mess; we have to admit that and redirect our resources to craft technologies that will really get us out, not just pile on more dangerous bandages or keep shifting the problems around in circles. This isn’t a practical problem that will get accidentally solved by a couple of people in a garage, it has grown too large for that now. If we don’t choose to accept, understand and work towards a solution the universe will eventually find one for us...

Backwards

It is easy to get lost when tackling a difficult problem. It is easy to go off on a dead end, wasting a lot of valuable time. When we are presented with a lot of choices, it is very difficult to pick the right one.

For whatever reason, most people are far more comfortable with approaching problems ‘forwards’ from the top down. However, we could see the path to a solution as winding through a rather dense tree of decisions and effort. At every opportunity there are plenty of choices, options. If you make the right choice, you move to down to the next level of the problem. If you pause or head down an irrelevant path, well…

The most optimal way I’ve found is to visualize everything backwards. That is, instead of starting at the beginning, you go straight to the end. If you understand what parts of a solution ‘must’ exist, then you can work backwards to bring them together. Along the way you can collapse any special cases to simplify them. As you keep going backwards from the end, you are naturally pruning off all of those dangerous decisions, all of that useless work. You are avoiding massive amounts of wasted time.

When the outcome is predetermined, the path between there and here is straightforward. If you can walk just that path, and only that path, then you can create a solution in the fastest time possible. All of the work to be accomplished has a very specific, and targeted use, none of it is wasted.

Software gets a little more difficult because it isn’t just one version, it grows over a long period of time often changing directions a bit. As you build a system, there are usually multiple destinations at once and with usage new ones become apparent. Still, if you go backwards from each, and you understand the overall territory you are moving through, then you easily become aware of multiple overlapping paths. You can take advantage of these to further optimize not just the next solution, but a whole series of them. The better you become at identifying viable opportunities to leverage existing work, the more ambitious you can be to tackle bigger and more interesting problems. Time is always the one resource that a Software Developer has far too little of, so saving huge chunks of it is way better than trading off little chunks now for massive problems in the future. The direction of a big project is never arbitrary, and if it seems like it is, then it is only because people haven’t spend enough time to really think it through. If you don’t get that time, because you are wasting it on unnecessary decisions and dead ends, then you’re squandering your most precious resource.

Most development projects are not nearly as difficult as they seem. Most of the endless series of decisions we make are not really necessary. Much of the work done has no real practical value. We get into time crunches most often because we are not spending our time doing valuable things. While often it's hard to see the right path going forward, it is much easier to see it if you are backing up from the solution. All you need to do is flip your perspective and the rest comes naturally.