Partial Knowledge

We live in an amazing time. Collectively our species knows more about the world around us than any one individual could ever learn in their lifetime. Some of that knowledge is the underlying natural complexity of our physical existence. Some of it is the dynamic complexity of life. On top of this, over a long history, we have built up hugely complex social interactions, that have been evolving to maintain our growing technological innovations. Altogether, the details are staggering, and like swiss cheese, our understanding of it all is filled with countless holes.

Full knowledge is rather obviously impossible for any individual. The best we can do is focus in on a few areas and try to absorb as many details as possible. We all have partial knowledge. To make matters worse, often the details are non-intuitive. An overview is a simplification, so it is easy on the outside to misunderstand, to think that the issues that matter are different from what really matters on the inside. We see this often, with people misdirecting resources because their knowledge is too shallow.

Given our out-of-control complexity, it is not surprising that there are frequent and long-running problems with our societies. Each new generation hyper-focuses on their own partial understanding, pursuing changes that rather than fix the problems, just shifts them elsewhere and slowly amplifies them. This gets continually repeated, as the underlying general knowledge gets watered down. In time most people start knowing less about the specifics because they get swapped with the volume, their understandings become increasingly destructive. Their changes converge on making things worse.

What we need as a species is to be able to merge our partial knowledge. To bring it all back together, generation after generation, so that we are moving forward, rather than just cycling between calamities.

Knowledge was initially trapped in an individual. We eventually learned to verbally communicate it to our peers, then to preserve it for generations in physical form, but other than the gradual simplifications brought on by our repetitive communication we really don’t know how to extract the essence of underlying truth from it. If we could pull out the truth from the things we say, and we could combine it across large numbers of people, we could reassemble it in ways that enlighten our perspective on our world, our lives, our future.

What lies under the heart of all communications is a cloudy mix of aspects of our existence. Never really correct, but often not obviously wrong. As it flows through people over time it maintains that anchor to its origins but reshapes itself from other exposure. It is slowly merging, but what we really want is to be able to quickly identify it, strip it down, collect it and build it back up again. We want to accelerate this, but also to ensure that we improve in quality, rather than ebb and flow. We have some means of moving forward, like the scientific method and the ideas underpinning engineering, but daily life is a constant reminder that these approaches fail often. They are not perfected and they really come to rely on individuals for success. That makes them subject to being overwhelmed by the growing tsunami of misinformation.

What we need are tools to help the individuals work together, to help them merge their partial knowledge and to allow the merging to reliably converge on a quality of understanding that allows us to fix our past mistakes without just shifting them elsewhere. We have such a tool: computers. They can amass large amounts of information and they are patient enough to sift through it, using the underlying structural information to measure fitness. We can build tools to hold massive contexts and to gradually merge these into larger and larger ones. We can build tools to help leverage these huge structures to correct and fix grand problems.

We do some of this now, but we do it on an ad hoc basis with very specific modeling. The next generation of tools needs to do it across everything we know, rather than just for tiny sub-disciplines. It’s a model and merge problem, but one that spans the perspectives and understandings of billions of people, not just a small group of researchers.

If we can’t utilize our partial knowledge to improve things then we know that complexity will continue to grow, now at an increasing rate, up to some threshold where it in itself will become the most significant problem for our species. That is, technology, being neither good nor bad, is just an amplifier. And at the point that it expands things to be large enough, the core gets so watered down that the entire edifice will stall, or even worse it will implode. It is a recurring pattern in history, it also seems to be a physical limitation of our universe, and it is this path that we are currently traveling on.

We see that we can not go on like this forever, but our reliance on partial knowledge also means that without better tools we will. That no single individual can lead us away from our fate, that now not even a small number of people will suffice anymore. We need to harness a gigantic understanding in order to correct our path and we haven’t even begun to accept this as ‘the problem’ let alone started working on it yet. In a sense, technology got us into this mess; we have to admit that and redirect our resources to craft technologies that will really get us out, not just pile on more dangerous bandages or keep shifting the problems around in circles. This isn’t a practical problem that will get accidentally solved by a couple of people in a garage, it has grown too large for that now. If we don’t choose to accept, understand and work towards a solution the universe will eventually find one for us...