The Depth of Knowledge

A few decades back, I decided that I wanted to know more about quantum physics. I've always been driven to indulged my curiosity of various disciplines, it's been a rather long-standing obsession since I was a kid. 

I went out to the bookstores (remember those :-) and bought a huge stack of books on what might be best described as the 'philosophy' underpinning quantum physics; stuff like the wave/particle duality and Schrödinger's rather unfortunate cat. It was fun and I learned some interesting stuff. 

I'm fairly comfortable with a wide range of mathematics, so I bought a book called Quantum Theory by David Bohm. The first chapter 'The Origin of the Quantum Theory' nearly killed me. I was so hopelessly lost by the first sixteen pages. I'm sure if I dedicated years to studying, I could at least glean 'some' knowledge from that tome, but it was likely that it just wasn't possible to get there as a hobby.

Knowledge has both breadth and depth. That is, there is a lot of stuff to know, but getting right down into the nitty gritty details can be quite time-consuming as well. Remembering a long series of definitions and facts is useful, but knowing them doesn't necessarily help one to apply what they've memorized. Knowledge is both memory and understanding. My leap into quantum physics is a great example, I learned enough to chat about it lightly at a party but not enough to actually apply it to anything useful, not even chatting to an actual quantum physicist. I got a very shallow taste of the higher level perspective, but even after all those books I still didn't really understand it.

That knowledge is both wide and deep causes lots of problems. People can easily convince themselves that they understand something when they've only scratched the surface. It's like trying to make significant contributions to a field were you have only taken the introductory 101 course. If it's a deep field, there are years, decades and possibly centuries of thinking and exploration built up. What's unknown, is buried so deeply below that accumulated knowledge. Finding something new or novel isn't going to be quick. If you want to contribute new stuff, you really have to build on what is already known, otherwise you'll just be right back at the starting point looking over well-tread ground.

And some knowledge is inherently counter-intuitive. That is, it contradicts things you already thought you knew, so that you literally have to fight to understand it. Not everything we understand is correct, some of it is just convenient over-simplifications to make it possible for us to operate on a day-to-day basis. That 'base code' helps, but it also hinders our deeper understanding. We see what we need to, not what is really there.

It's pretty amazing how deep things can really go, and how often what's under the surface is so different from what lies above. My curiosity has led me to all sorts of unexpected places, one of my favorite being how much can be learned from really simple systems like boolean equations. Going down, it can feel like a bottomless pit that endlessly twists and turns each time you think you it can't go any further. With so much to grok from just simple things, it's truly staggering to contemplate the full scope of what we know collectively, and what we've forgotten. That 'field' of all things knowable has exceeded our own individual capacities a long time ago, and has steadily maintained its combinatorial growth since then. At best we could learn but a fraction of what's out there.

Software is an interesting domain. It's breadth is huge and continuously growing as more people craft their own unique sub-domains. A quick glance might lead one to believe that the whole domain is quite shallow, but there are frequently deep pools of understanding that are routinely ignored by most practitioners. Computational complexity, parsing, synchronization, intelligence and data mining are all areas that sit over deep crevices. They are still bottomless pits, but depth in these areas allows programmers to build up the sophistication of their creations. Most software is exceptionally crude, just endless variations on the same small set of tricks, but some day we'll get beyond that and start building code that really does have the embedded intelligence to make people's lives easier. It's just that there is so much depth in those sort of creations and our industry is only focused on expanding the breadth at a hair-raising pace. 

It's easy to believe that as individuals we posses higher intellectual abilities, but when you place what our smartest people know along side our collective knowledge, it becomes obvious that our inherent capabilities are extremely limited. Given that, the rather obvious path forward is for us to learn how to harness knowledge together as groups, not individuals. Computers act as the basis for these new abilities, but we're still mired in the limits of the personalities involved in their advancements. Software systems, for example, are only as strong as their individual  programmers and they are known to degenerate as more people contribute. We only seem to be able to utilize the 'overlap' between multiple intellects, not the combination. That barrier caps our abilities to utilize our knowledge.

Despite all we know, and all of the time that has passed, we are still in the early stages of building up our knowledge of the world around us. If you dig carefully, what you find is a landscape that resembles swiss cheese. There are gapping holes in both the breadth and depth of what we know. Sure, people try to ignore the gaps, but they percolate out into the complexity of modern life at a mind numbing rate. We rooted our cultures on a sense of superiority over the natural world, but clearly one of the largest things we still have to learn is that we are not nearly as smart as we think we are. If we were, the world would progress a lot smoother than it has been. Knowledge is after all, a very significant way to avoid having to rely on being lucky.