My last couple of blog entries have -- once again -- been procrastinations from finishing off a difficult post. Most often, my difficulties in completing a piece come from uncertainties about the essence of what I am saying. Probably a good thing, as I am often trying to stretch my understanding to a new level.
In this case, since what I'm talking about is what I think can be, there is no foundation for me to fall back on. I can't check my memories, or facts, or even look to other people for help. I have a vague outline of what I really want to say, but it's not as entirely as certain as the words I used to describe it.
Because of that, I would suggest reading this one lightly, it may prove to be substantial, but then again it may not. Of course, comments are welcome, and any addition thoughts are always appreciated.
CHANGING TIMES
The 20th century tore a hole right through Newton's deterministic world. On impact, his simple discrete model of cause and effect in a linear existence seemed to shatter into a million pieces.
Our advancements were amazing. The special theory of relativity redefined how we saw geometry. We no long live in a flat space, instead it is curving and bending around us. Quantum physics shifted our foundations, basing them on probability rather than actual matter. Things are now so little that we can only guess where they are. Chaos theory showed us that relationships aren't necessarily linear, small things can have huge unexpected effects. This gave us reasonable explanations for phenomenon like weather, but left us unable to predict it. And fractals revealed the naivete of our calculus by opening it up to allow recursive, constantly morphing functions that are infinitely deep. Now we can relate the complexity of detail back to our mathematics, as more than just a simple subset of common shapes.
Our models of the world around us have taken huge leaps. In this, for all that we lost in realizing the world is not a simple deterministic place, we gained it back again in understanding complex behaviors like fluid motion, weather patterns or other previously unexplainable material events. The world around us may not be simple, but it is still possible to understand it.
In one of my earlier blog entries, "The Age of Clarity", I predicted that the we would pass through a new Renaissance driven by our increased understanding of the information around us. To some degree, I suspect that many people will assume that I am falling back into thinking that the world is simple and deterministic; that this new age will allow us to model it conclusively with sciences similar to classic physics.
From our vast increases in knowledge I am certainly aware of the dynamic nature of the world around us, but that does not stop me from envisioning a discipline that can fully identify and integrate these behaviors into our world. Just not deterministically. All things, as we know them, leave an imprint on the fabric of space and time around us. We are surround by vast amounts of information, we can record it and understand its structure, its inter-relationships, even if we cannot entirely project how it changes in the future. We can project some changes, particularly large ones, even if we cannot aligned to them over time. We know more than we realize, but we understand less than we know.
MONITORING CHANGE
At some point in our not-to-distant future, I would expect that we could easily be able to collect any relevant information for real-life things or events. With this information we would be able to process it in a way that would allow us to model the effect of at least large changes, with some extraordinary degree of precision. With this type of model, we will be able to understand how changes have positive and negative effects, we will also be able to determine whether or not things are getting better or worse.
More specifically, I mean that we should be able to monitor an organization like a government, measuring its important aspects. This we should be able to accomplish quickly, possibly spending only a couple of days setting up the process. From there we would be able to model the key variables, with a full and complete understanding that any 'non-monitored' variables are 'non-essential' and that their impact on the overall model is insignificant.
By virtue of our knowledge, we will know absolutely all of the variables that are significant for this type of model and all of the variables that are not significant. And we will be right, because what we are doing has rigour. We will able to understand and collect the significant pieces of information that relate back to our real world. With this type of model, we can then exploratively make changes and see their effects.
As the government considers new laws and regulations, we should be able to alter the model to get an estimate of the effect of the changes. As the changes are made, we should be able to get a quantitative feedback that proves that the changes are having a positive effect. In the end, we should know, conclusively that we have in fact improved the government; not a guess or a gut feel, but solid evidence that things are better to some degree.
Now, that type of statement is easy in a deterministic world, but in ours, modeling an organization for instance and introducing changes, then recalculating a new direction to any accurate degree is nearly impossible using today's understandings. We need to guess at what information to capture, and then guess at how changes influence it. We don't have enough of an understanding of either, but that won't always be true.
Our goal then is too be able to measure the correct things, model them completely and then project these models forward to some degree of previously unobtainable level of accuracy. All extremely hard tasks. With our current understandings they are nearly impossible and definitely not rigorously defined.
Our current abilities, when we computerize things is to spend years flailing away at the design of large systems. Often we guess about the structure of the data, and spend a great deal of our effort trying to slowly fix the serious problems caused by our poor initial guesses. Even when we have collected huge piles of data, we barely know how to go through and mine it for simple things. If we know what we are looking for it is easier, but still difficult. It takes years, often fails and is never particularly rigorous.
There are lots of available technologies, books, theories, etc. but none of them are fully objective. We kinda know how to do it, and it kinda works, but often fails.
Another aspect I expect for the future is that in assembling large piles of data, we will be able to decompose them into their elementary influences. Thus new and unexpected relationships will get discovered, just because of the amount of data we have collected, and the process of finding this relationships will be rudimentary. Once we understand the structure of data, we will also understand how it relates to other data in the world around us. Mining those relationships will be easy.
Our systems in the future will just present to us all of the causal relationships based on the increasing data pile size. We will no longer have to guess, and then go in search of the answers to back up our assumptions. Instead we will assemble the data, and the answers will be plainly visible (although it may still take us lots of time to accept them).
THE MISSING SCIENCE
To get there, we'll need at least one new science, if not many. For these goals we need a way to really understand information, including both its structure and its relationships.
One of our truly powerhouse accomplishments as an intellectual species, has been the creation and refinement of mathematics. In a very real sense we can see it as being a rigorous way of relating together numerical and symbolic qualities in an abstract container that are always universally correct. It's the absoluteness that is key.
We have a huge number of different 'branches' of mathematics, dealing with a wide range of axioms and theorems. Some are quite disparate from each other, but they all hang together because they are abstract, they are rigorous and they deal with numerical and symbolic entities.
If we stick our head directly into the abstract plane where mathematics hangs out and squint a bit, we can see other 'things' there as well.
We know 'information' and using information theory we know for instance that there is some undeniable minimum size for any piece of data. Claude Shannon proved that 'information' can only be compacted so far down, and no more. In that 'sense' information has a physical properly in our world, even if we can't see it. Information is numbers and symbols, but it is also so much more. Some branches of mathematics do 'indirectly' deal with the relationships of data, but their focus is on the numerical or on the symbolic.
We know we need a new science to apply rigour on top of our data, and we know that we want it to be an abstraction like mathematics, but dealing with information instead. I like the term 'informatics' as a name for this new science, but there is already an existing definition. The dictionary definition for informatics relates this term to Computer Science and to the study, storage, retrieval and classifying of information, however I would really like to dis-connect this term from any underlying hardware, software or even reality.
I'd prefer to recast this definition as the 'abstract' study of the structure and relationships of information, focusing on the shape and interrelationships between datum. Like numbers and symbols in mathematics, data may be markers for the real-world, but if we contain them within a rigorous abstract foundation, we can fully understand their nature prior to it becoming clouded by reality.
STRUCTURE AND RELATIONSHIPS
For everything we need a foundation; for this we want to study the abstract structure and relationships of data that mirror our real-world.
All information in our real world has a placeholder in any abstract one. We want to be able to conclusively say interesting things about the structure and relationships of the various placeholders. We want to be able to know how they are organized and see how they change over time.
Right now for instance, we may know some of the elements of how to model complex some real world entities, such as bank accounts or financial instruments. A lot of work over the last 50 years has gone into gradually producing more reliable models. Interesting, virtually none of it has been formal in anyway. We may know a lot about creating abstract models for financial instruments for example, but this is experimental propriety custom knowledge that is not shared.
The next step is to put structure and rigour to this knowledge. We don't just want a huge number of custom models, we want some unifying theory of how to reliably model the world around us. We need to dig into the foundations of information.
All information has a name. In fact most of it has many names. The names usually belong to complex taxonomies, and often have instance-related elements. For example, $128.02 is the price in CAD on June 5th, 2007 of a slice of a 1995 Canadian Govt Bond as traded in Toronto. BondPrice is a shorter name, and relative to things like market (Toronto) and date (June 5th).
Beyond that, a Bond is a type of financial instrument which is a type of contract, which is an agreement between two parties. So, we have both instance information and type information. For our purpose in this example we can mix the instance, type and categorical information, not distinguishing between the different varieties.
In an abstract sense, we can see the various breakdowns as being structural information about the underlying data. From some point, relative or absolute, we can construct a long-name, as a series of labels, that leads us through the structure and down to a specific datum.
The various parts of the long-name (price, date, instrument name, market) all form a pathway through the structure to describe the data. A collection of related pathways for similar data outlines the complete structure. An instance of price/yield information for a specific day, in a specific market is a data-structure. The static properties of the bond, its name, type, creation date, etc. all form a data-structure. These structures, plus many others form the structure of financial instruments. All data has some underlying structure that relates it various parts to each other.
While the pathways may be relative, or change with language, science or other categorical process, the underlying type structure of information never changes. Its structure is its structure. The idea is that underneath the same basic thing exists whether we use French or English to describe it. It exists whether or not we describe it relative to ourselves or to some fixed point like the universe. Given two different descriptions of data, if they are close to complete, then they are close to having the same structure and inter-relationships.
That is a huge idea, the notion that although our way of representing data at the high level changes, the underlying structure of data is immutable. Fixed. It does not change, it exists in the world with a specific static structure. The world changes around it, but it is the same in any language, with a relative or absolute point of reference.
Oddly, the static-nature of the information is there by definition. If the structure of the data is different, then the thing being described is different. They might be related, and similar but from a structural standpoint differences mean things are not the same. Not that that implies that instances cannot vary from each other, that will also always be the case. Dogs always have legs, usually four, and they come in a variety of colors and sizes. Variations are normal, changes in structure are not.
To utilize this, we need a way to objectively start with one datum, and use it to understand rest of the structure of the information we want to capture. One way, for now is to start assembling meta-models of all of the different data types. If we assemble models of financial instruments, social organizations, etc. We will be able to understand their existing structure. For every industry were we can model, we can be objective about the structure.
At some point however, understanding the structure should get beyond observation, particularly as we build up larger and larger abstractions for containing our world. We'd like a deterministic way of walking the information that by definition proved that the structure was actually correct. An algorithm for reliably mapping any point of data in a bigger picture.
MAPPINGS AND DIMENSIONS
All things are different to varying degrees. The strength in understanding any structure is being able to find isomorphisms and other mappings between the data at different levels. There are rather obvious mappings such as changes in language or terminology, and there are more exotic ones such as the similarity in structure between the events for different objects, or common patterns of data structures. Whatever type of mapping, it provides a way in which we can get a higher-level understanding of the underlying information. We always generalize to build up our knowledge, based on lots of individual bits.
For some types of information, the decomposition at some level is arbitrary. In that sense, it falls into some set of primitives at a specific level in the structure. As we seen, the underlying structure is immutable, then for some higher-level set of primitives where there are multiple ways of expressing them, there must be a mapping between the different sets.
An easier way of saying this, is that if we express something in terms of AND, OR and NOT primitives, at the same level, NAND will work as well. Anything expressible in the one set of primitives is also expressible in the other. Because they are the same, there is a mapping between them (although not necessarily one-to-one or onto).
If in a different language for example, there might be 50 words that describe types of snow. The mapping from English to the other language will have to include descriptive verbs as well as the noun 'snow'. Because the primitives decompose differently, that doesn't make the underlying thing different, it is just an alternative geometry for expressing it.
Different primitive sets will have different sizes, but still represent morphisms from one of set to another. We can show the expressiveness, and the coverage of each of the primitives and prove that the coverage of one set is identical to the coverage of another, thus the two are essentially the same. Thus information broken down into one set of primitives can be mapped to some other set of equivalent primitives.
Besides primitives, there are some interesting mappings that occur because all of the information in our real world exists in four dimensions. In our language we use nouns and verbs to describe the things in our space.
Nouns tend to be tied to objects sitting our our 3D space of geometry, while verbs, being actions are more often tied to our 4th time dimension. We also relate our 'business' problems in terms of nouns and verbs. We can by definition discussion all aspects of business using our verbal language. Putting structure directly to our communication decreases the likelihood of misinterpreting it.
Our language itself underneath is tied to the structure of our physical existence. That is an expected consequence because language is in a very real sense is just another space on which to bind real-world things onto abstract concepts such as words and sentences. Languages are loosely-structured, ambiguous, non-rigorous models of the world around us, but they are still models.
Similar mappings are important, but they are not the only class of mappings that have interest or value. Knowing as we do, that much of our world is 'fractal-based', we need some less rigid concept for mappings that are similar but not truly isomorphic, possibly a term like 'fractal-morphic', with a meaning of a twisted or skewed mapping. The scope of the mappings is complete, e.g. both sides are the same size, but the relationships are transformed with respect to one or more dimensions. A sense of degree would help redefine the concept, so we could say something was 2 degree fractal-morphic to something else, meaning that it was skewed in no more than 2 dimensions.
A corner-stone of any understanding of information will be creating rigorous definitions for these different types of similarities. If we could for instance boil the world down into a few dozen or so major meta-structures, then it becomes very easy to determine the properties of the data. What we know in general can help us to deal with the specific.
A SIMPLE HIERARCHY
Stepping back up to the higher-level, we have two abstract bases of theory. One specializes in numerical and symbolic qualities: mathematics, and the other specializes in structural and relationship attributes of information. As pieces, these two then fit into each other quite nicely.
With the various branches of mathematics we manipulate numbers or symbols in numerical ways. It is possible for one to believe that numbers form the base layer of all things, so that mathematics is and will always be the primary abstract field. However, where we can find mathematical expressions for the structure and relationships, we can also find structural and relationship expressions for the mathematics.
The structure and relationships of information is similar, but in many ways my expectation is that it supersedes mathematics. In a way, mathematics is the concrete 'relationships' between the abstract things in an abstract place, so the relative structure and abstract relationships should be a higher level of abstraction.
Irregardless of where it fits in, the two key properties for this new science of information are that it is abstract and that it is rigorous. Both are necessary to position it for maximum usability. The foundations need to be self-evident and universally true. It should not matter were you are, the basis is never changing. Even though we are dealing with 'real-world' information, the essence of what we are working with is totally abstract.
Some things will be tied to concrete entities, even thought the science itself is abstract. Some things will not. The parallel to mathematics can be seen in simple things like 'complex' numbers. They do not exist in our world, yet they can be the roots of equations that do have physical bindings. In that same way, there will be meta-data that comes to play, but has no real world equivalent. That arbitrariness may seem confusing, but it is not uncommon to have to leave the base dimensions of a problem to find or quickly compute a solution, some linear programming algorithms for example do that.
SOCIAL FACTORS
Some of what I image possible is the ability to better understand our social organizations. There are consequences to owning this knowledge.
The 20th century saw tremendous strides towards making manufacturing consistent. The idea that we can monitor, and control for quality the physical output of a lot of people, in an almost mechanised way made a huge difference in manufacturing. The modernization of factories was significant. It caused upheaval, but ultimately our societies integrated the changes.
The revolution was stunning, but it was really confined to the way we saw physical activities. One of the things we can't do, is apply those rules towards intellectual endeavours. We learned how to consistently build a model-T, but not how to consistently fill out a form or run a meeting.
Although using muscles is different than using the brain, the two share a relationship to each other. The biggest difference being that intellectual effort can't be visibly measured. This seemingly makes it an intangible quality like confidence or morale. Yet we know it is there and we know the output of it. We also know for instance that bad morale reduces physical and mental output, so it in itself is not really intangible. We just don't know how to measure it properly.
One of the things we should be able to do with our advanced understandings, is see these things based on their imprints from the world around us. In large organizations like governments, we should be able to determine how effective is the organization, and how many problems it has. We should be able to see the effects of 'intangible' qualities, and see how changing simple things improves or degenerates the circumstances. Intangible things may not be physical, but by definition they have some sort of an effect, or they are imaginary.
Some day we will be able to model these qualities. To get there we need to cross over the point where we understand how to mechanize intellectual effort because we need to relate it back some type of underlying model. This is the fear of all white collar workers, that we can measure their work. But until we can, we can't tell for instance how effect a social policy is. It is a horrible trade-off, but a necessary one.
It is the dream of the executroids, and the night-mare of the workers. But we must remember that factory-workers survived, so shall the white-collar workers and middle management.
SUMMARY
At some point, with each new discovery we shed light on a new and smaller, more refined set of details. Then over time as more details emerge we often coalesce our knowledge back into larger abstract theories. In a real sense we see the trees first, then the forest. We are forever in a cycle of slowly building up our knowledge.
It is amazing the things we know, but it is also amazing how we cannot truly verify them. Everyday we are bombarded by facts and figures based on bad science and dubious mathematics. Underneath all of this, is our use of computers to amplify the amount of data we are collecting and mis-interpreting.
We need help in quickly identifying bad information. You could spend time in a study for instance, to prove that the process was flawed, but it would be far easier just to discount it because it failed to use proper information gathering techniques. Since we don't have those specified, we have to go to the results on a line-by-line basis even thought we can easily suspect that the problem was the initial data gathering process.
Because of this, too much that should be objective, has been rendered subjective. We cannot be sure of its validity, and now often different groups are openly contradicting each other. We've fully fell into the age of mis-information, because we have easy ways of building up large piles of information but no easy ways of validating that those piles actually match reality in any shape or form.
It was computers that created this problem, so it will have to be computers that get us out of it in the end. Once we understand how to understand information, we can then go about recollecting it in ways that will really provide us with true underlying knowledge. With enough of an understanding, much of what is now very murky, will come clear.
We lack an understanding of our information, but now at least we know what it is that is missing. If our next step is to create a real science of information, one that is undeniably and universally true, then we can give ourselves the tools we need to really make sense of the world around us. Realistically it is not that complicated to get a Renaissance, if you can figure out that you are a pre-Renaissance culture. It is accepting where you are that is hard.
Software is a static list of instructions, which we are constantly changing.
Showing posts with label clarity. Show all posts
Showing posts with label clarity. Show all posts
Tuesday, March 11, 2008
Saturday, February 2, 2008
The Age of Clarity
"What do we really know? Hmmm." I pondered as we walked.
I was out with the dog the other night. The quiet tranquil nature of empty suburban streets is a great place for deep thinking. The cold chill of winter keeps one from wandering too far off topic while wandering aimlessly in the streets. Dogs make wonderful intellectual companions for these types of journeys because they don't interrupt with too many questions. They are very good listeners.
I was pondering information quality, and I foolishly started to wonder about how much inaccurate information was choking up my memory. Certainly, there are lots of spin, lies, half-truths, deceptions and other stuff built up over the years from less than quality sources like politics, news and TV. Somethings in my memory are just easy simplifications. Somethings are out right fabrications. There is also the changing nature of science, and our non-stop quest for learning. Some of my knowledge is just 'relative', it wouldn't stand up to a universal judge. It is considered true here and now, but won't be in the future. In an overall sense, how much of this is really accurate?
If you factor in all of the different reasons for low quality, and take a big sweeping guess, the amount of truth in our brains could be lower than 30%. Just a wild guess, but I could easily believe that 1 in every 3 three facts in my brain are true, while the other 2 are questionable for all sorts of reasons. I am just speculating of course, but in this misinformation age, we are full of a tremendous amount of low quality knowledge. And it feels like it is growing at an ever increasing rate, although that might just be our ability to confirm that it is suspect.
THOSE WHO FORGET HISTORY ARE DOOMED TO REPEAT IT
In the past, mankind was mostly ignorant of the accuracy of their information. They could take pride in their depth of knowledge without ever knowing how dubious it really was. Now all we have to do is check wikipedia and we can instantly find out that truth, well, err at least a pointer towards the truth.
How often have I pulled forth some ancient fact from the depths of my brain, only to discover that it was fundamentally untrue? Worse still is how those facts actually make it into my head in the first place. Some were obviously from disreputable sources, but others had come from well-known authorities, and were still incorrect. My problem is not loss or corruption of memory, it is the opposite, these 'facts' stay for far too long. If I just dumped them faster, I might find they were more accurate overall.
It is oddly telling. It allows us to guess that this huge degree of inaccuracy in our current knowledge is actually some type of pointer towards the future. The Renaissance was an awaking about the world that we live in. A moment when we first opened our eyes and saw it for what it actually was. This in turn drove the foundations for the industrial age, where we learned to create and use an unlimited number of machines. One of those machines, the computer, has driven us into an information age, where we collect huge piles of information, about virtually everything in this world. There is a trend here. The next age will follow along in this sequence.
Even though we have built up a tremendous collection of fantastic machines, they do not serve us well. We can build things, but we have trouble maintaining them. Our massive and complex cities crumble around us. We are forever fighting a losing battle against entropy; like a runner that has leaned too far forward we are continually off balance. We must continue to build to move forward, we don't know how to preserve what we have and we don't know how to live within our environmental means. We grow at a severe cost to the world around us.
With all of our equipment and learning, collecting data is still a hit or miss proposition. We just guess at what we want to collect and how it is structured. It is not orderly and we don't have any underlying theories that drive our understanding. Computer Science is still so young that is frequently wrong. Often it is just random guessing. We are currently only utilizing a small fraction of the capabilities of our computers because we keep bumping into complexity thresholds each time we try to build truly sophisticated systems. We are trapped with crude software.
Even thought we can collect the data, we continuously fail to be able to mine or interpret it. We gather the stuff, format it and then save it to backup tapes. But we get little actual use from all of our work in collecting it. Some decisions are made from the data, but given the real amount of underlying information contained in our efforts we could actually use what we have to make really sophisticated decisions. To actually know, for fact that we are changing things for the better. If we understood what we have.
THE YELLOW BRICK ROAD
Given those trends, it is not hard at all to predicate the future. The next step in the sequence. The path we must take is the only one available: machines to labour for us physically and mentally lead to vast infrastructures and vast piles of information. We built up these things, but we don't understanding them, and we have trouble keeping them going.
It is not like we will wake up one day and the light will get turned on, but I imagine that over time like a dull and steady wind blowing away the haze, much of what we know will become clear and finally fit into place. It will be a modern day Renaissance reoccurring not with our perspective of the world around us, but with our perspective of the information and knowledge that we have collected. It will take time. Many years, decades or even a century or two, but one day there will be an 'age of clarity', where mankind can finally see the information around them for what it actually is. That is, if we survive the turmoil of our current societies; we have so many dangers that await us, because of what we know, but don't yet understand.
And what could we expect in such an age? I imagine that we will have a real understanding of information, probably based on a currently unknown science. Maybe several. We will know how to quickly, conveniently monitor and collect information for any questions. Inherently, we will understand the truthfulness of what we collect, and we'll be able to immediately use this information to ascertain whether things are improving or getting worse. The term 'immediately' being one of the very key points.
Unlike now, this won't be a big effort, but rather something simple that people do as a matter of due diligence. Government effectiveness for example, will be based on simple true numbers that show that things are improving or getting worse. Unlike the statistics of our day, these numbers and their interpretation, based on science will be irrefutable. We will be able to show cause and effect relationships between policies and real life. We will be able to measure the effectiveness, not guess at it. If we say things are getting better, it won't just be 'spin'.
Underneath, if we capture enough data, we will get a vibrant picture of all of the relationships, how they fit with each other and what they really mean. When we choose to make changes, they will not be partially-informed guesses, they will be tangible deterministic improvements to our societies that will work as expected. In the same way that the industrial age leapt from wildly building things to the reproducible industrialization of products with a tremendous amount of consistent quality, we will shift our understanding of the data around us. Like the difference between B&W photography and color we will learn how to start really capturing the information that is of real value, and we will learn how to really interpret it.
AT THE EDGE OF REASON
Does it sound too overly deterministic or crazy? Whatever comes in the future, it must be something that isn't here now. So, if it isn't pushing the envelop of convention, then its not really much of a prediction, is it? Jules Verne wrote about ships that travelled underwater, a famously crazy concept if ever there was one, except that it now has become common knowledge. He wrote about air ships, defying gravity and hanging in the sky with birds, clearly another bit of wackiness. Yet, this too is common, and rather boring now.
Does it sound very similar to what we have now? For all we know, we know so very little. We have many approaches and methods to really prove things, to get the the real underlying truth, but because we can't do that easily on a grand scale we are awash in misinformation. All of this low quality knowledge chokes our pathways and keeps us from progressing. It becomes food for subjective arguments, and endless discussions. And while some of us may suspect falsehoods, proving it is costly and often distracting. We can't fight all of the battles all of the time, so the low quality stuff washes over us like a tsunami.
Sometimes when I am out walking the dog, my mind drifts around to us being so sophisticated that there is not much left in this world that we don't know. That 'proposition' is comforting in many ways, but patently false. Like the pre-Renaissance societies, we think we have reached some level of sophistication, but we barely even realize how to keep our own existence from spiraling out of control. And what we don't know, is the question: "what do we really know?" We feel pride in having built up knowledge bases like the World Wide Web, but realistically the things are a mess. What good is a massive unorganized pile of data, if we can't use it to answer the serious questions in our lives? We live in an age where subjective arguments are possible for most of what we commonly deal with in our lives. Everything is up for grabs; everything is based on opinion. We can barely distinguish the quality of our facts, let alone position them into some coherent and universally correct structure of the world around us. For all that we know, we are still incredibly ignorant.
The next big thing then is obvious. If we are too survive, then we have to pass through the Clarity Age. We have no choice. If this understanding hasn't already popped into someone else's brain, it was bound to sooner or later. You can't get very far down the path, if you don't know where the path lies. It is murky now, and for us to progress it must be clear.
Woof, woof, woof! My thinking and wanderings were interrupted because the dog spotted a raccoon. I was riped from the depths by the pulling, jumping and barking. In the here and now, I am reminded that it is best if I move on quickly to keep the dog from making too much racket. I don't want to wake up my whole neighborhood with the commotion. However much I long to spend time in the future, I must live with the world around me as it is now. These dark ages are apt to last a while, possibly my entire life. I ought not to waste it, pining for enlightenment.
I was out with the dog the other night. The quiet tranquil nature of empty suburban streets is a great place for deep thinking. The cold chill of winter keeps one from wandering too far off topic while wandering aimlessly in the streets. Dogs make wonderful intellectual companions for these types of journeys because they don't interrupt with too many questions. They are very good listeners.
I was pondering information quality, and I foolishly started to wonder about how much inaccurate information was choking up my memory. Certainly, there are lots of spin, lies, half-truths, deceptions and other stuff built up over the years from less than quality sources like politics, news and TV. Somethings in my memory are just easy simplifications. Somethings are out right fabrications. There is also the changing nature of science, and our non-stop quest for learning. Some of my knowledge is just 'relative', it wouldn't stand up to a universal judge. It is considered true here and now, but won't be in the future. In an overall sense, how much of this is really accurate?
If you factor in all of the different reasons for low quality, and take a big sweeping guess, the amount of truth in our brains could be lower than 30%. Just a wild guess, but I could easily believe that 1 in every 3 three facts in my brain are true, while the other 2 are questionable for all sorts of reasons. I am just speculating of course, but in this misinformation age, we are full of a tremendous amount of low quality knowledge. And it feels like it is growing at an ever increasing rate, although that might just be our ability to confirm that it is suspect.
THOSE WHO FORGET HISTORY ARE DOOMED TO REPEAT IT
In the past, mankind was mostly ignorant of the accuracy of their information. They could take pride in their depth of knowledge without ever knowing how dubious it really was. Now all we have to do is check wikipedia and we can instantly find out that truth, well, err at least a pointer towards the truth.
How often have I pulled forth some ancient fact from the depths of my brain, only to discover that it was fundamentally untrue? Worse still is how those facts actually make it into my head in the first place. Some were obviously from disreputable sources, but others had come from well-known authorities, and were still incorrect. My problem is not loss or corruption of memory, it is the opposite, these 'facts' stay for far too long. If I just dumped them faster, I might find they were more accurate overall.
It is oddly telling. It allows us to guess that this huge degree of inaccuracy in our current knowledge is actually some type of pointer towards the future. The Renaissance was an awaking about the world that we live in. A moment when we first opened our eyes and saw it for what it actually was. This in turn drove the foundations for the industrial age, where we learned to create and use an unlimited number of machines. One of those machines, the computer, has driven us into an information age, where we collect huge piles of information, about virtually everything in this world. There is a trend here. The next age will follow along in this sequence.
Even though we have built up a tremendous collection of fantastic machines, they do not serve us well. We can build things, but we have trouble maintaining them. Our massive and complex cities crumble around us. We are forever fighting a losing battle against entropy; like a runner that has leaned too far forward we are continually off balance. We must continue to build to move forward, we don't know how to preserve what we have and we don't know how to live within our environmental means. We grow at a severe cost to the world around us.
With all of our equipment and learning, collecting data is still a hit or miss proposition. We just guess at what we want to collect and how it is structured. It is not orderly and we don't have any underlying theories that drive our understanding. Computer Science is still so young that is frequently wrong. Often it is just random guessing. We are currently only utilizing a small fraction of the capabilities of our computers because we keep bumping into complexity thresholds each time we try to build truly sophisticated systems. We are trapped with crude software.
Even thought we can collect the data, we continuously fail to be able to mine or interpret it. We gather the stuff, format it and then save it to backup tapes. But we get little actual use from all of our work in collecting it. Some decisions are made from the data, but given the real amount of underlying information contained in our efforts we could actually use what we have to make really sophisticated decisions. To actually know, for fact that we are changing things for the better. If we understood what we have.
THE YELLOW BRICK ROAD
Given those trends, it is not hard at all to predicate the future. The next step in the sequence. The path we must take is the only one available: machines to labour for us physically and mentally lead to vast infrastructures and vast piles of information. We built up these things, but we don't understanding them, and we have trouble keeping them going.
It is not like we will wake up one day and the light will get turned on, but I imagine that over time like a dull and steady wind blowing away the haze, much of what we know will become clear and finally fit into place. It will be a modern day Renaissance reoccurring not with our perspective of the world around us, but with our perspective of the information and knowledge that we have collected. It will take time. Many years, decades or even a century or two, but one day there will be an 'age of clarity', where mankind can finally see the information around them for what it actually is. That is, if we survive the turmoil of our current societies; we have so many dangers that await us, because of what we know, but don't yet understand.
And what could we expect in such an age? I imagine that we will have a real understanding of information, probably based on a currently unknown science. Maybe several. We will know how to quickly, conveniently monitor and collect information for any questions. Inherently, we will understand the truthfulness of what we collect, and we'll be able to immediately use this information to ascertain whether things are improving or getting worse. The term 'immediately' being one of the very key points.
Unlike now, this won't be a big effort, but rather something simple that people do as a matter of due diligence. Government effectiveness for example, will be based on simple true numbers that show that things are improving or getting worse. Unlike the statistics of our day, these numbers and their interpretation, based on science will be irrefutable. We will be able to show cause and effect relationships between policies and real life. We will be able to measure the effectiveness, not guess at it. If we say things are getting better, it won't just be 'spin'.
Underneath, if we capture enough data, we will get a vibrant picture of all of the relationships, how they fit with each other and what they really mean. When we choose to make changes, they will not be partially-informed guesses, they will be tangible deterministic improvements to our societies that will work as expected. In the same way that the industrial age leapt from wildly building things to the reproducible industrialization of products with a tremendous amount of consistent quality, we will shift our understanding of the data around us. Like the difference between B&W photography and color we will learn how to start really capturing the information that is of real value, and we will learn how to really interpret it.
AT THE EDGE OF REASON
Does it sound too overly deterministic or crazy? Whatever comes in the future, it must be something that isn't here now. So, if it isn't pushing the envelop of convention, then its not really much of a prediction, is it? Jules Verne wrote about ships that travelled underwater, a famously crazy concept if ever there was one, except that it now has become common knowledge. He wrote about air ships, defying gravity and hanging in the sky with birds, clearly another bit of wackiness. Yet, this too is common, and rather boring now.
Does it sound very similar to what we have now? For all we know, we know so very little. We have many approaches and methods to really prove things, to get the the real underlying truth, but because we can't do that easily on a grand scale we are awash in misinformation. All of this low quality knowledge chokes our pathways and keeps us from progressing. It becomes food for subjective arguments, and endless discussions. And while some of us may suspect falsehoods, proving it is costly and often distracting. We can't fight all of the battles all of the time, so the low quality stuff washes over us like a tsunami.
Sometimes when I am out walking the dog, my mind drifts around to us being so sophisticated that there is not much left in this world that we don't know. That 'proposition' is comforting in many ways, but patently false. Like the pre-Renaissance societies, we think we have reached some level of sophistication, but we barely even realize how to keep our own existence from spiraling out of control. And what we don't know, is the question: "what do we really know?" We feel pride in having built up knowledge bases like the World Wide Web, but realistically the things are a mess. What good is a massive unorganized pile of data, if we can't use it to answer the serious questions in our lives? We live in an age where subjective arguments are possible for most of what we commonly deal with in our lives. Everything is up for grabs; everything is based on opinion. We can barely distinguish the quality of our facts, let alone position them into some coherent and universally correct structure of the world around us. For all that we know, we are still incredibly ignorant.
The next big thing then is obvious. If we are too survive, then we have to pass through the Clarity Age. We have no choice. If this understanding hasn't already popped into someone else's brain, it was bound to sooner or later. You can't get very far down the path, if you don't know where the path lies. It is murky now, and for us to progress it must be clear.
Woof, woof, woof! My thinking and wanderings were interrupted because the dog spotted a raccoon. I was riped from the depths by the pulling, jumping and barking. In the here and now, I am reminded that it is best if I move on quickly to keep the dog from making too much racket. I don't want to wake up my whole neighborhood with the commotion. However much I long to spend time in the future, I must live with the world around me as it is now. These dark ages are apt to last a while, possibly my entire life. I ought not to waste it, pining for enlightenment.
Subscribe to:
Posts (Atom)