An Informal Ramble

[ Author’s Note: I really don’t know what to say. It was just one of those days. If you don’t feel like reading a broad rambling on loosely connected ideas, please don’t read this! ]

A good place to start -- as always -- is with a few definitions. For this particular discussion it works better if I create them loosely and use somewhat non-conventional meanings. I’ll try to explain why that helps as I go along.

The first thing I want to define is a ‘system’. For this discussion it is a set of rules that act on ‘stuff’, where I intend stuff to be nearly as vague as possible. Stuff could mean static objects, or it could mean dynamic processes, or anything in between.

Systems can be broken down into two loose categories: formal and informal. A formal system rigidly constrains the stuff, so that it stays within the space occupied by the rules. That is, stuff is in the formal system when it obeys the rules, and it is invalid or outside the system otherwise. Thus the rules are ‘formal’ and essentially unbreakable.

Informal systems, on the other hand, obey the rules most of the time but there are plenty of things that can occur within the system that are outside of the rules. That is, an informal system loosely defines what most of the rules are, what the boundaries are, but there are always exceptions and somehow by some ‘stickiness’ these still fall within the system.

To put these two definitions in more candid terms: a formal system is black and white and an informal one is just shades of grey. Both are systems and have what amounts to deterministic behavior, but one only approximately follows its rules, while the other is bound by them.

So, why these definitions? Because we can apply them to many things within our world. The rigid abstract formalisms of mathematics are obviously formal systems. The rules are the axioms, the definitions, the lemmas, the theorems, etc. that are built up on top of the stuff which is the underlying well-defined mathematical objects. There are many different branches in mathematics and all of them have a plethora of underlying formal systems. Formal systems can also be built on top of other formal systems, by creating relationships between them.

So a very simple formal system in mathematics is a set of operators like: plus, minus, multiply and divide, that works on objects like: the set of all whole (positive) numbers. What is interesting about this system is that the subtract operator, when used on a small number and a much larger one, will produce a new number that is no longer within the whole numbers. That is 3 - 10 = -7, which is a negative number.

There are a number of ways to view this, but again I’ll go a little unconventional and say that the above formal system has an ‘exit’ point into a larger formal system. That is, if I had chosen integers, which include the negatives then there would be no problems with the subtract operator. Thus we can consider the first formal system as being an incomplete subset of the second, with respect to subtraction. We can also consider it to be an incomplete subset of one based on the continuum of real numbers, with respect to division. This gives us a large set of structural relationships between the various systems. These distinctions may seem a little weird, but they will come in useful later.

In contrast to a formal system, an informal one is not so well-defined. There are rules and there is stuff, but there is also a great deal of flexibility built in. That is, there are no real ‘exit points’. If something does not fit within the system, somehow it still seems to remain there. The types of informal systems I am thinking of include all different sizes of groups of people. Companies, clubs, governments and all other organizations are informal systems. They have rules which bind them together, but do change from time to time. Other examples are markets, schools, disciplines, professions, soft sciences, etc.

You may note that all of the informal systems I’ve listed so far are based around people. That’s more of a coincidence rather than a deliberate intent, for there are plenty of systems out there that we have tried to rigorously formalize but our underlying knowledge is incomplete, or the basis of the system is just too chaotic. Evolution, physics and weather are good examples. These too are informal systems, but our underlying comprehension of the rules is gradually approaching formalization. That is, rather than an exit point, outliers get feed back into the informal system to enhance our understanding of it as it approaches, but never quite reaches, formalization.

And this brings up a very important point. At the bottom of everything, as far a s we know, are particles. And although we don’t know all of the rules that define their behavior, we are pretty sure that the rule set is static. Thus there is one underlying formal system that quite possibly binds everything. That is, nothing can happen unless it is a physical possibility. On top of this we get ever increasing ‘epiphenomenon’; larger and larger collection of particles that at their collective levels behave within a set of what are essentially meta-rules. So we get galaxies, planets, stars, etc. And built on these we get even more: water, earth, plants, people, etc. And on these we build up even higher to stuff like buildings, farms, etc. And some epiphenomenon that are less physical, but still bind together stuff like organizations, companies, countries, etc.

So it would seem that for all of our informal systems, they exists within the bounds of lower and lower systems that eventually fall back to one rather low-level formal system.

What is rather odd though, is that we are also aware of the multitude of mathematical formal systems out there, and these seem to exist without any ties to those based on the many layers of epiphenomenon ones. We do however, use them to explain the others. That is, physics is a set of formal systems that are remarkably accurate in explaining the physical world around us. So in that very sense, mathematics is the meta-formality that binds together the epiphenomenon with some degree of precision. Thus it has its roots in our physical world, even if it just appears as an abstraction based on our intellectual thinking capabilities.

Any real argument about the how tangible the formal systems of mathematics are has to take in consideration computers. For the first time, our species has manager to instantiate one of our previously abstract-only formal systems -- Turing machines and/or lambda calculus -- into a physical reality. Well almost, since computers are still bound by finite resources and by the occasional hardware failure. Regardless, it is still a tremendous accomplishment and their existence has been rapidly transforming our societies ever since. One might argue that how physics models the world is similar, but we need to consider that that link between math and reality depends on a person utilizing the models in their head, in order to act on the knowledge. Thus the formal systems that underpin physics stay internal to our minds. Computers, on the other hand, do their computations entirely independently of people, and thus are a concrete physical manifestations of a formal system. A very different ballgame.

So why are these definitions of formal and informal systems useful? One of the main tasks of software developers is to construct useful formal systems that solve problems for people working in informal ones. That is, any software we write must be formal in its construction, but its usage lies in one or many informal systems.

If you are writing a system to track how a company deals with its clients, the company and the behavior of the clients is entirely informal. Most things happen according to plan, but lots of exceptions occur and the nature of the relationship changes as the market expectations change. But those rules need to be formalized, so they can be coded, so the computer can execute a deterministic series of instructions with each interaction between the client and the company.

From this angle, you can see the problem right away. The informality of the base systems leads to difficulties in finding a formal one that both precisely matches them and remains matching for any prolonged period of time.

There are ways around this. The underlying formality of the computer still allows for it to be dynamic with respect to its resources. The resources may be finite, but these days memory and disk are so large that their finiteness isn’t a significant factor. So, instead of coding a large number of finite static rules to contain the system, the designers find dynamic ones that bend and shape as the underlying informal systems move about. One of the great failures of software development these days is to ignore this reality most of the time, and rather just blame the inherent mismatch on contrived issues like ‘scope creep’.

To simplify this discussion somewhat, we can combine any underlying informal systems together and take a subset. This somewhat monstrous informal system is the object that we need to mirror with a formal software one. As one could easily guess, this type of combination can easily be contradictory or incomplete. And that is an all too frequent reality when building software systems.

While the relationship between software systems and the informal ones that the software is trying to solve is interesting, these days I am more interested in the general behavioral characteristics of informal systems by themselves. One thing we can do is use the size of an informal system as the metric of the underlying complexity. It isn’t the only way to view complexity, but it does prove useful in understanding how and where the complexity grows.

As an example, we can look at law. Since we’re generalizing, we won’t distinguish between civil and criminal law, nor tie the discussion to any specific country or international organization. Law is basically a set of rules that the people of a society have agreed to follow. There are often a lot of rules, and generally through government action they are always increasing. The laws define which actions are right or wrong but that is really a black and white view, where the world is rather grey. So along with the laws are the precedents from legal judgements that together often define how the law is interpreted and applied. Both of these can also rely on a vast number of legal definitions, that do not necessarily match the more common definitions used by industries or people. To add to this, some laws or interpretations of them have never been fully tested in the courts, so there is some ambiguity as to whether or not they are valid.

Thus the informal system that binds the laws of a country consists of the laws themselves, the history of precedence and some unknown guesses or assumptions about what is, and what is not, legal. And this is constantly being added to with new laws, or new interpretations of the laws. Thus it is perpetually getting larger all of the time.

At least from the outside there doesn’t seem to be any real mechanisms to reduce the size of the system. I’m sure there are some instances of people replacing laws with simpler ones, or laws just getting ignored, but in general this does not seem to be the trend. The system just gets larger every year, there are more lawyers and judges, and the complexity get worse. New interpretations that form precedence get added, but the old ones hang around and can always be dredged up again.

The irony is that at some point the sheer size of the system prevents people from being aware of the full scope of it. That seems to contradict its usefulness in being there as a common set of rules that everyone in a society should obey. Why have rules if most people are not aware of them, or they are not properly enforced?

In a formal system, one could likely find an equivalent system, but with a less complex set of rules and then reduce it to it. In an informal one, while that option may exist, it doesn’t seem to be the path chosen. The system gets larger, it get more convoluted, and any earlier discrepancies instead of getting fixed just get built upon for the later works.

An insider’s perspective on a informal system is generally predicated on familiarity. They understand the rules, or at least a subset of them, and they accept that this is the way it was, and the way it will be. An outsider’s perspective, particularly if they need to dig down into the depths and see all of the ugly warts and bumps that are being ignored by others, is quite different. They can see the twistedness of the informal system for what it is. The gradual accumulation of the rational and irrational, over a long period of time.

For software developers, if they are building systems to handle very specific domain problems, they are the outsiders. That is, they come at it from a distant perspective, and they get to evaluate the rules based not on history growth, but rather a rational logical foundation. They have no choice, they must map the informal system onto a very rigid and precise formal one. Any impedance mismatches between the two systems is both obvious and problematic.

That does give developers a rather unique point of view. We’re not looking at informal systems like law from the vantage point of acceptance. We’re looking at it from the vantage point of complexity, and mapping it to something simpler. Software systems, almost by definition, are going to be simpler and less complex than the informal systems they are mirroring. That’s a product of the sheer amount of resources to build them and keep them running, but also because they generally focus on just a small number of problems contained within the informal system, not the full system itself.

From all of my experience digging to many different informal systems there are a couple of things which really worry me. One is that virtually all of the informal systems I’ve seen contain some type of irrationality. There is always at least one thing there that is just not the product of rational thinking. It just gets built up at the cross roads between other parts of the system. Nothing that would even be constructed deliberately. It’s easy to see these because any domain expert explanations are contrived, messy, hidden or seem to change. That usually points to some underlying irrational component.

The other thing that worries me is that there is always a way for the system to grow, but almost never a way for it to shrink. One can see the consequences of this all over our societies. Everything gets more complicated and more convoluted. People try to justify the growth, but these often are just hollow excuses.

One might consider computerization a simplification method, and sometimes it is. It is not infrequent that implementing a new computer systems means changing the process to some degree to fit the new limits of the code. However development of systems often goes on for decades, and gradually the complexity heads back to its former glory.

Computers in general make it easier to manage complex informal systems. They can do a lot of the grunt labour for people, and they can do it fairly accurately, if they were correctly instructed to do so. We can see this shift as well. Part of the transformation of computers has been the ability to manage informal systems of such complexity that they would have easily collapsed without the aide of a computer. Overall, however, I’m not sure that this is a good thing.

A while back I read in a blog somewhere, that once a complex informal system gets large enough, the only remaining option is for it to collapse. That is, it falls apart suddenly and dramatically. History seems to confirm this view, as we have seen the rise and fall of many a great civilization. At some point the system can no longer increase, it can’t remain as it is and there is no ‘release valve’ for complexity. There are no other options. It basically implodes on itself.

That notion, if true, does not bode well for our current societies. We can see that our complexity has dramatically increased over the last half century, and we can see examples of systematic failure showing up at ever increasing rates, but we seem to have no viable paths of fixing what is wrong. We should be concerned.

If there is some way out of our historic fate, my sense is that we would need to rely on both computers and aggressive simplification to get there. Computers, because they can show us alternatives to our current informal systems, and help us model and control the side-effects of changing what are essentially chaotic systems of n-dimensional variables. And with that initial knowledge, we can start the slow gradual changes needed to simplify our informal systems. The changes have to be slow, just because in general people don’t adapt well and their fragility generates another form of complexity. No doubt it is easier said than achieved, but given the alternative of imploding it doesn’t seem that bad.

There is lots more I could say about formal and informal systems. Defined this way, they provide a general underpinning for a great deal of what happens in our lives. And along with the way we model things internally they help to organize the way we see the epiphenomenon around us. Perhaps, someday I’ll get more of a chance to delve deeper into this area, although all things considered, keeping up with modern life’s requirements is probably too time-consuming and complex these days for me to get another chance :-)