[ Author’s Note: I really don’t know what to say. It was just one of those days. If you don’t feel like reading a broad rambling on loosely connected ideas, please don’t read this! ]
good place to start -- as always -- is with a few definitions. For this
particular discussion it works better if I create them loosely and use
somewhat non-conventional meanings. I’ll try to explain why that helps
as I go along.
first thing I want to define is a ‘system’. For this discussion it is a
set of rules that act on ‘stuff’, where I intend stuff to be nearly as
vague as possible. Stuff could mean static objects, or it could mean
dynamic processes, or anything in between.
can be broken down into two loose categories: formal and informal. A
formal system rigidly constrains the stuff, so that it stays within the
space occupied by the rules. That is, stuff is in the formal system when
it obeys the rules, and it is invalid or outside the system otherwise.
Thus the rules are ‘formal’ and essentially unbreakable.
systems, on the other hand, obey the rules most of the time but there
are plenty of things that can occur within the system that are outside
of the rules. That is, an informal system loosely defines what most of
the rules are, what the boundaries are, but there are always exceptions
and somehow by some ‘stickiness’ these still fall within the system.
put these two definitions in more candid terms: a formal system is
black and white and an informal one is just shades of grey. Both are
systems and have what amounts to deterministic behavior, but one only
approximately follows its rules, while the other is bound by them.
why these definitions? Because we can apply them to many things within
our world. The rigid abstract formalisms of mathematics are obviously
formal systems. The rules are the axioms, the definitions, the lemmas,
the theorems, etc. that are built up on top of the stuff which is the
underlying well-defined mathematical objects. There are many different
branches in mathematics and all of them have a plethora of underlying
formal systems. Formal systems can also be built on top of other formal
systems, by creating relationships between them.
a very simple formal system in mathematics is a set of operators like:
plus, minus, multiply and divide, that works on objects like: the set of
all whole (positive) numbers. What is interesting about this system is
that the subtract operator, when used on a small number and a much
larger one, will produce a new number that is no longer within the whole
numbers. That is 3 - 10 = -7, which is a negative number.
are a number of ways to view this, but again I’ll go a little
unconventional and say that the above formal system has an ‘exit’ point
into a larger formal system. That is, if I had chosen integers, which
include the negatives then there would be no problems with the subtract
operator. Thus we can consider the first formal system as being an
incomplete subset of the second, with respect to subtraction. We can
also consider it to be an incomplete subset of one based on the
continuum of real numbers, with respect to division. This gives us a
large set of structural relationships between the various systems. These
distinctions may seem a little weird, but they will come in useful
contrast to a formal system, an informal one is not so well-defined.
There are rules and there is stuff, but there is also a great deal of
flexibility built in. That is, there are no real ‘exit points’. If
something does not fit within the system, somehow it still seems to
remain there. The types of informal systems I am thinking of include all
different sizes of groups of people. Companies, clubs, governments and
all other organizations are informal systems. They have rules which bind
them together, but do change from time to time. Other examples are
markets, schools, disciplines, professions, soft sciences, etc.
may note that all of the informal systems I’ve listed so far are based
around people. That’s more of a coincidence rather than a deliberate
intent, for there are plenty of systems out there that we have tried to
rigorously formalize but our underlying knowledge is incomplete, or the
basis of the system is just too chaotic. Evolution, physics and weather
are good examples. These too are informal systems, but our underlying
comprehension of the rules is gradually approaching formalization. That
is, rather than an exit point, outliers get feed back into the informal
system to enhance our understanding of it as it approaches, but never
quite reaches, formalization.
this brings up a very important point. At the bottom of everything, as
far a s we know, are particles. And although we don’t know all of the
rules that define their behavior, we are pretty sure that the rule set
is static. Thus there is one underlying formal system that quite
possibly binds everything. That is, nothing can happen unless it is a
physical possibility. On top of this we get ever increasing
‘epiphenomenon’; larger and larger collection of particles that at their
collective levels behave within a set of what are essentially
meta-rules. So we get galaxies, planets, stars, etc. And built on these
we get even more: water, earth, plants, people, etc. And on these we
build up even higher to stuff like buildings, farms, etc. And some
epiphenomenon that are less physical, but still bind together stuff like
organizations, companies, countries, etc.
it would seem that for all of our informal systems, they exists within
the bounds of lower and lower systems that eventually fall back to one
rather low-level formal system.
is rather odd though, is that we are also aware of the multitude of
mathematical formal systems out there, and these seem to exist without
any ties to those based on the many layers of epiphenomenon ones. We do
however, use them to explain the others. That is, physics is a set of
formal systems that are remarkably accurate in explaining the physical
world around us. So in that very sense, mathematics is the
meta-formality that binds together the epiphenomenon with some degree of
precision. Thus it has its roots in our physical world, even if it just
appears as an abstraction based on our intellectual thinking
real argument about the how tangible the formal systems of mathematics
are has to take in consideration computers. For the first time, our
species has manager to instantiate one of our previously abstract-only
formal systems -- Turing machines and/or lambda calculus -- into a
physical reality. Well almost, since computers are still bound by finite
resources and by the occasional hardware failure. Regardless, it is
still a tremendous accomplishment and their existence has been rapidly
transforming our societies ever since. One might argue that how physics
models the world is similar, but we need to consider that that link
between math and reality depends on a person utilizing the models in
their head, in order to act on the knowledge. Thus the formal systems
that underpin physics stay internal to our minds. Computers, on the
other hand, do their computations entirely independently of people, and
thus are a concrete physical manifestations of a formal system. A very
why are these definitions of formal and informal systems useful? One of
the main tasks of software developers is to construct useful formal
systems that solve problems for people working in informal ones. That
is, any software we write must be formal in its construction, but its
usage lies in one or many informal systems.
you are writing a system to track how a company deals with its clients,
the company and the behavior of the clients is entirely informal. Most
things happen according to plan, but lots of exceptions occur and the
nature of the relationship changes as the market expectations change.
But those rules need to be formalized, so they can be coded, so the
computer can execute a deterministic series of instructions with each
interaction between the client and the company.
this angle, you can see the problem right away. The informality of the
base systems leads to difficulties in finding a formal one that both
precisely matches them and remains matching for any prolonged period of
are ways around this. The underlying formality of the computer still
allows for it to be dynamic with respect to its resources. The resources
may be finite, but these days memory and disk are so large that their
finiteness isn’t a significant factor. So, instead of coding a large
number of finite static rules to contain the system, the designers find
dynamic ones that bend and shape as the underlying informal systems move
about. One of the great failures of software development these days is
to ignore this reality most of the time, and rather just blame the
inherent mismatch on contrived issues like ‘scope creep’.
simplify this discussion somewhat, we can combine any underlying
informal systems together and take a subset. This somewhat monstrous
informal system is the object that we need to mirror with a formal
software one. As one could easily guess, this type of combination can
easily be contradictory or incomplete. And that is an all too frequent
reality when building software systems.
the relationship between software systems and the informal ones that
the software is trying to solve is interesting, these days I am more
interested in the general behavioral characteristics of informal systems
by themselves. One thing we can do is use the size of an informal
system as the metric of the underlying complexity. It isn’t the only way
to view complexity, but it does prove useful in understanding how and
where the complexity grows.
an example, we can look at law. Since we’re generalizing, we won’t
distinguish between civil and criminal law, nor tie the discussion to
any specific country or international organization. Law is basically a
set of rules that the people of a society have agreed to follow. There
are often a lot of rules, and generally through government action they
are always increasing. The laws define which actions are right or wrong
but that is really a black and white view, where the world is rather
grey. So along with the laws are the precedents from legal judgements
that together often define how the law is interpreted and applied. Both
of these can also rely on a vast number of legal definitions, that do
not necessarily match the more common definitions used by industries or
people. To add to this, some laws or interpretations of them have never
been fully tested in the courts, so there is some ambiguity as to
whether or not they are valid.
the informal system that binds the laws of a country consists of the
laws themselves, the history of precedence and some unknown guesses or
assumptions about what is, and what is not, legal. And this is
constantly being added to with new laws, or new interpretations of the
laws. Thus it is perpetually getting larger all of the time.
least from the outside there doesn’t seem to be any real mechanisms to
reduce the size of the system. I’m sure there are some instances of
people replacing laws with simpler ones, or laws just getting ignored,
but in general this does not seem to be the trend. The system just gets
larger every year, there are more lawyers and judges, and the complexity
get worse. New interpretations that form precedence get added, but the
old ones hang around and can always be dredged up again.
irony is that at some point the sheer size of the system prevents
people from being aware of the full scope of it. That seems to
contradict its usefulness in being there as a common set of rules that
everyone in a society should obey. Why have rules if most people are not
aware of them, or they are not properly enforced?
a formal system, one could likely find an equivalent system, but with a
less complex set of rules and then reduce it to it. In an informal one,
while that option may exist, it doesn’t seem to be the path chosen. The
system gets larger, it get more convoluted, and any earlier
discrepancies instead of getting fixed just get built upon for the later
insider’s perspective on a informal system is generally predicated on
familiarity. They understand the rules, or at least a subset of them,
and they accept that this is the way it was, and the way it will be. An
outsider’s perspective, particularly if they need to dig down into the
depths and see all of the ugly warts and bumps that are being ignored by
others, is quite different. They can see the twistedness of the
informal system for what it is. The gradual accumulation of the rational
and irrational, over a long period of time.
software developers, if they are building systems to handle very
specific domain problems, they are the outsiders. That is, they come at
it from a distant perspective, and they get to evaluate the rules based
not on history growth, but rather a rational logical foundation. They
have no choice, they must map the informal system onto a very rigid and
precise formal one. Any impedance mismatches between the two systems is
both obvious and problematic.
does give developers a rather unique point of view. We’re not looking
at informal systems like law from the vantage point of acceptance. We’re
looking at it from the vantage point of complexity, and mapping it to
something simpler. Software systems, almost by definition, are going to
be simpler and less complex than the informal systems they are
mirroring. That’s a product of the sheer amount of resources to build
them and keep them running, but also because they generally focus on
just a small number of problems contained within the informal system,
not the full system itself.
all of my experience digging to many different informal systems there
are a couple of things which really worry me. One is that virtually all
of the informal systems I’ve seen contain some type of irrationality.
There is always at least one thing there that is just not the product of
rational thinking. It just gets built up at the cross roads between
other parts of the system. Nothing that would even be constructed
deliberately. It’s easy to see these because any domain expert
explanations are contrived, messy, hidden or seem to change. That
usually points to some underlying irrational component.
other thing that worries me is that there is always a way for the
system to grow, but almost never a way for it to shrink. One can see the
consequences of this all over our societies. Everything gets more
complicated and more convoluted. People try to justify the growth, but
these often are just hollow excuses.
might consider computerization a simplification method, and sometimes
it is. It is not infrequent that implementing a new computer systems
means changing the process to some degree to fit the new limits of the
code. However development of systems often goes on for decades, and
gradually the complexity heads back to its former glory.
in general make it easier to manage complex informal systems. They can
do a lot of the grunt labour for people, and they can do it fairly
accurately, if they were correctly instructed to do so. We can see this
shift as well. Part of the transformation of computers has been the
ability to manage informal systems of such complexity that they would
have easily collapsed without the aide of a computer. Overall, however,
I’m not sure that this is a good thing.
while back I read in a blog somewhere, that once a complex informal
system gets large enough, the only remaining option is for it to
collapse. That is, it falls apart suddenly and dramatically. History
seems to confirm this view, as we have seen the rise and fall of many a
great civilization. At some point the system can no longer increase, it
can’t remain as it is and there is no ‘release valve’ for complexity.
There are no other options. It basically implodes on itself.
notion, if true, does not bode well for our current societies. We can
see that our complexity has dramatically increased over the last half
century, and we can see examples of systematic failure showing up at
ever increasing rates, but we seem to have no viable paths of fixing
what is wrong. We should be concerned.
there is some way out of our historic fate, my sense is that we would
need to rely on both computers and aggressive simplification to get
there. Computers, because they can show us alternatives to our current
informal systems, and help us model and control the side-effects of
changing what are essentially chaotic systems of n-dimensional
variables. And with that initial knowledge, we can start the slow
gradual changes needed to simplify our informal systems. The changes
have to be slow, just because in general people don’t adapt well and
their fragility generates another form of complexity. No doubt it is
easier said than achieved, but given the alternative of imploding it
doesn’t seem that bad.
is lots more I could say about formal and informal systems. Defined
this way, they provide a general underpinning for a great deal of what
happens in our lives. And along with the way we model things internally
they help to organize the way we see the epiphenomenon around us.
Perhaps, someday I’ll get more of a chance to delve deeper into this
area, although all things considered, keeping up with modern life’s
requirements is probably too time-consuming and complex these days for
me to get another chance :-)