One of my favorite parts about building software is that I am free to indulge my endless curiosity. I prefer to dig deeply into the domains I’ve tackled so that I can tailor my solutions to actual problems, not just the high-level perception of them. The world is wonderfully complex place and it can be fascinating to delve into the details and underlying history.
really solve a person’s problem with some code running on a computer
you have to understand the root causes. A 10,000 foot view is just too
high to grok the details, which often become counter-intuitive as you
descend. As I've switch from domain to domain, I've obviously found huge
differences between the issues but I’ve also found many underlying
this post I thought I'd pull back a little and discuss one of these
larger veins running through much of my work. I’ll pick a very abstract
example, but only because I prefer not to antagonize both my past and
can start with the distinction between formal and informal systems.
Formal systems are such because the underlying rules are rigid. All
things must follow the rules (or they are invalid and no longer within
the system). A few examples are mathematics and running software
systems on the other hand often involve people, although not always.
The rules are mostly followed, but there are always exceptions. Informal
systems are not rigorously constrained. There are a huge number of
examples of informal systems, but any sort of social interaction --
companies, governments, etc. -- form the basis of many of them.
Basically all you need is some 'objects' (entities, nouns, etc) and a
set of rules that are applied to them.
an overly simple example I keep coming back to cars driving on a
highway. The underlying road supports a number of vehicles that are
attempting to get somewhere, each with a unique destination. There are
mandated constraints such as following the speed limit and staying in
particular lanes. Many highways these days are challenged with a lot of
problems: congestion, accidents, speeding etc. They are fairly simple
systems, but their behavior is fairly complex.
it doesn’t really matter, I’ll focus my discussion loosely around the
highways in Ontario, Canada that I drive regularly. Please don’t get too
lost in the specifics, I’m only using these examples to illustrate the
larger patterns that are common amongst similar and very different
informal systems. Cars, rules and roads just help as reference points,
key constraint when driving on the highway is a speed limit. The rules
and limits on speeding were set long ago, most likely when cars were
considerably less sophisticated. So for instance a speed limit of 100
km/h was probably created long ago as a protective measure to decrease
the likelihood of serious accidents. When highways were poorly paved and
cars crude, no doubt these rules made a great deal of sense and
contributed significantly to the safety of the roadways. These days, for
a well-maintained highway, modern cars which handled better are quite
capable of navigating the same terrain at a significantly higher speeds.
That is, 140 km/h for many drivers in a modern cars is well within
their ability to react correctly to hazards, road conditions and other
vehicles. Because of this, in Ontario at least, 'speeding' is epidemic.
Everybody exceeds what seems like the relatively slow limit of 100 km/h.
response to people increasingly speeding was the creation of traffic
police whose role is to enforce the rules. We can see this as a
subsystem that sprang into life because of a degeneration of drivers in
obeying rules like the speed limit (although there are probably many
other causes). This new subsystem quickly figured out that ticketing
people, beside enforcing the rules, was extremely profitable. And even
more profitable if they put less effort into the overall ‘general’
enforcement but instead choose to hang out at inconvenient locations
where people tend to speed naturally. Thus ‘speed traps’ sprang up from
perhaps rather noble goals, but financial incentives quickly had
considerable influence. Once create the local drivers quickly figured
out where they were located, so they shifted their behavior to speed
only in areas that they knew were safe.
this informal system rule for speed limits sprang up, degenerated, was
buttressed with a new subsystem of ticketing but that too degenerated.
The key point here is that when we are analysing the creation of speed
traps we need to go beyond just the obvious cause, that more people were
violating the speed limit. Getting to the 'root' causes is extremely
important in analysing informal systems. It is easy to say that speed
limits and traps exist because people are speeding more often, but
that's not really capturing the whole picture. Rather the deeper root
cause is that modern cars started allowing people to drive at higher
speeds. They handle better, have bigger engines and can accelerate
rapidly. The root cause for speeding is an advance in technology and
that technology was getting applied by a large number of drivers to
violate a pre-existing set of rules, which is a downstream consequence.
particularly true of informal systems is that their interactions are
extremely complex. To fix the speeding problem by just creating traffic
police is not enough, and because it didn't really fix the problem the
traffic enforcement directives gravitated towards finding their own
usefulness (generating money). Thus adding speeding tickets to the
overall system did not really improve it in any significant way. Some
people probably speed less, but most people are likely to just be more
picky about where they are speeding. At various times and locations
speeding is reduced, but many drivers are probably over-compensating now
for the added inconvenience. People aren't going to stop speeding for
instance, and their expectation for travel time isn’t going to change.
solution like raising the speed limit wouldn't fix the issue either.
The newer cars may allow good drivers to go faster, but highways contain
more than just good drivers. They contain many poor ones, without the
ability to safely handle higher speeds. It's perhaps this constraint
that keeps the authorities from changing the rules. A faster highway
might be significantly more dangerous if there are enough drivers on the
road that can't react fast enough. So the average or less-skilled
driver keeps the status quo intact, but as in the case of Toronto people
simply moved forward and start creating their own rules of thumb for
how fast to drive. Many in TO believe for example that +10 km/h for the
city and +20 km/h for the highway are acceptable. This is so common that
in the absence of any traffic police nearby, this is frequently the
pace of most vehicles. Thus a breakdown in the informal system causes a
creation of another even less formal system built on top.
systems based on people always morph as time progresses. Some systems
such as the highways find their own equilibriums by spawning off new
subsystems. Some systems fluctuate, but generally drive themselves
downward. We know how to make things more complex, but we seem to lack
in the ability to reduce complexity. A degenerating informal system can
often acquire enough unwarranted artificial complexity that its only
path forward is to explode. The issues simply can't be fixed anymore.
idea of speeding tickets was no doubt someone’s best effort to deal
with the speeding problem. But as is true for all complex informal
systems, the whole systemic complexity exceeds the ability of most, if
not all, people to visualize and correctly modify it. Speeding tickets
created speed traps. Speed traps are a rather grey aberration of the
laws of highways, since they aren't helping to keep locals from
speeding, but rather they are preying on unsuspecting strangers. This
causes a hierarchy of drivers, creating a division between 'us' and
'them'. In that sense it is morally questionable, since a civilized
society these days demands that the rules are applied equally to all
people, not just some. A speed trap functions because it is based on
ignorance. Non-local drivers following the normal rules of thumbs are
caught simply because they strayed too far from their own locale. Stated
that way, the enforcement part of society is actually preying on people
for revenue, not to uphold the honor and virtue of the rules.
technology got traffic into trouble, it is likely that it will also get
it out of it. Automatic driving may allow cars to chain together into
larger loosely affiliated trains headed into common directions. If that
comes to pass cars will scout for a suitable train, then join up. As
that becomes more common, the efficiencies of the highway -- which are
poor now and getting worse -- will start to improve creating less
traffic congestion. These trains of cars will follow limits set by
computers, so speeding tickets and speed traps will become a thing of
the past. Of course some people will continue to 'manually' drive, but
official enforcement of any remaining rules will likely shift to some
other basis, like asserting that speeds of 140 km/h are 'reckless'
rather than 'speeding'. The rule remains, but gets generalized to
include a wider collection of behaviors.
systems are everywhere. They form the basis of all human organizations
and they fracture into an endless variety of associated subsystems.
These days computers have made it possible to quickly introduce new
poorly thought-out rules and enforce them, so the underlying complexity
has exploded. Most large companies, for instance, are so outrageously
complicated that no single human could hope to know or understand the
basis behind even a fraction of the rules.Thanks to badly designed
software and archaic historic rules many of these are in various stages
of systemic breakdown, with rules degenerating and inefficient
subsystems spewing off everywhere to replace high-level but hopeless
directives. Computers have the capacity to solve problems for people,
but their dark side is that they can allow for massive increases in
artificial complex that can quickly get gouged into the status quo.
Ambitious, but misguided people tunnel onto little aspects of the
overall problem and seek change, but once the system has gone beyond a
human's ability to comprehend, most such changes ultimately make the
problems worse not better.
back to traffic, if someone really wanted to stop speeding and
installed a massive number of high-tech cameras to cover every inch of
the road, they might think that they'd be able to stop anyone, anywhere,
from speeding. But sorting through that footage would be daunting, and
just creating the infrastructure to fine and collect all of the tickets
would be massive. So, ultimately there would still be blind spots, and
over time these would become known. People, hoping to reduce their
travel time would fly recklessly through the blind spots because they
were forced to slow down once in the spotlight. Then some areas would
become safer for driving, but some, well... they'd just push the envelop
for danger until they too caused various spin offs. To fix complex
system breakdowns the one thing that definitely doesn't work is to
narrow the focus down to something believed to be 'tangible'. That type
of tunnel vision only begs for disaster.