An Informal Ramble

[ Author’s Note: I really don’t know what to say. It was just one of those days. If you don’t feel like reading a broad rambling on loosely connected ideas, please don’t read this! ]

A good place to start -- as always -- is with a few definitions. For this particular discussion it works better if I create them loosely and use somewhat non-conventional meanings. I’ll try to explain why that helps as I go along.

The first thing I want to define is a ‘system’. For this discussion it is a set of rules that act on ‘stuff’, where I intend stuff to be nearly as vague as possible. Stuff could mean static objects, or it could mean dynamic processes, or anything in between.

Systems can be broken down into two loose categories: formal and informal. A formal system rigidly constrains the stuff, so that it stays within the space occupied by the rules. That is, stuff is in the formal system when it obeys the rules, and it is invalid or outside the system otherwise. Thus the rules are ‘formal’ and essentially unbreakable.

Informal systems, on the other hand, obey the rules most of the time but there are plenty of things that can occur within the system that are outside of the rules. That is, an informal system loosely defines what most of the rules are, what the boundaries are, but there are always exceptions and somehow by some ‘stickiness’ these still fall within the system.

To put these two definitions in more candid terms: a formal system is black and white and an informal one is just shades of grey. Both are systems and have what amounts to deterministic behavior, but one only approximately follows its rules, while the other is bound by them.

So, why these definitions? Because we can apply them to many things within our world. The rigid abstract formalisms of mathematics are obviously formal systems. The rules are the axioms, the definitions, the lemmas, the theorems, etc. that are built up on top of the stuff which is the underlying well-defined mathematical objects. There are many different branches in mathematics and all of them have a plethora of underlying formal systems. Formal systems can also be built on top of other formal systems, by creating relationships between them.

So a very simple formal system in mathematics is a set of operators like: plus, minus, multiply and divide, that works on objects like: the set of all whole (positive) numbers. What is interesting about this system is that the subtract operator, when used on a small number and a much larger one, will produce a new number that is no longer within the whole numbers. That is 3 - 10 = -7, which is a negative number.

There are a number of ways to view this, but again I’ll go a little unconventional and say that the above formal system has an ‘exit’ point into a larger formal system. That is, if I had chosen integers, which include the negatives then there would be no problems with the subtract operator. Thus we can consider the first formal system as being an incomplete subset of the second, with respect to subtraction. We can also consider it to be an incomplete subset of one based on the continuum of real numbers, with respect to division. This gives us a large set of structural relationships between the various systems. These distinctions may seem a little weird, but they will come in useful later.

In contrast to a formal system, an informal one is not so well-defined. There are rules and there is stuff, but there is also a great deal of flexibility built in. That is, there are no real ‘exit points’. If something does not fit within the system, somehow it still seems to remain there. The types of informal systems I am thinking of include all different sizes of groups of people. Companies, clubs, governments and all other organizations are informal systems. They have rules which bind them together, but do change from time to time. Other examples are markets, schools, disciplines, professions, soft sciences, etc.

You may note that all of the informal systems I’ve listed so far are based around people. That’s more of a coincidence rather than a deliberate intent, for there are plenty of systems out there that we have tried to rigorously formalize but our underlying knowledge is incomplete, or the basis of the system is just too chaotic. Evolution, physics and weather are good examples. These too are informal systems, but our underlying comprehension of the rules is gradually approaching formalization. That is, rather than an exit point, outliers get feed back into the informal system to enhance our understanding of it as it approaches, but never quite reaches, formalization.

And this brings up a very important point. At the bottom of everything, as far a s we know, are particles. And although we don’t know all of the rules that define their behavior, we are pretty sure that the rule set is static. Thus there is one underlying formal system that quite possibly binds everything. That is, nothing can happen unless it is a physical possibility. On top of this we get ever increasing ‘epiphenomenon’; larger and larger collection of particles that at their collective levels behave within a set of what are essentially meta-rules. So we get galaxies, planets, stars, etc. And built on these we get even more: water, earth, plants, people, etc. And on these we build up even higher to stuff like buildings, farms, etc. And some epiphenomenon that are less physical, but still bind together stuff like organizations, companies, countries, etc.

So it would seem that for all of our informal systems, they exists within the bounds of lower and lower systems that eventually fall back to one rather low-level formal system.

What is rather odd though, is that we are also aware of the multitude of mathematical formal systems out there, and these seem to exist without any ties to those based on the many layers of epiphenomenon ones. We do however, use them to explain the others. That is, physics is a set of formal systems that are remarkably accurate in explaining the physical world around us. So in that very sense, mathematics is the meta-formality that binds together the epiphenomenon with some degree of precision. Thus it has its roots in our physical world, even if it just appears as an abstraction based on our intellectual thinking capabilities.

Any real argument about the how tangible the formal systems of mathematics are has to take in consideration computers. For the first time, our species has manager to instantiate one of our previously abstract-only formal systems -- Turing machines and/or lambda calculus -- into a physical reality. Well almost, since computers are still bound by finite resources and by the occasional hardware failure. Regardless, it is still a tremendous accomplishment and their existence has been rapidly transforming our societies ever since. One might argue that how physics models the world is similar, but we need to consider that that link between math and reality depends on a person utilizing the models in their head, in order to act on the knowledge. Thus the formal systems that underpin physics stay internal to our minds. Computers, on the other hand, do their computations entirely independently of people, and thus are a concrete physical manifestations of a formal system. A very different ballgame.

So why are these definitions of formal and informal systems useful? One of the main tasks of software developers is to construct useful formal systems that solve problems for people working in informal ones. That is, any software we write must be formal in its construction, but its usage lies in one or many informal systems.

If you are writing a system to track how a company deals with its clients, the company and the behavior of the clients is entirely informal. Most things happen according to plan, but lots of exceptions occur and the nature of the relationship changes as the market expectations change. But those rules need to be formalized, so they can be coded, so the computer can execute a deterministic series of instructions with each interaction between the client and the company.

From this angle, you can see the problem right away. The informality of the base systems leads to difficulties in finding a formal one that both precisely matches them and remains matching for any prolonged period of time.

There are ways around this. The underlying formality of the computer still allows for it to be dynamic with respect to its resources. The resources may be finite, but these days memory and disk are so large that their finiteness isn’t a significant factor. So, instead of coding a large number of finite static rules to contain the system, the designers find dynamic ones that bend and shape as the underlying informal systems move about. One of the great failures of software development these days is to ignore this reality most of the time, and rather just blame the inherent mismatch on contrived issues like ‘scope creep’.

To simplify this discussion somewhat, we can combine any underlying informal systems together and take a subset. This somewhat monstrous informal system is the object that we need to mirror with a formal software one. As one could easily guess, this type of combination can easily be contradictory or incomplete. And that is an all too frequent reality when building software systems.

While the relationship between software systems and the informal ones that the software is trying to solve is interesting, these days I am more interested in the general behavioral characteristics of informal systems by themselves. One thing we can do is use the size of an informal system as the metric of the underlying complexity. It isn’t the only way to view complexity, but it does prove useful in understanding how and where the complexity grows.

As an example, we can look at law. Since we’re generalizing, we won’t distinguish between civil and criminal law, nor tie the discussion to any specific country or international organization. Law is basically a set of rules that the people of a society have agreed to follow. There are often a lot of rules, and generally through government action they are always increasing. The laws define which actions are right or wrong but that is really a black and white view, where the world is rather grey. So along with the laws are the precedents from legal judgements that together often define how the law is interpreted and applied. Both of these can also rely on a vast number of legal definitions, that do not necessarily match the more common definitions used by industries or people. To add to this, some laws or interpretations of them have never been fully tested in the courts, so there is some ambiguity as to whether or not they are valid.

Thus the informal system that binds the laws of a country consists of the laws themselves, the history of precedence and some unknown guesses or assumptions about what is, and what is not, legal. And this is constantly being added to with new laws, or new interpretations of the laws. Thus it is perpetually getting larger all of the time.

At least from the outside there doesn’t seem to be any real mechanisms to reduce the size of the system. I’m sure there are some instances of people replacing laws with simpler ones, or laws just getting ignored, but in general this does not seem to be the trend. The system just gets larger every year, there are more lawyers and judges, and the complexity get worse. New interpretations that form precedence get added, but the old ones hang around and can always be dredged up again.

The irony is that at some point the sheer size of the system prevents people from being aware of the full scope of it. That seems to contradict its usefulness in being there as a common set of rules that everyone in a society should obey. Why have rules if most people are not aware of them, or they are not properly enforced?

In a formal system, one could likely find an equivalent system, but with a less complex set of rules and then reduce it to it. In an informal one, while that option may exist, it doesn’t seem to be the path chosen. The system gets larger, it get more convoluted, and any earlier discrepancies instead of getting fixed just get built upon for the later works.

An insider’s perspective on a informal system is generally predicated on familiarity. They understand the rules, or at least a subset of them, and they accept that this is the way it was, and the way it will be. An outsider’s perspective, particularly if they need to dig down into the depths and see all of the ugly warts and bumps that are being ignored by others, is quite different. They can see the twistedness of the informal system for what it is. The gradual accumulation of the rational and irrational, over a long period of time.

For software developers, if they are building systems to handle very specific domain problems, they are the outsiders. That is, they come at it from a distant perspective, and they get to evaluate the rules based not on history growth, but rather a rational logical foundation. They have no choice, they must map the informal system onto a very rigid and precise formal one. Any impedance mismatches between the two systems is both obvious and problematic.

That does give developers a rather unique point of view. We’re not looking at informal systems like law from the vantage point of acceptance. We’re looking at it from the vantage point of complexity, and mapping it to something simpler. Software systems, almost by definition, are going to be simpler and less complex than the informal systems they are mirroring. That’s a product of the sheer amount of resources to build them and keep them running, but also because they generally focus on just a small number of problems contained within the informal system, not the full system itself.

From all of my experience digging to many different informal systems there are a couple of things which really worry me. One is that virtually all of the informal systems I’ve seen contain some type of irrationality. There is always at least one thing there that is just not the product of rational thinking. It just gets built up at the cross roads between other parts of the system. Nothing that would even be constructed deliberately. It’s easy to see these because any domain expert explanations are contrived, messy, hidden or seem to change. That usually points to some underlying irrational component.

The other thing that worries me is that there is always a way for the system to grow, but almost never a way for it to shrink. One can see the consequences of this all over our societies. Everything gets more complicated and more convoluted. People try to justify the growth, but these often are just hollow excuses.

One might consider computerization a simplification method, and sometimes it is. It is not infrequent that implementing a new computer systems means changing the process to some degree to fit the new limits of the code. However development of systems often goes on for decades, and gradually the complexity heads back to its former glory.

Computers in general make it easier to manage complex informal systems. They can do a lot of the grunt labour for people, and they can do it fairly accurately, if they were correctly instructed to do so. We can see this shift as well. Part of the transformation of computers has been the ability to manage informal systems of such complexity that they would have easily collapsed without the aide of a computer. Overall, however, I’m not sure that this is a good thing.

A while back I read in a blog somewhere, that once a complex informal system gets large enough, the only remaining option is for it to collapse. That is, it falls apart suddenly and dramatically. History seems to confirm this view, as we have seen the rise and fall of many a great civilization. At some point the system can no longer increase, it can’t remain as it is and there is no ‘release valve’ for complexity. There are no other options. It basically implodes on itself.

That notion, if true, does not bode well for our current societies. We can see that our complexity has dramatically increased over the last half century, and we can see examples of systematic failure showing up at ever increasing rates, but we seem to have no viable paths of fixing what is wrong. We should be concerned.

If there is some way out of our historic fate, my sense is that we would need to rely on both computers and aggressive simplification to get there. Computers, because they can show us alternatives to our current informal systems, and help us model and control the side-effects of changing what are essentially chaotic systems of n-dimensional variables. And with that initial knowledge, we can start the slow gradual changes needed to simplify our informal systems. The changes have to be slow, just because in general people don’t adapt well and their fragility generates another form of complexity. No doubt it is easier said than achieved, but given the alternative of imploding it doesn’t seem that bad.

There is lots more I could say about formal and informal systems. Defined this way, they provide a general underpinning for a great deal of what happens in our lives. And along with the way we model things internally they help to organize the way we see the epiphenomenon around us. Perhaps, someday I’ll get more of a chance to delve deeper into this area, although all things considered, keeping up with modern life’s requirements is probably too time-consuming and complex these days for me to get another chance :-) 

The Engineering of Software

The last thing I ever wanted to be -- twenty-six years ago -- was an engineer. When I started university I had heard that 70% of all engineers hate their jobs and that they were often bored at work. Boredom sounded horrible.

What I wanted, was to master the black art of coding. To follow in the foot steps of those early magicians I had seen in the computer magazines. The ones that crafted weird cryptic concoctions to do really neat things to their machines. I wanted to be a hacker.

It’s funny how time and experience temper one’s thoughts; how your actions change once you really understand the consequences; how much you grow once you see the larger picture.

Development shops span the range between the cowboys, who are just hacking at the code furiously, and the bureaucrats, that are so ground down in organizational paperwork that little actually gets accomplished.

In my younger days the cowboys where fun. It’s a dynamic environment. Things are happening. And they are happening fast. Well, OK, there are a lot of needless problems cropping up, but hey, that just adds to the excitement, doesn’t it? You could leave work feeling like you got something accomplished.

Well, that lasts for a while. Until you start to get caught in the same hollow accomplishments, over and over again. That is, you’ve fixed the same bug before, wrote the same code before or stuck a band aid on the same gaping wound before. Doing it once or twice is fun. Doing it repetitively, while giving up your life to endless overtime, starts to become depressing after a while. Then suddenly one day you find yourself looking down at the growing pile of band-aids before you and you start thinking “isn’t this all just a waste of my time and my abilities?”

Early on I moved away from the cowboys. They were driving me nuts. I landed deep in the heart of a reasonable engineering culture. Some people on the outside might have confused that development process with a classic waterfall one, but it wasn’t really. Although there was a significant design process up front, the last of the details were always worked out in the various iterations. It was actually a healthy mix. Some thinking, some paperwork, then a hard run at the code. Some testing, a release and then back to thinking again.

Later when I set up my own shop, it naturally started as chaos. Startups are sensitive to cash-flow problems so sometimes the only way to deal with that is an ugly short-term hack. But as time progressed -- every time I got the chance -- I applied long-term fixes. These gradually paid off and when I finally moved on, the place was humming along smoothly releasing a steady stream of mostly high quality code and the occasional patch within a day or two of finding a bug. Each development cycle was a bit easier than its predecessor. Everything was tracked. We had managed to built a big application. One that was far larger, and far more sophisticated than you would have expected from such a tiny shop. I learned a lot in those years.

I found that it isn’t all that complicated. If you boil it down again and again, software development keeps coming back to its roots in engineering. Not in the classic sense that I was afraid of when I was younger -- the large, slow, boring, mindless process -- but rather in the sense that as the work progresses, developers like myself needed to be hacking less, guessing less, and understanding more. The project may start with a lot of unknowns, but at some point it always gets down to routine engineering. That is, we should be doing the ‘right’ things that work because we know that they work, and fixing the little problems -- permanently -- before they become big serious ones.

My earlier fears were misplaced. Engineering itself is not boring. I’m not even sure where I heard that statistic about unhappiness. Engineering, particularly when it comes to something like software, is simply knowing what works and applying it correctly so that the project is successful. It isn’t hack and hope, it isn’t wild guesses, and it isn’t a spin job to cover up the fact that the system is horribly broken. No, quite simply, it is the path to dreaming up cool solutions and then implementing them as cool solutions.

Yes, there are boring times. There is a lot of digging to see what is already out there. No sense reinventing the wheel especially if it’s already decades old. Note that that doesn’t mean you can’t recode the wheel -- making a better one -- but rather that you don’t try to rethink the wheel, that work is already done and accessible.

There is sometimes paperwork. But if one focuses on documentation that’s useful, rather than on the documentation a management consultant would want, then it’s not so bad either. It helps to refine ideas and highlight missing corner-cases long before they become too serious to fix. A little design goes a long way and any organizational chaos, at any level, is always going to grow into a serious problem someday. What you’re missing, or not thinking about, often becomes really obvious when you try to write it down. It also helps you to understand the vocabulary of the problem and to empathize with your users (if you can’t easily explain it in a document then the users aren’t going to be able to understand it either).

There is a lot of thinking. And ironically, although programmers preach heavily about their need for freedom, many choose to turn off their minds when they code, resorting to as much brute force as possible. But at the end of the day, code is just the by-product of thinking about the solution, so it is only as good as the thinking that went into it. To build sophisticated things you have to have a deep understanding of the problem and you have to spend a lot of time working through it. Thoughtless code is dangerous code because of it lacks intelligence. It just causes stupid problems.

And of course, if you’ve really set a firm direction and you know where you are going with the development then the rest of it is just hard work. Little issues have to be sorted out, details need gathering, but the main flow of the development can be worked out in advance. The project may drift but either the long-term perspective still holds, or it is rethought again to insure that it is still valid.

I remember reading decades ago in Scientific American about how the pharmaceutical industry moved from art, to science and then on to engineering. The pills we pop these days are highly likely to contain exactly what they say they will. That wasn’t always the case. It took them a long time to go from alchemy to engineering. The article said that software also started as an art, and has some scientific basis, but it will eventually follow the same path to engineering. I remember the article because I figured I’d bail when that happened. Sounded boring. But these days, as I am surrounded by more software with increasing complexity and decreasing quality I realize that what I want most right now is to get a whole lot more of the engineering mindset into the software development industry. It is time, I think, that we started taking our profession seriously.

There is still room out there for the hackers and the cowboys. Sometimes a band-aid is the best choice, or the solution is so small it doesn’t matter how much of a mess it is. For quick one-offs I think hacking  will always be common, but when I go out and rely on someone else’s code, on their thinking and on their intelligence, I would like to know that they weren’t just mindlessly flailing at the keyboard or that they skipped their homework and just tossed it together. It makes a difference. I don’t want to build on a house of cards. I don’t want to find out later that I’m dependent on somebody who didn’t really ‘get it’ at the end of the day. I don’t want that degree of instability in what I am doing. I can’t build great software on top of a mess; dealing with other people’s mistakes eats through my resources like a knife cuts butter.

So these days I am all in favor of us seriously moving into engineering. Not doing so is a major obstacle that has prevented software from living up to its true potential. I think we can do it, and do it in a way that it is not boring and it isn’t just paperwork. We can still build quickly, but we can do so from a place of understanding, not one of panic. It’s about time that software development grew up and became a real profession, one that is taken seriously. Even in our deplorable state we are changing the world, so can you imagine what we could achieve if our collective works didn’t suck?

Getting to the Truth

One of the most enjoyable things about building software is that it requires the developers to dig around in other domains. If you’re writing financial software, you need to know how the industry works, understand the terminology, and all of the complex details underneath. If you build something for the printing industry, you need to grok their equipment and business models. Everything you write depends on knowledge from elsewhere.

For a naturally curious person like myself, this is a huge benefit. Basically a license to try and learn as much about the world as I can.

Every domain I’ve ever worked in has had a fascinating array of history and depth. You can’t actually solve any of their problems with software until you really understand them; until you get down to what makes them tick. Get to the roots. Get to the truth.

Software is very rigid and to automate any processes they need to be fully understood. To collect valid data you need to understand its structure, quality and volume. To make the interfaces usable you need to understand the environmental issues and the people. If you don’t get to the truth, the software either won’t work or it will cause more problems than it fixes. If your goal is to actually solve something, the only place you can do that is in reality.

But getting to the truth is a funny business. There are all sorts of people out there that will stand in your way. Sometimes it’s because they have something to hide, sometimes it because they don’t want to think about it. Sometimes they just don’t care or figure it isn’t necessary. The causes very.

Lately I’ve come to think of these people obstructing the work as ‘confused’. They are confused about what’s happening below, or above, or why you’re trying to find out, or even what value the software will bring to their lives. The reasons don’t matter, they are just confused.

Building a system on confusion isn’t any better than building an apartment building on a foundation make of foam. It’s obvious that it’s not going to work, even if it looks promising for a moment. If you build on fantasies or delusions or ignore hidden secrets, the result may run, but it’ll quickly drift away from being useful. It doesn’t matter how much data you collect if the data is scrambled or incorrect. And if you ignore the people who ultimately use the work, then they won’t use it or they’ll use it badly, causing massive support headaches. A forced, but miserable user-base can disruptive.

Software has the power to transform, but only if the embedded intelligence within it is correct, and that only occurs if the programmers were exposed to the truth. No truth, then there is no intelligence, and so the results are just burning through wasted resources. This is the fundamental rule that binds the code to something useful in the real world. We’ve known this for a long time, referring to issues like ‘ivory towers’ when talking about programmers who write systems far away from their users.

But this can also happen when the business or management players seek to block access to the truth. And it can also come about as a failure to properly complete reasonable ‘business analysis’, or because the domain experts are confused or lack enough experience. There are a multitude of different ways that confusion can enter into the process and obscure the truth. Distort reality. And without enough truth there is no understanding, which always prevents success.