An Informal Ramble

[ Author’s Note: I really don’t know what to say. It was just one of those days. If you don’t feel like reading a broad rambling on loosely connected ideas, please don’t read this! ]

A good place to start -- as always -- is with a few definitions. For this particular discussion it works better if I create them loosely and use somewhat non-conventional meanings. I’ll try to explain why that helps as I go along.

The first thing I want to define is a ‘system’. For this discussion it is a set of rules that act on ‘stuff’, where I intend stuff to be nearly as vague as possible. Stuff could mean static objects, or it could mean dynamic processes, or anything in between.

Systems can be broken down into two loose categories: formal and informal. A formal system rigidly constrains the stuff, so that it stays within the space occupied by the rules. That is, stuff is in the formal system when it obeys the rules, and it is invalid or outside the system otherwise. Thus the rules are ‘formal’ and essentially unbreakable.

Informal systems, on the other hand, obey the rules most of the time but there are plenty of things that can occur within the system that are outside of the rules. That is, an informal system loosely defines what most of the rules are, what the boundaries are, but there are always exceptions and somehow by some ‘stickiness’ these still fall within the system.

To put these two definitions in more candid terms: a formal system is black and white and an informal one is just shades of grey. Both are systems and have what amounts to deterministic behavior, but one only approximately follows its rules, while the other is bound by them.

So, why these definitions? Because we can apply them to many things within our world. The rigid abstract formalisms of mathematics are obviously formal systems. The rules are the axioms, the definitions, the lemmas, the theorems, etc. that are built up on top of the stuff which is the underlying well-defined mathematical objects. There are many different branches in mathematics and all of them have a plethora of underlying formal systems. Formal systems can also be built on top of other formal systems, by creating relationships between them.

So a very simple formal system in mathematics is a set of operators like: plus, minus, multiply and divide, that works on objects like: the set of all whole (positive) numbers. What is interesting about this system is that the subtract operator, when used on a small number and a much larger one, will produce a new number that is no longer within the whole numbers. That is 3 - 10 = -7, which is a negative number.

There are a number of ways to view this, but again I’ll go a little unconventional and say that the above formal system has an ‘exit’ point into a larger formal system. That is, if I had chosen integers, which include the negatives then there would be no problems with the subtract operator. Thus we can consider the first formal system as being an incomplete subset of the second, with respect to subtraction. We can also consider it to be an incomplete subset of one based on the continuum of real numbers, with respect to division. This gives us a large set of structural relationships between the various systems. These distinctions may seem a little weird, but they will come in useful later.

In contrast to a formal system, an informal one is not so well-defined. There are rules and there is stuff, but there is also a great deal of flexibility built in. That is, there are no real ‘exit points’. If something does not fit within the system, somehow it still seems to remain there. The types of informal systems I am thinking of include all different sizes of groups of people. Companies, clubs, governments and all other organizations are informal systems. They have rules which bind them together, but do change from time to time. Other examples are markets, schools, disciplines, professions, soft sciences, etc.

You may note that all of the informal systems I’ve listed so far are based around people. That’s more of a coincidence rather than a deliberate intent, for there are plenty of systems out there that we have tried to rigorously formalize but our underlying knowledge is incomplete, or the basis of the system is just too chaotic. Evolution, physics and weather are good examples. These too are informal systems, but our underlying comprehension of the rules is gradually approaching formalization. That is, rather than an exit point, outliers get feed back into the informal system to enhance our understanding of it as it approaches, but never quite reaches, formalization.

And this brings up a very important point. At the bottom of everything, as far a s we know, are particles. And although we don’t know all of the rules that define their behavior, we are pretty sure that the rule set is static. Thus there is one underlying formal system that quite possibly binds everything. That is, nothing can happen unless it is a physical possibility. On top of this we get ever increasing ‘epiphenomenon’; larger and larger collection of particles that at their collective levels behave within a set of what are essentially meta-rules. So we get galaxies, planets, stars, etc. And built on these we get even more: water, earth, plants, people, etc. And on these we build up even higher to stuff like buildings, farms, etc. And some epiphenomenon that are less physical, but still bind together stuff like organizations, companies, countries, etc.

So it would seem that for all of our informal systems, they exists within the bounds of lower and lower systems that eventually fall back to one rather low-level formal system.

What is rather odd though, is that we are also aware of the multitude of mathematical formal systems out there, and these seem to exist without any ties to those based on the many layers of epiphenomenon ones. We do however, use them to explain the others. That is, physics is a set of formal systems that are remarkably accurate in explaining the physical world around us. So in that very sense, mathematics is the meta-formality that binds together the epiphenomenon with some degree of precision. Thus it has its roots in our physical world, even if it just appears as an abstraction based on our intellectual thinking capabilities.

Any real argument about the how tangible the formal systems of mathematics are has to take in consideration computers. For the first time, our species has manager to instantiate one of our previously abstract-only formal systems -- Turing machines and/or lambda calculus -- into a physical reality. Well almost, since computers are still bound by finite resources and by the occasional hardware failure. Regardless, it is still a tremendous accomplishment and their existence has been rapidly transforming our societies ever since. One might argue that how physics models the world is similar, but we need to consider that that link between math and reality depends on a person utilizing the models in their head, in order to act on the knowledge. Thus the formal systems that underpin physics stay internal to our minds. Computers, on the other hand, do their computations entirely independently of people, and thus are a concrete physical manifestations of a formal system. A very different ballgame.

So why are these definitions of formal and informal systems useful? One of the main tasks of software developers is to construct useful formal systems that solve problems for people working in informal ones. That is, any software we write must be formal in its construction, but its usage lies in one or many informal systems.

If you are writing a system to track how a company deals with its clients, the company and the behavior of the clients is entirely informal. Most things happen according to plan, but lots of exceptions occur and the nature of the relationship changes as the market expectations change. But those rules need to be formalized, so they can be coded, so the computer can execute a deterministic series of instructions with each interaction between the client and the company.

From this angle, you can see the problem right away. The informality of the base systems leads to difficulties in finding a formal one that both precisely matches them and remains matching for any prolonged period of time.

There are ways around this. The underlying formality of the computer still allows for it to be dynamic with respect to its resources. The resources may be finite, but these days memory and disk are so large that their finiteness isn’t a significant factor. So, instead of coding a large number of finite static rules to contain the system, the designers find dynamic ones that bend and shape as the underlying informal systems move about. One of the great failures of software development these days is to ignore this reality most of the time, and rather just blame the inherent mismatch on contrived issues like ‘scope creep’.

To simplify this discussion somewhat, we can combine any underlying informal systems together and take a subset. This somewhat monstrous informal system is the object that we need to mirror with a formal software one. As one could easily guess, this type of combination can easily be contradictory or incomplete. And that is an all too frequent reality when building software systems.

While the relationship between software systems and the informal ones that the software is trying to solve is interesting, these days I am more interested in the general behavioral characteristics of informal systems by themselves. One thing we can do is use the size of an informal system as the metric of the underlying complexity. It isn’t the only way to view complexity, but it does prove useful in understanding how and where the complexity grows.

As an example, we can look at law. Since we’re generalizing, we won’t distinguish between civil and criminal law, nor tie the discussion to any specific country or international organization. Law is basically a set of rules that the people of a society have agreed to follow. There are often a lot of rules, and generally through government action they are always increasing. The laws define which actions are right or wrong but that is really a black and white view, where the world is rather grey. So along with the laws are the precedents from legal judgements that together often define how the law is interpreted and applied. Both of these can also rely on a vast number of legal definitions, that do not necessarily match the more common definitions used by industries or people. To add to this, some laws or interpretations of them have never been fully tested in the courts, so there is some ambiguity as to whether or not they are valid.

Thus the informal system that binds the laws of a country consists of the laws themselves, the history of precedence and some unknown guesses or assumptions about what is, and what is not, legal. And this is constantly being added to with new laws, or new interpretations of the laws. Thus it is perpetually getting larger all of the time.

At least from the outside there doesn’t seem to be any real mechanisms to reduce the size of the system. I’m sure there are some instances of people replacing laws with simpler ones, or laws just getting ignored, but in general this does not seem to be the trend. The system just gets larger every year, there are more lawyers and judges, and the complexity get worse. New interpretations that form precedence get added, but the old ones hang around and can always be dredged up again.

The irony is that at some point the sheer size of the system prevents people from being aware of the full scope of it. That seems to contradict its usefulness in being there as a common set of rules that everyone in a society should obey. Why have rules if most people are not aware of them, or they are not properly enforced?

In a formal system, one could likely find an equivalent system, but with a less complex set of rules and then reduce it to it. In an informal one, while that option may exist, it doesn’t seem to be the path chosen. The system gets larger, it get more convoluted, and any earlier discrepancies instead of getting fixed just get built upon for the later works.

An insider’s perspective on a informal system is generally predicated on familiarity. They understand the rules, or at least a subset of them, and they accept that this is the way it was, and the way it will be. An outsider’s perspective, particularly if they need to dig down into the depths and see all of the ugly warts and bumps that are being ignored by others, is quite different. They can see the twistedness of the informal system for what it is. The gradual accumulation of the rational and irrational, over a long period of time.

For software developers, if they are building systems to handle very specific domain problems, they are the outsiders. That is, they come at it from a distant perspective, and they get to evaluate the rules based not on history growth, but rather a rational logical foundation. They have no choice, they must map the informal system onto a very rigid and precise formal one. Any impedance mismatches between the two systems is both obvious and problematic.

That does give developers a rather unique point of view. We’re not looking at informal systems like law from the vantage point of acceptance. We’re looking at it from the vantage point of complexity, and mapping it to something simpler. Software systems, almost by definition, are going to be simpler and less complex than the informal systems they are mirroring. That’s a product of the sheer amount of resources to build them and keep them running, but also because they generally focus on just a small number of problems contained within the informal system, not the full system itself.

From all of my experience digging to many different informal systems there are a couple of things which really worry me. One is that virtually all of the informal systems I’ve seen contain some type of irrationality. There is always at least one thing there that is just not the product of rational thinking. It just gets built up at the cross roads between other parts of the system. Nothing that would even be constructed deliberately. It’s easy to see these because any domain expert explanations are contrived, messy, hidden or seem to change. That usually points to some underlying irrational component.

The other thing that worries me is that there is always a way for the system to grow, but almost never a way for it to shrink. One can see the consequences of this all over our societies. Everything gets more complicated and more convoluted. People try to justify the growth, but these often are just hollow excuses.

One might consider computerization a simplification method, and sometimes it is. It is not infrequent that implementing a new computer systems means changing the process to some degree to fit the new limits of the code. However development of systems often goes on for decades, and gradually the complexity heads back to its former glory.

Computers in general make it easier to manage complex informal systems. They can do a lot of the grunt labour for people, and they can do it fairly accurately, if they were correctly instructed to do so. We can see this shift as well. Part of the transformation of computers has been the ability to manage informal systems of such complexity that they would have easily collapsed without the aide of a computer. Overall, however, I’m not sure that this is a good thing.

A while back I read in a blog somewhere, that once a complex informal system gets large enough, the only remaining option is for it to collapse. That is, it falls apart suddenly and dramatically. History seems to confirm this view, as we have seen the rise and fall of many a great civilization. At some point the system can no longer increase, it can’t remain as it is and there is no ‘release valve’ for complexity. There are no other options. It basically implodes on itself.

That notion, if true, does not bode well for our current societies. We can see that our complexity has dramatically increased over the last half century, and we can see examples of systematic failure showing up at ever increasing rates, but we seem to have no viable paths of fixing what is wrong. We should be concerned.

If there is some way out of our historic fate, my sense is that we would need to rely on both computers and aggressive simplification to get there. Computers, because they can show us alternatives to our current informal systems, and help us model and control the side-effects of changing what are essentially chaotic systems of n-dimensional variables. And with that initial knowledge, we can start the slow gradual changes needed to simplify our informal systems. The changes have to be slow, just because in general people don’t adapt well and their fragility generates another form of complexity. No doubt it is easier said than achieved, but given the alternative of imploding it doesn’t seem that bad.

There is lots more I could say about formal and informal systems. Defined this way, they provide a general underpinning for a great deal of what happens in our lives. And along with the way we model things internally they help to organize the way we see the epiphenomenon around us. Perhaps, someday I’ll get more of a chance to delve deeper into this area, although all things considered, keeping up with modern life’s requirements is probably too time-consuming and complex these days for me to get another chance :-) 

The Engineering of Software

The last thing I ever wanted to be -- twenty-six years ago -- was an engineer. When I started university I had heard that 70% of all engineers hate their jobs and that they were often bored at work. Boredom sounded horrible.

What I wanted, was to master the black art of coding. To follow in the foot steps of those early magicians I had seen in the computer magazines. The ones that crafted weird cryptic concoctions to do really neat things to their machines. I wanted to be a hacker.

It’s funny how time and experience temper one’s thoughts; how your actions change once you really understand the consequences; how much you grow once you see the larger picture.

Development shops span the range between the cowboys, who are just hacking at the code furiously, and the bureaucrats, that are so ground down in organizational paperwork that little actually gets accomplished.

In my younger days the cowboys where fun. It’s a dynamic environment. Things are happening. And they are happening fast. Well, OK, there are a lot of needless problems cropping up, but hey, that just adds to the excitement, doesn’t it? You could leave work feeling like you got something accomplished.

Well, that lasts for a while. Until you start to get caught in the same hollow accomplishments, over and over again. That is, you’ve fixed the same bug before, wrote the same code before or stuck a band aid on the same gaping wound before. Doing it once or twice is fun. Doing it repetitively, while giving up your life to endless overtime, starts to become depressing after a while. Then suddenly one day you find yourself looking down at the growing pile of band-aids before you and you start thinking “isn’t this all just a waste of my time and my abilities?”

Early on I moved away from the cowboys. They were driving me nuts. I landed deep in the heart of a reasonable engineering culture. Some people on the outside might have confused that development process with a classic waterfall one, but it wasn’t really. Although there was a significant design process up front, the last of the details were always worked out in the various iterations. It was actually a healthy mix. Some thinking, some paperwork, then a hard run at the code. Some testing, a release and then back to thinking again.

Later when I set up my own shop, it naturally started as chaos. Startups are sensitive to cash-flow problems so sometimes the only way to deal with that is an ugly short-term hack. But as time progressed -- every time I got the chance -- I applied long-term fixes. These gradually paid off and when I finally moved on, the place was humming along smoothly releasing a steady stream of mostly high quality code and the occasional patch within a day or two of finding a bug. Each development cycle was a bit easier than its predecessor. Everything was tracked. We had managed to built a big application. One that was far larger, and far more sophisticated than you would have expected from such a tiny shop. I learned a lot in those years.

I found that it isn’t all that complicated. If you boil it down again and again, software development keeps coming back to its roots in engineering. Not in the classic sense that I was afraid of when I was younger -- the large, slow, boring, mindless process -- but rather in the sense that as the work progresses, developers like myself needed to be hacking less, guessing less, and understanding more. The project may start with a lot of unknowns, but at some point it always gets down to routine engineering. That is, we should be doing the ‘right’ things that work because we know that they work, and fixing the little problems -- permanently -- before they become big serious ones.

My earlier fears were misplaced. Engineering itself is not boring. I’m not even sure where I heard that statistic about unhappiness. Engineering, particularly when it comes to something like software, is simply knowing what works and applying it correctly so that the project is successful. It isn’t hack and hope, it isn’t wild guesses, and it isn’t a spin job to cover up the fact that the system is horribly broken. No, quite simply, it is the path to dreaming up cool solutions and then implementing them as cool solutions.

Yes, there are boring times. There is a lot of digging to see what is already out there. No sense reinventing the wheel especially if it’s already decades old. Note that that doesn’t mean you can’t recode the wheel -- making a better one -- but rather that you don’t try to rethink the wheel, that work is already done and accessible.

There is sometimes paperwork. But if one focuses on documentation that’s useful, rather than on the documentation a management consultant would want, then it’s not so bad either. It helps to refine ideas and highlight missing corner-cases long before they become too serious to fix. A little design goes a long way and any organizational chaos, at any level, is always going to grow into a serious problem someday. What you’re missing, or not thinking about, often becomes really obvious when you try to write it down. It also helps you to understand the vocabulary of the problem and to empathize with your users (if you can’t easily explain it in a document then the users aren’t going to be able to understand it either).

There is a lot of thinking. And ironically, although programmers preach heavily about their need for freedom, many choose to turn off their minds when they code, resorting to as much brute force as possible. But at the end of the day, code is just the by-product of thinking about the solution, so it is only as good as the thinking that went into it. To build sophisticated things you have to have a deep understanding of the problem and you have to spend a lot of time working through it. Thoughtless code is dangerous code because of it lacks intelligence. It just causes stupid problems.

And of course, if you’ve really set a firm direction and you know where you are going with the development then the rest of it is just hard work. Little issues have to be sorted out, details need gathering, but the main flow of the development can be worked out in advance. The project may drift but either the long-term perspective still holds, or it is rethought again to insure that it is still valid.

I remember reading decades ago in Scientific American about how the pharmaceutical industry moved from art, to science and then on to engineering. The pills we pop these days are highly likely to contain exactly what they say they will. That wasn’t always the case. It took them a long time to go from alchemy to engineering. The article said that software also started as an art, and has some scientific basis, but it will eventually follow the same path to engineering. I remember the article because I figured I’d bail when that happened. Sounded boring. But these days, as I am surrounded by more software with increasing complexity and decreasing quality I realize that what I want most right now is to get a whole lot more of the engineering mindset into the software development industry. It is time, I think, that we started taking our profession seriously.

There is still room out there for the hackers and the cowboys. Sometimes a band-aid is the best choice, or the solution is so small it doesn’t matter how much of a mess it is. For quick one-offs I think hacking  will always be common, but when I go out and rely on someone else’s code, on their thinking and on their intelligence, I would like to know that they weren’t just mindlessly flailing at the keyboard or that they skipped their homework and just tossed it together. It makes a difference. I don’t want to build on a house of cards. I don’t want to find out later that I’m dependent on somebody who didn’t really ‘get it’ at the end of the day. I don’t want that degree of instability in what I am doing. I can’t build great software on top of a mess; dealing with other people’s mistakes eats through my resources like a knife cuts butter.

So these days I am all in favor of us seriously moving into engineering. Not doing so is a major obstacle that has prevented software from living up to its true potential. I think we can do it, and do it in a way that it is not boring and it isn’t just paperwork. We can still build quickly, but we can do so from a place of understanding, not one of panic. It’s about time that software development grew up and became a real profession, one that is taken seriously. Even in our deplorable state we are changing the world, so can you imagine what we could achieve if our collective works didn’t suck?

Getting to the Truth

One of the most enjoyable things about building software is that it requires the developers to dig around in other domains. If you’re writing financial software, you need to know how the industry works, understand the terminology, and all of the complex details underneath. If you build something for the printing industry, you need to grok their equipment and business models. Everything you write depends on knowledge from elsewhere.

For a naturally curious person like myself, this is a huge benefit. Basically a license to try and learn as much about the world as I can.

Every domain I’ve ever worked in has had a fascinating array of history and depth. You can’t actually solve any of their problems with software until you really understand them; until you get down to what makes them tick. Get to the roots. Get to the truth.

Software is very rigid and to automate any processes they need to be fully understood. To collect valid data you need to understand its structure, quality and volume. To make the interfaces usable you need to understand the environmental issues and the people. If you don’t get to the truth, the software either won’t work or it will cause more problems than it fixes. If your goal is to actually solve something, the only place you can do that is in reality.

But getting to the truth is a funny business. There are all sorts of people out there that will stand in your way. Sometimes it’s because they have something to hide, sometimes it because they don’t want to think about it. Sometimes they just don’t care or figure it isn’t necessary. The causes very.

Lately I’ve come to think of these people obstructing the work as ‘confused’. They are confused about what’s happening below, or above, or why you’re trying to find out, or even what value the software will bring to their lives. The reasons don’t matter, they are just confused.

Building a system on confusion isn’t any better than building an apartment building on a foundation make of foam. It’s obvious that it’s not going to work, even if it looks promising for a moment. If you build on fantasies or delusions or ignore hidden secrets, the result may run, but it’ll quickly drift away from being useful. It doesn’t matter how much data you collect if the data is scrambled or incorrect. And if you ignore the people who ultimately use the work, then they won’t use it or they’ll use it badly, causing massive support headaches. A forced, but miserable user-base can disruptive.

Software has the power to transform, but only if the embedded intelligence within it is correct, and that only occurs if the programmers were exposed to the truth. No truth, then there is no intelligence, and so the results are just burning through wasted resources. This is the fundamental rule that binds the code to something useful in the real world. We’ve known this for a long time, referring to issues like ‘ivory towers’ when talking about programmers who write systems far away from their users.

But this can also happen when the business or management players seek to block access to the truth. And it can also come about as a failure to properly complete reasonable ‘business analysis’, or because the domain experts are confused or lack enough experience. There are a multitude of different ways that confusion can enter into the process and obscure the truth. Distort reality. And without enough truth there is no understanding, which always prevents success.

Problems and Solutions

The world is littered with a growing number of problems. Some of these problems are solvable by using a computer, some not. To solve a problem with a computer all it takes is to design and construct a software solution, then get people to use it. Easy?

Solving a problem with software is not nearly as simple as it looks. Many more solutions fail then succeed. The stats are ugly: 66% of all software projects fail, as do 90% of all startups (most of which are founded around software solutions). Why are these numbers so high?

I think the root of the problem is that a good software solution looks simple. It’s very easy to miss the depth of both the thinking and the work that have gone into making it possible. This leads to a lot of people casually tossing around ideas for solutions, with the expectation that they’ll actually solve a problem. But few of these ideas actually do, and in that very limited remaining subset, the execution and the details mean everything.

To actually solve somebody’s problem with software, the developers needs to look long and deeply at the underlying problem. They need to understand it, not only from the 10,000 foot view, but also right down to each of the tiny details. A shallow understanding of the problem is not nearly enough to solve it. The real issues are often counter-intuitive, hidden and not easily grasped.

Even more complex is understanding the nature of how people use technology. Without some type of incentive, many ‘features’ of a system are useless. Making software ‘sticky’ is a rather complex combination of various issues including stability, usability, psychology, feedback and empathy. Again a broad perspective is not enough to understand these depths.

As well, the developers need to understand the uses and limits of modern technology. Software is still fairly crude and marketing claims for it rarely match reality. Technological issues are often deeply rooted in their theory and history, so both of these need to be well understood before you can properly assess the limitations. You really need to understand these limits before you can apply the technology to solve a problem.

Running a system development project is no picnic either. Software development is like a large train moving slowly down the track, from station to station. You can’t just throw in a right turn here or there. You can’t really change its direction until you hit a station, and even then the number of other accessible stations is very limited. It is easy to miss these inherent and often dangerous properties.

Right down at the core, software is about programming. But just being able to program is not enough either. Some code is good, while some is just a disaster waiting to happen. Experience teaches one more about the code they shouldn’t write, then about what they should. Knowing what’s disorganized, or fragile, or outright dangerous isn’t well documented and sadly only comes with a tremendous amount of experience. Hacking one’s way through a problem space won’t work; won’t produce enough of a stable, coherent solution. Just a convoluted mess. Really understanding what that means in practice takes a lot of time and a lot of experience. It’s not something that people can just read about, or take a course in, or figure out on the fly.

So why is the rate of failure so high? Easy, most of the people out there proposing solutions don’t have a deep enough understanding of what works and what doesn’t, to be able to propose valid solutions. Most programmers are viewed as just ‘code monkeys’ whose task is to build someone else’s solution. So the success or failure of their labours depends heavily on the quality of the proposed solution. A crappy solution will either fail outright, or limp along for years. A good solution can be ruined by a lot of crappy sub-solutions, slowly mutating into something horrific.

It’s for this reason that I often distinguish between programmers and software developers. A programmer can code, but a software developer knows how to solve real problems correctly with software. There is a huge difference. It’s also for this reason that I’ve never liked the term ‘stakeholder’. It seems to imply that they have the right to choose the solution because they have a ‘stake’ in the outcome. But if they don’t have enough of a background to pick a valid solution, then the work will fail. I’ve also seen a great number of entrepreneurs out there with the belief that as business people they should be the ones to pick the solution. That somehow a business background is enough for them to fully grok the problem, the people and the technical issues. This is probably why most startups produce software that doesn’t actually solve any real problems. It just looks shiny and new. So they fail.

The takeaway from all of this is that if you want to create a software solution to solve people’s problems, you need to give the task to someone with experience in creating working solutions with software. An experienced software developer who understands the history, limits and possibilities of software. If you leave it to someone with no experience, the solution is unlikely to be valid. 

The Big Idea

Way back when I was a co-op student, I got a job at a small company that built analytics applications for statistics. My previous co-op employer had laid off the whole division, so I had to find a new gig. There were lots of choices back then, so I picked the company because I would get experience programming in C. It seemed like a good idea at the time.

At first the job was great. I got to work on an interesting little problem for a product about to be released. But as soon as that dried up, the trouble began.

One of the senior statisticians had been asking for resources so he could get a pet project done. He sat way back in a corner office, I hadn’t really noticed him before. In hindsight the location of his office should have been enough of a clue.

The re-assignment started out OK. He had a bunch of little programs he wanted written in C. Nothing complex, not too hard, and the only big challenge for me was the using the language well enough. I had experience in programming Pascal, but C was a little trickier.

My problems started when he begin explaining his ‘big idea’. Variables, he said, where really hard to create and name when coding. It takes time to think up the names, and then you have to remember the names you thought up. Instead, he proclaimed, the secret to great coding is to declare a bunch of global arrays at the top of each program. One for each data-type, e.g. int, float, double, char *, etc. Then all you have to do is index these to get to the correct variable. That is, if you have an integer variable that holds the number of iterations, all you need to do is remember that it was your seventh variable and type in ‘integer[7]’ and you’ve got it. No need to remember some complex name, you just need to know the type and when you created it. Simple.

He didn’t really understand why I recoiled in horror. This was, of course, the most absurd thing I’d ever heard in my short little career. I understood just enough to know why this wasn’t just a step backwards, but rather a quantum leap backwards. It was ‘off the charts’ crazy as far as I was concerned.

I tried to explain why this would render the code completely unreadable (and probably full of bugs), but I was just a student programmer so he dismissed it. I ignored his directive to implement it this way and wrote the code normally, you know with real variable names and scoping as tight as possible. But I wasn’t used to C and I used some of its rope to hang myself (my code was worse than flaky). And so my failure re-affirmed his belief that he was right. The situation grew worse.

Fortunately I was saved by the co-op work term coming to an end. It was a horrific experience, but on the bright side I did end up with C on my resume, which lead to my next job (where I apprenticed with someone who actually taught me how to code properly).

I learned a lot of different things from that experience, and even after more than two decades I’m still learning new things. But the most important of them was how easily it was for someone to get it horribly wrong. Although he had written far more code than I had at the time, he hadn’t written nearly enough to really understand what he was doing. A big idea is far away from actual understanding. But without the latter, the former is unlikely to be reasonable. And unfortunately for us, software development appears simple to those who haven’t plumbed its depths. Thus we get a lot of big ideas …

Thinking About It

That dreaded moment comes in every software project development cycle. The plans are too ambitious, the time too short. After an exciting start, the groove descends into the valley of just plugging away, day after day, while trying hard to keep to an impossible schedule.

It comes every time, on every project (if it doesn’t please, please let me know where you work :-)

What I’ve witnessed over and over is that this moment pushes programmers into speeding up their pace. They convince themselves that what’s needed -- what would make it better -- is more code. So off they go, coding faster and faster, cutting more and more corners.

But that sense of panic is working against them. And against their goal of getting the software released.

In an abstract sense, one can view software development as the act of ‘thinking’ about how to solve problems in a robust manner. The code itself is just a by-product of that thinking. The work is problem solving, but the final output is code.

When you speed up programming, it comes at the cost of reducing thinking. The programmers just fall back into writing the most basic rudimentary splat of code that appears to run. Because of the rush, that code is inherently fragile; the little niceties and less obvious corner-cases are always the first casualties. Spaghetti is a real possibility.

Even worse ‘code re-use’ goes completely out the window. So now the system is rapidly getting filled with redundant code that is fragile, inconsistent and most likely convoluted.

As this first wave results in a mounting bug list, it gets plowed under by even more waves, each one going faster; each one with a deeper sense of urgency. But each one causing more damage. The bug list explodes, the specs change and the technical debt swells to some epic proportion, often ending the project or at very least, resulting in an unstable release that is a nightmare to use or upgrade.

And thus speeding up the coding is a recipe for failure.

What does need to happen is to either speed up the thinking (if that is even possible) or start slashing the goals. Deep, but well-thought-out cuts (to avoid painting oneself into a corner) can push the excess work into later cycles. So long as enough thought about the future is applied, the added technical debt can be minimized. Meanwhile the development pace should be maintained. Evenly.

Sure it’s scary, but with experience this point in the process can be anticipated and even managed properly. A code panic, on the other hand is most likely to lead to ruins. 

Problem Solving

"Mainframe guys?! What do they know? They’re dinosaurs, the industry passed them by years ago...” was my response twenty years ago, when I first got out of school.

Someone had recommended that I take a look at how the mainframe guys (and gals) had solved a particular problem way back, while I was still just finding my identify in high school. But I dismissed it far too easily. Things, I felt, were changing too far, too fast.

In hindsight, of course, it took me a long long time to understand how wrong I really was. There were indeed things that I could learn from all of that earlier intellectual work. Problems that were solved, done and dusted. And what I truly failed to grok was that the work was done when there were less expectations, and more of an ability to focus longer and deeper on getting better solutions.

“Don’t reinvent the wheel” isn’t about using libraries or other existing code. It’s about not re-solving problems that in many cases were solved decades ago. And even deeper, it’s about not just quickly solving them in a brute force manner if there is already an elegant solution that has been deeply thought about and made available.

Still, being young, I wanted to solve problems right away and I guess that orients one to search for low hanging fruit. But inevitably, as there has been a tremendous amount of water under the bridge already, pretty much all of the low hanging fruit has been tasted by others. Many others.

The trick then is to spend some time first, to see if the problem is really new and unique. If it is, great -- you can learn from what people already know, but if it’s not then avoid falling into the trap; try to utilize what was done before. Under time pressure that can seem time-consuming and dangerous, but reinventing really common wheels is a guaranteed time sink that will only make the time pressures worse. After all, it is hubris to think that one can do a better job in a week of two of hacking, then was done decades before with thorough analysis and experimentation. Most solutions for low hanging problems out there are far better than anything we can dream up in a rush. The time spent learning them is time saved from falling prey to well-known issues.

But alas, I think it took me about 15 years to figure that out ...

Layering, Architecture and Convergence

The central problem in software development is managing complexity. This complexity comes from the sea of details and decisions, large and small, that need to converge into a stable, working system in order for it to be successfully used for its intended purpose. That not only includes the code, but also the analysis, management, documentation, packaging, distribution, training, support, configuration and operations that inevitably precede a user being able to utilize some functionality on a computer.

Unchecked, rampant complexity growth in any area can swamp a project, making it impossible to continue substantial progress. About the only way I know to contain complexity is by breaking it off into manageable chunks and getting it encapsulated into well-defined sub-problems. This works at a base level, but new problems emerge. Overlapping sub-problems create redundancies which add significantly to the complexity. This can be managed typing and categorizing the sub-problems carefully, so that overlap is rare or negligible, which gets one past the first hurdle. But as the number of sub-problems grows, their inherent complexity needs to checked as well. At this upper level, they just form a slightly higher, more abstract version of the overall problem, but all of the same issues apply.

Thus, at some point in order to manage the complexity another higher layer becomes necessary and the same issues repeat over again. As the complexity grows, the number of layers to manage it grows as well. Thus, in any attempt to manage growing complexity, a steadily increasing number of layers will need to be created. This is the inevitable consequence of our existence, we can’t handle unlimited sized problems. Smart people or well-running teams may have higher limits, but there is always some limit, and it is always reached earlier than expected.

Layering occurs in all aspects of software development, but for this post I’d like to just focus on how it effects the code, data and operations of software.

The first and most obvious layer in software is the underlying code itself. Programmers get presented with a number of domain or technical problems for which they assemble a large number of instructions for a computer to follow. It is well understood that failure to organize these instructions results in ‘spaghetti code’. That is code with a logical structure so intertwined it resembles the noodles in a plate of spaghetti. Of course this is bad, because any attempt to alter the code is hampered by a significant wall of complexity. From this, within the context of coding, we can draw the conclusion that dis-organization increases complexity.

As the code base begins to grow, it becomes necessary to break it off into more and more sub-problems to manage it. This layer is most often referred to as system’s architecture. It introduces conceptual groups like modules, sub-systems, components, etc. Each of these pieces encapsulates a larger subset of the functionality, and the interactions between them gets formalized. Although I’ve never seen it mentioned, dis-organization at this level can produce a spaghetti architecture, one that impedes both changes and overall stability. Experience coding in the base layer does not easily translate into understanding the organizational issues at this layer. They appear similar, but they are fundamentally different problems.

Once a system has grown out of its base components, it is often desirable to widen its functionality to a larger group of developers. This creates essentially a platform layer where the core abilities are exposed, and controlled, allowing for the existing work to be leveraged for more divergent purposes. Once again, failure to organize this layer results in more pasta, and of course understanding the dynamics of the necessary organization is not directly related to the layers below.

Continuing upwards we get to the product or system layer. This layer generally introduces a new set of problems because it is driven more often by the desires or needs of non-technical people. Organization at this level comes more as ‘paths’ that are followed serially or in parallel, but they still need the same generalized principles of organization. Experience at this level generally involves far more people and negotiating skills, than technical ones. A product manager for a commercial system for instance is balancing the technical issues against the direction and current of the market place.

The final layer that I’m going to discuss is the one that is completely ignored by most technical people. For users to access the system, it has to be running somewhere. In an effort to contain the complexity of design and work most systems form vertical silos. That is they handle a very specific problem and don't easily inter-operate with the other systems being run in their environment. There are some interconnections, but standards are weak because they interfere with individual goals or profit margins. As such, most modern large organizations have a large number of these silos. Like every other layer, redundancies cause significant problems. Thus, at an operational level, all large organizations need an organizational scheme to coordinate the building or purchasing of these silos into a manageable affair. This drives concepts such as the common enterprise software ‘categories’ like CRM and CM, which implicitly contain the primitive building blocks of the operational environment. Again, dis-organization at this level results in an enterprise wide plate of pasta, which diminishes that usefulness of the tools.

No doubt there are many more layers, since they are dynamic and unlimited in number. For any given layer, as it gets better organized we can say it is ‘converging’. By this we mean that in some overall way, its base complexity matches closely to the underlying ‘necessary’ complexity of the layer. If it’s not managed then it diverges, generally as a result of disorganization, over-engineering, over-simplification or any other amplifier that isn’t fundamentally inherent to solving the problem.

It should be noted that there is a subjective (human-based) component to minimal complexity. That is, for reasons based on the way individuals think, the lines between over, under and minimal complexity tend to move around. One person’s perfect trade-offs can appear overly complex to another person. Thus rather than a fixed point, a layer will generally converge within a range, particularly if there are many people involved.

Thus in software development, in order to manage complexity, we want to see the overall system converge on a reasonable number of layers, while each layer is converging on a reasonable system of organization. Development is an ongoing progress, and thus so is converging. We simply need to track all of these ‘paths towards organization’ and continually insure that they are headed for the correct goal. At the bottom layer that is accomplished by programmers pushing themselves for elegance, while at the upper layers is it often architects or managers trying to place some organizational constraints on the chaos below. Every layer needs to get close in order to win.

Contradictions

Programming is easy. You just assemble large lists of instructions for a computer to follow. And these days there is an abundance of information out there about which combinations of instructions will produce the best effects. If you can’t figure it out, there are plenty of people to ask.

But programming is hard, because there is never enough time to assemble all of the instructions yourself so you end up relying on other people’s lists. And often, the behavior of their collections doesn’t match your expectations. They skipped functionality or built it weird, or just hacked it together without thinking.

But it is easy because there are plenty of the stable ‘components’ that have been around for years or even decades. There are loosely-followed standards and although some are quirky, once you understand how to use them properly they’ll do what you expect. It may take a while to find the ‘grain’, but once you know how to utilize something, you can make it work for you.

But then it’s hard because the more stuff you build on, the farther away you get from what is happening underneath. This detachment often causes people to over-simplify their understanding, leading to invalid assumptions about the way things really work.

But it’s easy again because there are plenty of conventions, knowledge and advice out there to follow. Sometimes you have to dig a bit, but it’s out there somewhere. At some point, someone has delved into the details and has spent time explaining them. It just might take a while to find and assemble that information, but if you take the time to learn how to do it properly, you can skip a lot of the frustration and pain.

But then programming is hard, because it often attracts people who want to utilize it, but they don’t have any real experience or patience to learn. They over-simplify the amount of effort involved, and think that short-cuts will get them there faster.  Looking at a screen, many people think that the work involved is close to what they see. But it’s in what lies beneath that all the details and complexity brew. The screens are the easy part, particularly if you follow the existing user interface conventions.

But it can be easy. If you spend the time to learn it, then do it properly, and gradually work through all of the issues one-by-one until they are complete. Eventually you’ll get there. A good solid organized process with no cheap short-cuts will always result in something that is usable. It may just take a while.

But then we’re back to hard, because to overcome other’s lack of experience and impatience we need to point them to an authoritative reference. Somewhere that explains the basis for getting the work completed properly. There are little clues spread every where; some correct, some wrong. But no ‘building code’. Without that, new people coming in just ignore what was known for decades, without realizing that it’s been worked out already.

So is programming easy or hard? When the people around you accept it as hard, then although it may take a while, it doesn’t have to be painful or a mess. But when the people around you think it is easy, they get impatient and take really bad short-cuts to avoid the necessary work, so it becomes increasingly difficult to get anything finished properly within the chaos. Thus it all depends on whether the people around you get that in order for it to be easy, they really have approach it like it is hard. Or slightly restated: it is easy right up until you fall into the trap of thinking it is easy ...