Unwinding Complexity

I keep coming back to complexity. It underlies so much of what software developers do for a living. Building a big, sophisticated software system is all about juggling millions of little moving parts. When it goes well the results are organized, stable and hopefully incredibly useful. We strive to build software that people can rely on to make their lives easier.

Building software is like building anything else, except that few people can see the final results. They can see the interface, feel the performance and sense the stability, but rarely can they stand back and appreciate all of the peices. This allows programmers to hide a tremendous amout of dirt in the system from their stakeholders. Given the high expectations and short timeframes, programing has become increasingly about sweeping dirt under the 'rug'. People chase every possible shortcut, even when they know the long-term consequses are dire. Added to those problems, deep analyse of any business reveals lots of skeletons in their closets plus a raft of inefficiencies that have built up over history. Rarely does a development project clean up the mess it finds, rather it just lifts the rug a little higher and sweeps more dirt under it.

Thus the millions of little parts in a project aren't just code and configuration. They are the huge collection of what is essentially artifical complexity, both internally and externally, that has built up over time. There is never enough time to get it cleaned up and reorganized, so the results just contribute to the ever growing organizational backgroud noise that increasingly hampers progress. Of course it is far worse for larger and older organizations, but even little companies have their messes. 

You'd think that a big new development would be the perfect time to clean house but no one ever leaves the time to get it done. No one ever has the energy. The inevitable consequence is that the dirt builds up layer upon layer. Decades go by while the artificial complexity grows into an ever larger Gordian knot. In many ways our rush to automate with software has just made these problems worse. Computers can simplify, but they can more easily obfuscate, and the way we build software these days, the latter is the most frequent occurrence. 

At some point this complexity exceeds the ability of people to deal with it. These days most large non-technical organizations just flitter around that threshold, spending most time barely in control. The natural response we have to this is to try to tighten down the processes, usually with new rules, but these rules just add more rope to the growing knot. The core, all of those skeletons and dirt, almost never gets touched. The ball of complexity just grows larger. This vicious cycle is what facinates me. Surely there is some way to reverse the cycle; to unwind the problems?

In the early days of the Industrial Revolution people were automating manual tasks in a rather ad hoc fashion. New inventions were siloed, and then thrown into use to deal with isolated aspects of the existing labor. In time, people like Henry Ford approached the problem from a broader perspective. They set the stage for modern factory production. Well-organized 'lines' of production that smoothly turn out works of high precision. An individual craftsman may put more skill and care into handcrafting their work, but their individuality shows in the output. Each peice is slightly different. Factories took some 'soul' out of the work, but they brought down the price while increasing the quality. This type of tradeoff is now well-established for a lot of manual work. We know how to mass product things like cell phones and build incredibly sophisticated projects like skyscrappers. In a sense we can use scale to minimize the overall complexity to a point where we can control quality. But all this knowledge onlyapplies to physical things.

Organizational processes and software both share the same property that the are crafted mostly with intellectual effort, not physical effort. The people doing this work these days operate as individual craftsmen, each infusing the output with our own personal stamp. Added to that, we have no way to measure the quality of our intellectual effort on either the small scale or the large one. We don't know if the results have been well-thought out or are half-baked. Anyone can propose an idea to fix a process problem, but there is no way to measure the quality or depth of that idea. It may fix the issues or it may be misguided and make them worse. There is no way to tell.

What I keep wondering about complexity is whether or not it really is possible to unwind it correctly, in a manner that is controlled? Could I, for instance, actually fix some of the deep company problems that I've run across? Not just in the software, but also in the organization?

My best guess is that complexity can be unrolled, and that it can be unrolled in a controlled process that is predictable. I keep thinking about what the equivalent of a factory line might be in this case. The problems roll in on one side and the appropriate thinking rolls out the other. In between, any and all of the necessary intellectual work that needs to be completed is done in an albeit uncreative, but very reliable way. That is, there is some definiable process that one should be apply to intellectual pursuits that eliminates artificial complexity by essentially normalizing the work in thinking to a finite number of defined steps. To clean up the mess, you just need to correctly follow the process. It might be time-consuming, but it is also deterministic. Once the work gets stripped down that way, it should gain intellectual precision but of course it loses that individual touch. It clearly wouldn't be as fun or creative as intellectual work is today, but the results would hopefully make up for that.

Industrialized thinking would be a much larger leap forward than was factory automation. We could, for instance, tidy up our governmental organizations so that for once they operate efficiently and with the least cost. That probably wouldn't decrease our taxes, rather our countries could provide more services at a much higher quality. The costs incrued by a bedraggled bureaucracy would finally be minimized instead of gradually eating more money for less output. Companies would be freed from their internal fog to really be able to compete well in their markets. Costs would be controllable.

I can definitely see how these ideas could change our world, but are they really possible? It could just be that the invisible nature of thinking negates this possibility in the same way that software isn't visible to most people. We can't yet measure what we can't see. It is this indirect link that binds building software to any possible industrial thinking concept. How do you guarantee the quality of intellectual output? You can't just hire anybody for quality control, and you can't even be assured that the people you hire are really seeing things as they truely are. These types of problems already dominate software development.

So far I don't have any decent answers for this problem. Some people might suggest documentation, but I've seen that fail too often as an approach. People don't proof read very well, and when they do it is usually for syntax not concepts like semantics or higher. To industrialize, the output has to be checkable in some way that would occur in an intermediate stage, long before it is too late. It would have to be reliable.

One approach that might have merit is to trace all things backwards. Start at the end and traverse back to the start. That is, if you wanted to simplify something like the laws of a country, you start tracking those that are actually being applied. Where they depend on other laws for properties like consistency or integration those get added into the mix as well. At some point you'll end up with a denormalized tree (dag actuallly) with what's currently in practice. Normalized these relative to a consistent underlying set of principles and the results should hopefully be a considerably smaller sets of 'clean' laws that could effectively replace the older ones. Even from the description you can tell that this whole process sounds rather tedious and boring, but if it really has those qualities and it is complete when done than it is likely the sort of industrialized approach that we are looking for. It might not be as fun as crafting endless new laws on top of each other, but compressing the overall complexity of the legal system would make it far easier for people to get fairer resolutions to their problems. Some lawyers might not be happy about that, but it would benefit society.

Roughly, it took about a hundred years for manual automation to get industrialized and it probably took another one hundred to get to today's level of precision. Cleaning up our disorganizations and ad hoc thinking processes will most likely follow suit at some point in the future. It may take generations, but it is likely inevitable. Until then, the only ways to clean up artificial complexity are time and luck, and pehaps the occasional brilliant insight.