Out of Control

The complexity of modern software systems has gotten out of control. Decades back, computers served very well-defined roles. They computed specific values, or they kept track of manually entered data. It was all very constrained. Nice and simple.

These days, technical ability has embedded itself deeply into many industries; they need their computers to remain competitive. And these machines need mass amounts of data just to keep up. Everything is now interconnected, churning around in an information haze.

But the silos that enabled the earlier systems are now the impediments to utilizing all of this collected data. Above this lies layers of spaghetti so intertwined by history that there is no hope of sorting through the whole hideous clump of knots. All this is serviced by increasingly stressed operations departments just trying to stay afloat of the shifting technologies, security issues, devices, weird processes and out-of control user expectations. Most of these groups, just one small step away from catastrophe.

Modern systems are intrinsically complex, but their rough evolution has hugely amplified the problems. Underneath software is fairly simple. It rigorously models attributes of the world, grinding through the data to provide some insight into behaviour. We have enough knowledge now to really understand how to organize the data, and construct the code to achieve our goals. However, history, politics and a general lack of understanding inflate the issues forcing what could have been a straightforward engineering effort into a swirling cloud of chaos whose results are most often disappointing. People rush to build these systems, skipping over what is known in order to bend to the pressure of getting things out too early. These are of course self-inflicted injuries. Weak code, badly organized data, lack of standards, and no empathy for users or operations results into piles of unstable code fragments that behave badly when exposed to the real world. Compound this with an obsessive need to restart from scratch continually or to just blindly build on something else, and what results are sprints that prematurely burn out then come crashing down.

There are no short-cuts in software development. There are trade-offs, but there is no easy way to bypass the consequences of a long series of poor decisions. To get things working, people have to work through all of the problems diligently and to great detail which is often a slow and painful process. You can’t skip it, defer it until later or rely on luck. For code to not suck, it has to be well thought out, there is no way around this since code is essentially a manifestation of the the underlying knowledge of the programmers involved. If they don’t understand what they are writing, then the code reflects that. And if above them the environment is not organized, then the system and the data reflect that as well. In that sense a system is just a mirror of where it was created and where it is running. It is only as stable and reliable as it’s environment.

The irony of software development is that lots of experience makes one understand how easy it could be, yet exposes them to the full ugliness of how it usually is done. For programmers, once you can see above the code, the silos, the technologies, etc. you can imagine the possibilities of building way more sophisticated and usable systems, things that would radically change the value to the users, but now you can also now see the environmental, organizational and chronological roadblocks that will often prevent you from achieving this. Software could be better, but its rare.