Essential Development Problems

Over the years I've worked in many development sites, read tonnes of code and heard way too many horror stories about development projects gone bad. While software development is inherently risky, many of the problems I have witnessed were self-inflicted, thus fixable. It is an odd historically-driven aspect of programming, our need to make the work more difficult than necessary.

I wasn't focusing on writing yet another list of things to simplify development, but in my other writings the same issues kept bubbling up over and over again. Sometimes even when you explicitly see something, it is not easy to put a name or description to it. In this case, as I explored more elementary topics I kept getting these 'pieces' floating to the top, but not fitting into the other works. Once I collected these four together, they fit quite naturally.

For this blog entry, I'd like to address this commonality with basic development problems, but try as much as possible to avoid falling into simple platitudes (no guarantees). These problems are elemental, mostly obvious, but surprisingly persistent. Software developers, I am strongly aware, are their own worst enemy. Sure, there are good rants to be had at the expense of management, users or technology, but the present of other problems doesn't validate the desire to overlook the fixable ones.

PROBLEM: PERCEPTION

One significant problem that rears its ugly head again and again, is our actual perception of what we are doing. We all know that if you think you will fail, you definitely will. There must also be some well-known universal truth that states if you go in with the wrong viewpoint, it will significantly reduce your likelihood of success. Even in the case where you are positive, if you really don't understand what you are doing, then you are implicitly relying on luck. And some days, you're just unlucky. If you want to be consistently good at something, you really need to understand it.

When I first started programming I subscribed to the 'magic broom' theory of programming. The best illustration comes from the episode called the Sorcerer's Apprentice in Disney's animated film Fantasia. There is at least one youTube link to it:

http://www.youtube.com/watch?v=LD8HDta7Z_4

In the episode, Mickey Mouse as a Wizard's apprentice, gets a hold of the 'magic hat', then starts issuing special commands to an inanimate broom to deliver water to the Wizard's workshop. Mostly because he falls asleep, the circumstances start to go rapidly out of control. Awaking in a panic, he is unable to stop the growing problem. A hatchet job only intensifies the issue. In the end, he even tries consulting the manual, a desperate step to be sure, but it is far too little, far too late. Is this not an excellent allegory of a programming project gone awry?

My early view of programming was similar. You issued a series of magic commands to the various 'brooms', and presto, blamo, your work is done for you. It is all magic underneath. In this animation, the fact that any apprentice with access to the hat can easily issue commands, even when they do not fully understand the consequences of their actions, relates this back to computers quite effectively. As this common quote likes to describe:

"To err is human, but to really foul things up requires a computer." -- Farmers' Almanac, 1978

Computers amplify their user's abilities; they are the mental equivalent of a bulldozer.

Needless to say, I was disappointed when I discovered that computers were just simple deterministic machines that never deviated from what they are told, and that the commands themselves where only abstractions thought up by other programmers. A significant amount of the 'magic' is just artificial complexity added in over years and years of development.

This is nothing magical about computers. They are machines that unless there is a hardware failure, consistently do the same things day after day. Of course, that's no longer so obvious to most users of our modern 'personal computers'. They have becomes so complex and often so erratic, that many people feel they have personality. Some even think that their machines are actively plotting against them.

THE ART OF COMPUTER PROGRAMMING

For a quite a while I subscribed to the 'writing software is an art form' school of thinking. This view sees programming as a very specific art form, similar to painting or writing poetry. It is an inherently elitist viewpoint based around the idea that some people are naturally born to 'it', while others shouldn't even try. Not everyone is cut out to be an artist, or in this case a programmer.

While not quite as fantastical as the magic broom theory, there is an implication that software is not repeatable, in the same way that any large great work of art is not repeatable. If you accept this, the consequences are quite scary. Given that we are dependent on software, this view directly implies that there is no way to consistently make software stable. The 'fate' of any version of the system rests with the ability to get a hold of the right 'artists'. And these are limited in quantity. So limited, that many programmers would insist that there is only a small handful. Or, at very least a few thousand. But certainly not the masses of people slaving away in IT right now.

Software as an art form also implies that there is some 'essence' that we can't teach. Great art for me goes beyond technique and materials, it happens when an 'artist' manages to capture 'emotion' in the work along with all of the other details. A grand painting inspires some emotional response, while a work of 'graphic art' just looks pretty. It's the same with musicians and pop music. Rarely does a pop song paint a vivid picture, instead people just tap along to the beat. When it 'gets' to you it is art, otherwise it is just 'easy listening'.

Given that definition, it's hard to picture a software application that directly inspires the user on an emotional level. That would be a word processor that makes you depressed just by starting it, long before you've been disappointed by its inherent bugs. Or a browser that just makes you happy, or angry, or something. The idea, then in this context is clearly silly. A tool, such as a hammer, may look pretty but it does not carry with it any emotional baggage. For that you need art, and the 'thing' that a true artist has, that is not 'common' in the rest of the population (and often requires a bit of mental instability), is the capability to embed 'emotion' directly in their work for all to see. A very rare skill.

What is driving a lot of programmers towards the 'art form' theory is their need to see what they do as 'creative'. As it involves complex analysis, and building tools to solve specific problems, there is a huge amount of creativity required to find a working solution to a set of problems. But, and this needs to be said, once that design is complete you've got to get down to work and actually build the thing. Building is just raw and ugly work, it is never pretty. If you are doing it well, it shouldn't be creative.

I don't want for example, the electrician in a new house randomly locating the plugs based on his own emotional creativity at the moment. That would suck, particularly if I have to plug things into the ceiling in some rooms because it was a bad hair day. Artists can be temperamental and have periods were they cannot work. Software developers are professionals, and come to the office every day, ready to work (lack of available coffee can change this).

As for the elitist perspective, there are certainly skills in software development, such as generalization and analysis that are extremely difficult to master. And in the case of generalization, some people naturally have more abstract minds than others. Beyond that, the 'encoding' of a design into chunks of code with specific structures is not difficult. Not unless it is just not specified. Elegance is also hard to master, and certainly some people see that right away, but most people should be able to write a basic software program that works at least. Of course, 'just working' is only a small part of the overall solution.

HARD-WORK

Software development is often 5% creativity to find the solution and 95% work to get it implemented. Sometimes, particularly at the beginning, when you are still looking for the 'shape' of the tool, it may be more like 10% or 15% creativity, but ultimately the work done to implement the code is just work and nothing more. If we want it to be more reliable, then we need to accept it as less creative.

We like to make programming hard, because it satisfies our egos. We feel good if we solve problems that we believe other people cannot solve. Our own personal self-esteem is ridding on this. But our desire to inject 'creativity' into the implementation process, is really quite insane. In no other field would it be acceptable to make a plan, then while implementing the plan, go off and do something completely different. Is our high rate of failure, not directly linked to this self-destructive behavior? Even more oddly, the often stated evolution from foolishly ignoring the 'plan', is to just not have one in the first place. That way you don't have to ignore it. That can't end well.

For me, I have grown as a programmer only when I accepted that there is less creativity in what I do, than I would like. That's OK, I look to my 'life' to fulfil some of my creative juices, with things like drawing or photography. Really good hobbies that allow unadulterated expression to bloom. With that concession, I can see software development through an objective eye. From that perspective I found I could build faster, better and more accurately.

There are very complicated parts of programming. In particular, analysis, because of the inherent messiness of people is hard and often because it is the initial cause that knocks over the other dominoes in the project. But, once in having decided on what it is that we need to build, the 'hardness' of the problem is significantly reduced. The creative part comes and goes. We shouldn't hold onto it because we want to appear smart, or because we are trying to be an artist, or even because it makes the days more entertaining. Adding 'artistic' inconsistencies to a program is fun, but people hate it, it is messy, and it ends up being more work. You get a lot of misery in exchange for very little creativity.

PROBLEM: DISCIPLINE

Good programmers write elegant bits of code. Great programmers write consistent code. More so than any other 'skill', programmers need self-discipline to keep their code clean and consistent, after all they will be working on it for years and years. Forget all of the techniques, approaches or styles, it all comes down to using one, and only one consistent way to do things. Even if that way is arbitrary, that is irrelevant (well, its not, but it is close enough that it doesn't matter).

Honestly, it doesn't actually matter what style of code you write so long as you are consistent. If you implement some programming construct into the code repeatably, but identically, it becomes a simple matter of removing it and refactoring it with something more advanced. If you implemented it into the system in twenty different ways, the problem of just finding what to change and what not to change is complicated, never mind replacing the actual code. A stupid idea implemented consistently is worth far more than a grand idea implemented in multiple different ways. At least with a consistent stupid idea, you can fix it easily.

It isn't hard to stop every once in a while and refactor the code to cleanup the multiple ways of handling the same problem or consolidate several different versions of the essentially same block of code. It is just self-discipline. I've always like to start all major development with some warm-up refactoring work. In that way, as the project progresses, the code base gets cleaner. Gradually over time the big problems are handled, and removed.

For this, programmers love to blame management for not giving them the time, but if it is a standard at the start of any development cycle, the time is usually available. Just don't give management the option to remove the work, make it a mandatory part of the development. Because it is first, it will get done, even if the current cycle ends in a rush. Cleaning up the code isn't hard, it usually isn't time consuming, and it is definitely professional.

For both individuals and groups, if you look at their code base, you get a picture of how well they are operating. Big ugly messy code bases, show an inherent sloppiness. Sloppiness is always tied heavily to bugs and frequent interface inconsistencies. It is also commonly tied to schedule slippages and bad work estimates. Sloppy code is hard to use and nearly impossible to leverage. Sloppy code is common. Even most of the cleanest commercial products barely score a 6 out of 10 on any reasonable cleanliness scale, while most are far less than that. Other than procrastination, there often isn't even an excuse for messy code.

We don't like discipline because it is repetitive. We don't like repetition because it isn't creative. These two problems are clearly linked. The most 'creative' programmers often sit atop of huge messes. Sure, they get it, now. But should they ever come back in eight years and have to alter their own code, as I did, they too would get to thinking "boy, did I make a mess of this".

An important corollary of maintaining self-discipline is to accept that the code isn't just written for yourself, successful code will have lots of authors over the years.

PROBLEM: GENERALIZATION

Computers need to follow long sequences of instructions in order to implement their functionality. Using 'brute' force as a technique to write a program is an approach that comes from pounding out, in as specific a manner as possible, all of the possible instructions in a computer language that implements that piece of functionality. Aside from being incredibly long, the results tend towards being fragile. Mostly the repetitive nature of the instructions, as they get changed over time tend towards the various blocks of code falling out of sync with each other. These differences create the instabilities in behavior so often called bugs. They also make the software fugly. While brute force is slow to build, fragile and hard to maintain, it is by far the most common way for programmers to write systems.

If you generalize the code, a smaller amount of it can solve much larger problems. It is this 'batching' of code that is often needed to keep up with the demand and schedules for development. If you need to write 50 screens, each one takes a week, so you've got a year's worth of effort. But if you take two weeks to write a generalize screen that would act in place of 10 other screens, then you could compact your work down to 10 weeks. Each new screen is twice as hard as the original, but leveraging them cuts the work by 1/5. Of course you'll never be that lucky, you might only get 4 out of 5 to be generalized, but your now at 18 weeks which is still half the time. That type of 'math' is common in coding, but most people don't see it, or are afraid to go the generalization route out of fear that a screen might take 4 to 8 weeks instead of 2. Note that: 8*4 = 32 + 10, is still less than 50, and your 4X off your estimate. Doesn't that still allow for 8 weeks of vacation?

Generalization is hard for many people, and I think it is this ability that programmer's so often confuse with creativity. Generalizing is probably a form of creativity, but it is the ability to alter ones perspective that is key. To generalize a solution, you need to take a step back from the problem and look at it abstractly:

http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html

There have been many attempts to take generalizations, as specific abstractions and embed them directly in the programming languages or process. Object-oriented design and programming, for example appear to follow in the Abstract Data Type (ADT) footsteps, by taking a style based abstraction and extending it to various programming languages. Design Patterns are essentially style based abstractions that are commonly, and incorrectly used as base implementation primitives. While a language or a style may provide a level of abstraction, the essence of abstracting is to compact the code into something that is generalized, reusable and tight. Any of the inherent abstractions provided assistance, but used on their own or abused they will not meet the requirements of really reducing the size of the problem. Often, when improperly used they actually convolute the problems and make the complexity worse.

Even as we generalize, we still need the solution to be clean, consistent and understandable by the many people that will come in contact with the code over its years of life. Abstractions that convolute are not the same as ones that simplify. Focusing on the data always helps, and researching existing algorithms can be a huge savings in experimentation. Often the best approach is to start with some simplified algorithm, and extend it to meet the overall problem space.

This higher-level view simplifies the details, allowing less code, reuse, optimizations and many other benefits. Of course it is harder to write, but with experience and practice it comes smoothly for most programmers. It should be considered a core software developer skill.

Generalization, when it is implemented properly is an abstraction that is encapsulated at specific layers within the system. An elegant implementation of an abstraction is the closest thing we have to a silver bullet. Most programming problems resemble fractals with lots of little similar, but slightly different problems all over the place. If you implement something that handles the base and can be stretched in many ways to meet all of the variations, then one simple piece of code could solve a huge number of problems. That type of leverage is necessary in all programming projects to keep up or meet the deadlines. The reduction of code is necessary to allow for easy expansions of the functionality.

PROBLEM: ANALYSIS

If you have master the first three things in this post, you have the ability to build virtually anything. However, that doesn't help you, unless you actually know 'what' to build.

The weakest link and clearly the hardest task of most programming projects is the analysis. It is usually weak for several key reasons (a) the problem domain is not well understood, (b) the other influencing domains, like development and operations are not factored in and (c) the empathy for the user is missing, or was misdirected.

Software is just a tool for the users to work with. The only thing it can actually do is be used to build up a pile of data. Given that simplicity you'd think it would be easy to relate the tool back to the problem domain. But, as with all things, two common problems exist, (1) the designers never leave their shop to take a look at the actual usage of their tools, and (2) even if they do, they don't do it enough, so that they have the wrong impression about what is actually 'important'.

To be able to build up a workable software tool, you first need to understand the vocabulary of the industry. Then, you need to understand the common processes. Then you need to tie it together to find places where specific digital tools would actually make the lives of the users easier, not harder. Nothing is worse than a tool that just makes misery in a poor attempt to 'control' or 'restrict'. If you want the users to appreciate the tool, it needs to solve a problem for them. If you really want them to love the tool, it needs to solve lots of problems.

The key to understanding a problem domain is listening to the potential users. They have the various 'fragments' that are needed to piece together the solution. Although the users are the experts in the problem domain, the developers should be the experts in building tools. That means that the nature and understanding of the problem comes from the users, while the nature and understanding of the solution comes from the developers.

Analysis is an extremely difficult skill to master, it is not just taking notes, writing down 'requirements' or any other simple documentation process. It is listening to what the users are saying, and reading between the lines to put together a comprehensive understanding of what is really driving their behavior and how that relates back to their need to amass a big pile of data about something. In the end, thought, it is always worth remembering that software is just a tool to manipulate data. Knowing a lot about the users is important, but knowing everything is not necessary. It is usually best to stick close to the data, the processes and the driving forces behind their actions.

Understanding the problem domain is important, but the whole space is not just the problem domain, but also the development and operations domains. How the program is created and maintained are often key parts of the overall problem. Building software in a place that cannot maintain it is asking for trouble. Building something that is hard to install is not good either. The development, testing and deployment domains have an effect if you consider the 'whole' product as it goes out over time and really gets used.

Removing big programming problems produces more stable releases, requiring less testing. That in turn means the code gets into usage faster. If you utilize this, the releases can be smaller, and the benefits get to the user faster. This shorten cycle of development, especially in the early development days helps to validate direction more closely. It doesn't save you from going down the wrong path, but it does save you from going down too far to turn back.

An important point for analysis is to always err on the side of being too simple. The primary reason for this is that it is easy to add complexity when you determine that it is needed, but very difficult to remove it. Once you've zoomed yourself in believing a model or rules are correct, deleting parts of it becomes impossible. Working up from simple, often meets with the development goals and the product goals as well. Another reason is that a tool that is too complex is not usable, while one that is too simple is just annoying. It is better to keep the users working, even if they are frustrated, then it is to grind it all to a halt. They have goals to accomplish too.

Many programmer's are quick to put down the complains of their users as misdirected, thinking they just noise, or ignorance. Where there is smoke, there is usually fire. A lesson that all programmers should consider. The biggest problem in most systems today is the total failure of the programmers to have any empathy for their users. And it shows. "Use this because I think it works, and I can do it", produces an arrogant array of badly behaving, but pragmatically convenient code. That of course is backwards, if the code needs to be 'hard' to make the user experience 'simple' then that is the actual work that needs to be done. Making the code 'simple', and dismissing the complaints from the users, is a cop-out to put it mildly.

People are messy, inconvenient, lazy and they almost always take the easiest path. Sometimes this means that you need to control their behavior. However, overly strict tools diminish their own usefulness. Sometimes this means giving the users as much freedom as possible. However this is more work. Balancing this contradiction is a key part of building great tools. If the essence of software development rests on deterministic behavior, the essence of understanding users rest on their irrational and inconsistent actions. This is nothing wrong with this, in so long as you build any 'uncertainty' into the architecture. If the users really can't decide between two options, then both should be present. If they choose easily, then so can you. Match the fuzziness in the responses of the users with the fuzziness in the analysis with the fuzziness in the design.

Feedback is the all important final point. If your tool simplifies the issues, then there will be problems with its deployment. This should feedback into the upcoming development cycles, in a way that allows for the overall understanding to grow properly. Ultimately, some new code should be added, and some code should be deleted. It is worth noting, again, for self-discipline reasons, that any code that should be deleted, is actually deleted. It should go.

In this way, over time the tool expands to fill in the solution and more and more of the corner-cases get handled. The results of a good development project will get better over time, that may seem obvious, but with so many newer commercial versions of popular software getting significantly 'worse' than their predecessors, we obviously need to explicitly say this.

IT SHOULD, AND CAN BE EASIER

Hang around enough projects and you'll find that most 'fixable' implementation problems generally come from one the above issues. There are, of course, other issues and possibly serious management problems circling around like vultures, but those are essentially external. You just have to accept some aspects of every industry as they are forever beyond your sphere of control; a very wise senior developer once told me when I was young: "pick your battles". There is no point fighting a losing battle, it is a waste of your energy.

Those things that come from within, or nearby are things that can be controlled, fixed or repaired. Flailing away at millions of lines of sloppy code is painful, but enhancing millions of lines of elegant code is exhilarating. The difference is just refactoring. It could be a lot of time, for sure, but it will never get done if you never start, and to benefit, you needn't do it all right away. Cleaning up code is an ongoing chore that never ends.

You may have noticed that the these development problems are mostly personal things. Things that each developer can do on their own. For individuals and small teams this is fairly easy. Big teams are a whole other problem. Often the real problems with the huge teams are at their organizational level. Inconsistencies happen because the teams are structured to make them happen. At the lower-levels in a big team the best you can do is assure that your own work is reasonable and that you try to enlighten as many of your colleagues as possible to follow suit. Getting all the programmers to 'orient' themselves in the same direction is a management issue. Mapping the architecture in a way to avoid or minimize overlaps and inconsistencies onto the various teams is a design problem; part of the project's development problem domain.

Finally, you know you've stumbled onto something good when it summarizes well:

If you change your perspective and keep the discipline, mostly your code will work. If you add in good analysis it will satisfy. If you generalize, then you can save masses of work, and get it done in a reasonable amount of time. These four things more than any others are critical. Dealing with them makes the rest of software development easy and reliable.

The Construction of Primitives

There is still plenty of 'meat' left in exploring fundamental development issues. We could spend time on the more interesting higher-order topics, but when the discussions get gummed up in problems with basic terminology, understanding or process we don't make significant headway. As a young industry, Computer Science has established some level of competency, but many of our key assumptions are still open for discussion. We have built on our knowledge, but our results are clearly not reliable.

Software developers, it seems, have all gone off to their own little worlds, a situation which is quite possibly analogous to mathematics prior to Sir. Issac Newton managing to enforce notational standards. With everyone running around using their own unique terms and definitions, we are getting a lot of impedance mismatches in our conversations. As far as I know, there isn't even an 'official' body on which we all could agree as being the respected authority on software development. Our industry has fragmented into a million pieces.

For this post I figured I would continue on digging deeply with a look into the way we choose to decompose code into primitive functions. Mostly I'll use the rather older term 'function' when I talk about a block of code, but this discussion is applicable to any functional paradigm including object-oriented, for which the term 'method' is more appropriate. But I'll stick to using function because it is the term with the least connotations.

Again, as with any of these abstract discussions this will start fairly conventionally and then get plenty weird as it winds it way into the depths.

PRIMITIVE FUNCTIONS

We are all familiar with arithmetic; it is a simple branch of mathematics that consists of the four basic operators +, -, * and /. True number theorists would also define some information with respect to the underlying set of numbers, e.g. integers, real, complex, etc. but for this discussion I'd rather stay away from group and ring theory; they are two rather complex branches of mathematics that are interesting, but not immediately relevant.

What is important is that we can also see the above operators as the functions 'add', 'subtract', 'multiple' and 'divide'. Each one of these functions takes two arguments and returns a resulting value. We get something vaguely like:

add(x,y) := return (x + y);
sub(x,y) := return (x - y);
mult(x,y) := return (x * y);
div(x,y) := return (x / y);

There are other 'wordings' that are possible to describe these functions. I think in some cases you might call them axioms, in some contexts they could represent a geometry, or we could talk about them as language tokens that have a specific expressibility. In x86 assembler the following functions for handling unsigned types are similar:

ADD -- Add
SUB -- Subtraction
MUL -- Unsigned multiply
DIV -- Unsigned divide

There are many more 'isomorphic' ways of addressing the same four functions, but in the end, they are all essentially a description of the same four 'primitive' functions that comes from the collection of operators available in arithmetic.

Whatever terminology or space you want to use to look at these four functions, what I want to draw towards you attention is not the verbiage used to describe them, nor their underlying behaviors, instead it is the relationship between the four things themselves. Note that a) they are all of the possible operators* b) they do not overlap** and c) they form a complete set***.

* all of the operators for 'arithmetic' only, on the ring for real numbers for + and * or something like that (it has been decades since I took ring theory).
** multiply is really X added Y times to itself, but you need the Y for the number of adds, so even though it is rooted in addition, it is still something unique.
*** add is the inverse of sub, div is the 'sortof' inverse of mul (see group theory). They have all of the consistent and complete properties: associative, commutative, inverse, entity, etc.

We can see that these four primitive functions form a 'language' for which we can express all possible arithmetic problems. If you need to do any arithmetic, all you need to know and understand are these four functions. They are simple, consistent and complete.

On a lexicographical note, I prefer to the use the term 'primitive' to describe any set of functions which do not overlap with each other. I think 'atomic' was another term we used in school, because each operation was an atom at the data structure level -- which might have possibly been a pre-quantum-physics usage -- but we have added so many connotations to that term that it has a very definitive meaning to most people. Also, 'atomic' sometimes means a function that will execute in one all-or-nothing shot, such as 'test and set'. Other terms like elementary, fundamental and primal, etc. are OK, but 'primitive' works well because we use it in regard to type information as well, e.g. 'int' is a primitive type. So, the functions are primitive, but not in the sense that they are crude. It is the sense that they are at the very bottom level and are not decomposable into smaller 'primitives'.

A DIFFERENT SET OF FUNCTIONS

So far, so easy. Just a set of the four base functions that completely cover and define arithmetic. Now suppose we created eight more functions, each taking two values, and working them as follows:

f1 = (2*x*x - y) / 2*x
f2 = x + 2*y
f3 = 4*x
f4 = 3*x
f5 = x*(-2*x - 5*y)
f6 = 2*x*y
f7 = x - y
f8 = y*( -2*x*(x + y) + x)

What is interesting about these eight functions is that none of them is identical to any of the original four functions. Also, they are obviously quite complex. Interestingly, combined they cover the exact same space as the original four functions:

add = f1 - f2 / f3
sub = f2 * f4 + f5
mul = f4 + f5 / f6
div = f6 * f7 + f8

However, as functions they have a significant amount of overlap with each other. f3 and f4 for example are close but not identical. Most of the functions use the * operator. All of them have at least one one underlying arithmetic operator, while one has as many as 6. Together they form another 'complete' interface for arithmetic, but they are clearly not primitive functions. They are composites.

So, we have two complete sets of functions, each forming an API that can be used to express any and all arithmetic problems. They are similar, equally expressive, but clearly the second one with eight functions is considerably more complex than the first. It would be possible to memorize all eight functions and use them entirely in place of +, -, * and /, but given their rather useless added complexity who would bother?*

*You can speak Klingon? You're kidding :-O

Now this might seem to be a contrived example, but it happens frequently on a larger scale. Consider a library for a graphic user interface (GUI) that consists of 300 basic functions. Now consider another one that has over 4000 functions. It is obvious that if we could find 300 functions to express most of the functionality needed for handing a GUI application, then few of the 4000 functions would likely be 'primitive'. While the overlap between the two libraries might not be exact, without even getting into the specifics we can start seeing when things are composed of primitives and when they are composed of many composite functions.

But we do have to be a little careful. There can of course be multiple different independent sets of primitives that share the same 'space'. The classic example is with Boolean logic. AND, OR and NOT form a complete set of operators. NAND by itself does as well. The two sets of primitives are completely interchangeable, consistent and complete, although one is three times the size of the other.

Primitive functions can exist for anything, although we generally tend to see them as elementary operations on things like abstract data types. For example, linked-lists generally have the functions: new, find, delete and insert defined. Although data-structures are a more popular usage, every sequence of instructions can be broken down into a reasonable set of primitives. And every set of primitives can be consistent and complete.

WHY IS THIS IMPORTANT?

We break our programs into smaller pieces so that we can manage the complexity.

At each different layer we provide some type of interface so that ourselves or more often, other programmers can access the underlying layer. We always want the simplest possible 'interface' that can do the job, given whatever level of abstraction and encapsulation are necessary. Any interface that is even marginally more complex is 'artificially' complex, because it is possible to remove that complexity.

There may be circumstances where artificial complexity is acceptable because of the abstraction or the need for an optimized non-primitive version of the function, but that is probably a weakness in the interface and most likely represents a bad design. Non-primitives are repetitive, which increases the risk of bugs.

Our best interfaces are the ones that presents a consistent set of primitives to the user without overlap. The functions themselves form the basis for the grammar needed to spell out the problem at a higher level. We simply 'say' the higher-level problem in terms of the lower-level primitives. Arithmetic, for example is spoken in terms of addition, subtraction, multiplication and division. It forms the basis of any book-keeping system. With these basic functions we could start to lay the foundations for a double entry accounting program.

A good interface, with its consistency allows the user to remember only a small number of 'axioms' that they can then use to spell out a larger number of higher-order problems. Completeness and consistency with primitives makes it easier, and thus significantly faster to express the solution with the overall set of available functions. All of the extra artificial complexity massively gums up the works. It is easy to remember the four operators for arithmetic, but although it is only another four operators, try actually writing out any type of significant arithmetic calculation using the arbitrarily complex eight functions in the first example. Although the API is only slightly larger, the complexity is through the roof.

GETTIN' JIGGY WIT IT

In an earlier entry on abstraction I suggested that data was only an abstract projection of something in the real world:

http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html

That abstract view of data is extremely useful in accepting some of the behavioral characteristics of data within a computer. Extending this a bit, there are essentially only two things in all software: data and code. There is nothing else; everything is either one or the other. Going farther, all code is broken down into functions, and on a very esoteric level there is no real difference at any give point in time, between the data itself and the functions as they exist in the computer.

The key point here is 'in time'. Functions execute, but frozen in time they are just data as it exists in the system. A great visualisation example of this is a fractal generation program. Before it has been run, the fractal program will only have high-level information regarding a Mandelbrot fractal. But that information, in the form of data, is stored as a set of symbolic finite discrete parameters that explain to the computer the underlying 'details' of a fractal. When you run the program, you pick a range for the fractal to be 'visualized' within. The program generates from its internal information a large set of data that it can display as an image. This image shows what the fractal looks like at a specific range.

The information within a fractal is infinite, you can endlessly explore the depths forever, there is no limit*. But at any given point in 'time' all of the data on a computer is discrete and finite. That you appear to be browsing through this infinitely deep Mandelbrot fractal is somewhat of a trick. The computational power of your computer is stretching out the fractal definition infinitely over the time dimension while inside of the computer at point there exists just the finite set of information describing the fractal, and a few images of it at various ranges. Nothing more.

* there are algorithms, such as Karatsuba that can deal with an arbitrarily large precision, given an arbitrarily large amount of time. At some depth, physical resources like disk space may become an issues as well.

A function as a piece of data at a point in time is a hard concept to grasp. The converse, making functions out of all of the data in the system is much easier and a standard for object-oriented programming. Just about everything you access in most modern object-oriented programs is a 'method' for some underlying object. The data exists, but is almost always wrapped in a functional layer. Interestingly enough, most of the Design Patterns center around data, but some are effectively data'izing function calls, such as the Command pattern. Commands which are 'tied' to functions, once issued become data, allowing them to be 'undone'.

Although it is rather an abstract notion, data and functions are essentially the same thing. Well, at least at any discrete slice in time. There are many languages and techniques where data and functions are mixed to great effect. Being able to abstractly see one in terms of the other allows for a broader level of generalization.

PRIMITIVE DATA

Given that data and functions are mostly interchangeable, we can generalize some of our understanding about primitive functions.

Data can be primitive as well, but the internal meaning is a little different than functions. Primitive data is information that has been parsed down into its smallest usable units 'within' a system. If for example, the system stored and used Universal Resource Locators, better known as URLs, to maintain the location of dependencies on the Web, these are the base level and there is no need to decompose them any further. However some systems may need to break up the data to access smaller pieces such as the embedded host name or protocol information. In that case, they need to parse the URLs to access the sub-data. URLs may be primitive in one system, but not in another.

Dealing with non-primitive data brings up two common problems: a) redundant data and b) data overloading. If for example, the code requires both the URL and the host name, some programmers would parse one from the other and store both.

Redundant 'anything' in a computer is always dangerous because as the code is changed frequently by the programmers, there is a significant risk that changes might not occur equally to both copies. At all levels in the code, the data should be manipulated only in its most primitive form. That might change depending on the level, but it needs to be consistent with each segment of the code. Composite data should be deconstructed into primitives immediately. Keeping that type of 'discipline' in development ensures that there are far less bugs to be found.

Data overloading is a related problem. The programmer essentially combines two separate and distinct variables into a single one. Most often we find this happening with calculated conditions that are lying around in the code, such as user-defined options or internal flags. The original definition means something specific, but over time more and more code gets attached to a conditional variable, but with very different meanings. The code breaks when a fix is applied that fixes one usage for the variable, but breaks it for the other one.

The fix for this type of problem is really easy, just break the variable into two distinct variables, updating the correct data type for each. While it is a hard problem to explain, it is a common one in coding and an easy one to fix. Programmers tend towards saving a bit of space from time-to-time, so they frequently overload the meaning of existing variables, instead of creating new ones.

Primitive data is easier to see in a system because all you need do is follow the data around and see if it is decomposed any further. If it is, the 'parsing' should be pushed back as early as possible in the code, and the 're-assembly' as late as possible. If all of the data is stored at the same depth within the code, copies of it are not necessary. In many systems, there can be dozens of copies of the same data as it winds it way through the logic, buffers and caching. This redundant data is dangerous, costly and mostly avoidable.

SUMMARY

There are a couple of very important points spawned by this discussion.

The first of which is that as software developers we should endeavour to pick primitives for all of our API function calls and all of our internal data. We do this because it make the code a) easier, b) more memorable and c) less complex.

For functions, we need to realize that everything we write is an interface and the 'users' are our fellow programmers. In building a system, every function call belongs to some interface. Beyond just good design and simpler code we should also have empathy for those poor fellows stuck in the same position as ourselves. Nobody wants to have to memorize 200 convoluted function calls when 20 would do.

The same is true for data. Systems that are constantly breaking up data, then reassembling it, waste huge amounts of resources and are inherently convoluted. It is not a complicated amount of refactoring to push 'parsing' to be earlier and 'reconstruction' to be later.

If we stick to primitives, it simplifies the code and keeps it closer to optimal; we don't end up with a lot of convoluted data manipulations that aren't necessary. We really want to decompose things in the smallest consistent bits because that is the most expressive and least complex approach towards building.

For functions we also want to 'implement' the 'complete' API at any layer in the system, whether or not we think we are going to use it right way. Ignoring the 'Speculative Generality' code smell issue, whenever we implement a layer for ourselves or other programmers to use, that layer is an interface. The Speculative Generality smell is about not spending effort building something because you are just speculating that someone might use it. That is different than only doing a 'half' job on an interface. At some point, always, if half the code exists, someone will need the other 'half'. That is not speculating, that is fact. It is about finishing off what you started. Where you can add something, you should be able to delete it. If you can edit it, you can save it or create a new one. Half implemented tools are horrible.

The final point is that as consumers of other's code, we should demand that they provide to us the simplest possible code to get the job done. It is not uncommon to see very large and overly complex APIs full of overlapping, inconsistent poorly thought-out functions. These types of interfaces require considerable extra effort in order to utilize them. Not only is effort wasted in development, but it is massively wasted in utilizing this type of code. We need to be more vocal in not accepting this type of mess. If you build on a dependency that is convoluted now, how much worse will it get in the future? Thanks to 'backward compatibility' bad things no longer get removed from code, they just fester and grow worse over time. If we accept a mess now, it will only get worse. Programmers need to stop being fooled by shiny new technology; it is often only shiny because of excessive marketing.

The core thing to remember is that there is really no technical reason for us to have to frame our solutions in non-primitive constructs. And each and every time we do, we have just added artificial complexity to the system. It is always possible to get to an underlying set of primitives. It may take some work to 'find' or 'fix' the code, but the 'value' of that work is beyond doubt.

Abstraction and Encapsulation

Some of our most fundamental programming concepts are steeped in confusion. Because software development is such a young discipline, we often question and redefine many of the basic definitions to suit our personal understandings.

While this is a problem, it is also to be expected. Terms and definitions are the building blocks on which we build all of the higher concepts, if these are weak or incorrect they can lead us in circles long before we realize it is too late. Thinking in depth about their inner meanings is important. Older disciplines have no doubt survived much more discourse on their base topics; we are really only at the beginning of this stage. Other than what is mathematically rigorous, we should absolutely question our assumptions, it is the only way for us to expose our biases and grow. So much of what we take for fact is not nearly as rigid as we would like to believe.

For this posting, I'll bounce around a couple of basic terms that I've been interested in lately. There are no doubt official definitions somewhere, but if you'll forgive my hubris, I want to stick with those definitions that have come from my gut, driven by my experiences in the pursuit of software. When we reconcile experience and theory, we stand on the verge of true understanding.

A SIMPLE DEFINITION OF ENCAPSULATION

Encapsulation is a pivotal concept for software development. Its many definitions range from simple information hiding all of the way up to encompassing a style of programming. For me, the key issue is that encapsulation isolates and controls the details. It is how it relates back to complexity that is truly important. In a couple of my previous earlier posts I clarified my perspective of it:

http://theprogrammersparadox.blogspot.com/2007/11/art-of-encapsulation.html

and I added a few examples of what I think 'partial' encapsulation means and why it is such a big problem:

http://theprogrammersparadox.blogspot.com/2007/12/pedantically-speaking.html

These two posts define encapsulation as a way of breaking off parts of the system and hiding them away. The critical concept is that encapsulation is a way to manage complexity and remove it from the higher levels of the project. Controlling software development is really about controlling complexity growth. If it overwhelms us, the likelihood is failure. Partial encapsulation, which is a common problem, allows the details to peculate upwards which essentially undoes all of the benefits of encapsulation.

A SIMPLE DEFINITION OF ABSTRACTION

It may have just been a poor-quality dictionary, but when I looked up the definition of 'abstraction' it was self-referential: an abstraction is defined as an 'abstract' concept, idea or term. Where some people see this as a failure, I see it as a chance to work my own definition, at least with respect towards developing software. Following in the footsteps of all of those math textbooks that tortured me in school I'll leave a more generalized definition of the word as an exercise to the reader.

Although it is not necessary reading, in another earlier post I delved into the nature of 'simple':

http://theprogrammersparadox.blogspot.com/2007/12/nature-of-simple.html

this entry included a few ideas that might help to understand this some of the following craziness.

An abstraction is a simplification of some 'thing' down to its essential elements and qualities, that still -- under the current circumstances -- uniquely defines it relative to itself and any other 'thing' within the same abstraction. If two 'things' are separated, the two things when abstracted are still separated. Mathematically most abstractions involve isomorphic mappings (unique in both directions: one-to-one and onto) onto a simpler more generalized space, but if the mapping is not isomorphic, than any of the 'collisions' (onto, but not one-to-one?) must be unimportant. Thus for a non-isomorphic abstraction, it is workable if and only if it is projected onto a space that is simplified with respect towards a set of non-critical variables. If not, then it is no longer a valid abstraction for the given data. Other than collisions, if anything is not covered by the abstraction (not one-to-one?), then it too, at least in its present form is not a valid abstraction.

Lets get a little more abstract.

We all know, in our own ways that software is not real; it is not concrete. In that it lives encased in a mathematical existence, it has the capacity for things not tied to the real world, like perfection or a lack of the effects of entropy, for example. All software does is manipulate data, and it is there that we need to focus: a piece of data in a computer is an abstract representation of that data in the real world. It is nothing more than a shadow of the 'thing' onto some number of bits of 'information' that represent it to us. If I have an entry in the computer for me as a user, 'I' am not in the computer, but some information 'about' me is. And, that information is only a subset of all there is to know about me, a mere fraction of the complete set of information in the real world that is essentially infinite.

So all of our data within the computer is nothing more than an abstract representation of something in reality. Well, it could actually be several steps back from reality, it may be an abstract representation of a summary of abstract representations of some things in reality, or something convoluted like that. But we can ignore that for now.

Getting less meta-physical, the data in the computer is a 'placeholder' for some data in reality. The 'captured' relationships between pieces of data in the computer are supposed to mirror the 'important' relationships between the 'things' in reality. In the real world for any given piece of data, there is an 'infinite' number of relationships with other data, but the computer can only capture a discrete number of these relationships. It is entirely finite at any instance in time. We can simulate active infinite relationships, but only through the expenditure of computational power, thus allowing our 'time' dimension to be appear infinite. We can also simulate the infinite handling of things symbolically, but again this comes only through computational power.

It is perhaps an odd way of looking at the overall landscape, but this perspective is important in being able to explore some various points.

A FEW ESSENTIAL POINTS

If we step back and see the data in the software as a placeholder for stuff in reality, that leads us to surmise:

1. Everything in the computer is by virtue of its existence, abstract. Nothing is concrete, and all of this abstract stuff lies entirely in the domain of mathematics. This is why the essence of software development -- Computer Science -- is in some schools considered to be a branch of mathematics. It absolutely is. It is applied, to be sure, but it is all mathematics even if we think we are dealing with real world data. Software inhibits a bizarrely mathematical world, dragged down only by the reality of our problems.

2. Because everything is already an abstraction, there is always at least one abstraction that can contain the current abstraction without losing some essential virtual characteristic. The trivial case of course is every abstraction can be abstracted by itself. This is seemingly useless, except that it leads to the next item.

3. Every 'real-world' based abstraction has at least one complete higher-level abstraction. At very least some type of summary. Most real-world based abstractions have many many higher-level abstractions; often diverging on very different paths. At very least, they have the taxonomy for their specific category of information. Many have multiple taxonomies in which they are situated. For example, poodle is a type of dog, which is a type of mammal, which is a type of animal, etc. But poodle is also a type of pet, which is an animal with a specific relationship towards man, which are sentient beings, etc. Multiple languages act as other alternative taxonomies. Taxonomies are bounded by the messiness of mankind but there are generally a large number of mostly undisputed taxonomies for all things in reality. In fact, there is one for everything we know about, at least for everything we can tell each other about. Everything has a name or at very least a unique description.

4. The abstraction of the data is fundamentally a mathematical concept, which is not the same as the 'implementation' of the abstraction for a specific piece of software. Instantiating an abstraction into a specific computer based language is an implementation, which while based on the abstraction, it is not the same as the abstraction. The abstraction for instance could be correct, while the implementation might not be. One is fundamentally an idea, while the other is a thing. More concretely, the implementation is a physical thing existing at least as a series of magnetic impulses on a disk, if not another similar series of impulses loaded into a couple of silicon chips. The ideas behind software are abstract, but once written software is not. It has a physical manifestation; tiny but still physical.

5. An abstraction as a mathematical concept is not related to encapsulation. Encapsulation is a black box that contains some piece of complexity, encapsulating it away from the rest of the system. Whether or not the encapsulation is abstract, or even which implementation is used inside of the black box, is entirely independent. An implementation may encapsulate an abstraction, but it is more likely that the abstraction itself is the interface towards the encapsulated implementation, or something like that. Keeping abstraction and encapsulation separated is important because both have very different effects on software development.

6. An abstraction fits over an underlying 'data' abstraction completely. If it does not, then while it still may be an abstraction, it is not necessarily related to the underlying data one. E.g. if 80% of the abstraction is correct, then it is a badly fitting abstraction. Just because it partially fits, does not make it a good abstraction. However, multiple simple abstractions can be combined together to form an abstraction that fits entirely over another one, even if the individual ones do not. This new composite abstraction is more or less complete, if it contains no holes, but that doesn't necessarily mean that it isn't 'ugly'.

Together these points about abstractions, encapsulation and implementations have some interesting properties that often seem to be mis-understood. Consider these points, as we look at two related essays and explore some of the base assumptions.

THE LAW OF LEAKY ABSTRACTIONS

In his essay, Joel Spolsky defines abstractions, says they can leak and then defines a law of leaky abstractions: http://www.joelonsoftware.com/articles/LeakyAbstractions.html

While his problems are real, one cannot blame an abstraction for the fact that a) it doesn't fit the underlying problem, or b) the implementation is bad.

An abstract representation is one that fits over the underlying details. So if it only fits over 80%, then it is a poor abstraction. If the programmer doesn't encapsulate the details away from his users, then he is a poor programmer. Often we focus so hard on the code we build that we forget about the people using it.

In his essay Joel says that TCP/IP is an example of a leaky abstraction because the implementation chooses to allow the packet connection to timeout at some point. It could have just as easily never timed out, and then delivered the data on an eventual reconnect. There are publish/subscribe infrastructures that do exactly that, but they do require some interim holding storage that has the potential of overflowing at some point. Because of that, the choice to time-out is a resource vs. timing trade-off that is related to a generalized implementation of TCP. The idea of implementing a reliable protocol over an unreliable one is still a reasonable abstraction, regardless of which trade-offs must be made for an implementation to actually exist at some point.

In another example he says iterating memory in a higher level language leaks through because the order in which the programmer iterates through the array affects the performance. But it is possible for the iteration code to be examined by the software at compile time or even runtime and the 'direction' changed to that of the fastest approach. Some Existing languages such as APL do similar things. While this is not commonly done as an optimization by many languages, that does not imply that the abstraction of the language over assembler is any less valid. The implementation allows this detail through, not the abstraction. The abstraction just creates a higher level syntax to allow for more complicated instructions to be inputted with a smaller number of base primitives through a slightly different idiom.

I don't think abstractions can leak, although some are clearly poorly fitting. Everything has at least one valid abstraction that fits entirely over it. For all but the most esoteric applications, there is a huge number of potential abstractions that could be used. Just because we so frequently choose to jam the wrong abstractions over the wrong data doesn't mean that abstractions are the cause of the problem. They are actually the opposite. Our increases in sophistication come from our discovery and usage of ever increasing complex abstractions. Blaming a mathematical tool for the ugliness of our real world implementations is just not fair.

However, not to lose Joel's main point, there is an awful lot of 'complexity' bubbling up from the lower layers of our software. Every time we build on top of this stuff, we become subject to more and more of these potential problems. Getting back to my earlier writings, I think the real underlying cause is that the implementation is only partially encapsulating the underlying details. Often this is because the programmers want to give the users of their stuff some larger degree of freedom for the way they can utilize the code. Once the holes are open, the details 'leak' through to the higher levels and cause all sorts of fun and wonderful complexity ripples in the dependent code or usage. Because of this, I think Joel's law should exist, but be renamed to: "the law of leaky partial encapsulations". It is not as pretty a name, but, I think it is more accurate.


DIVING INTO AN ABSTRACTION PILE:

Another interesting read is: http://www.ericsink.com/Abstraction_Pile.html

Although I think Eric Sink does a excellent and interesting job of digging into most of the layers existing for his computer software, I disagree somewhat with his three initial rules and some of his terminology. The various 'encapsulated' layers in the system he mentions often have abstractions, but again we need to keep abstractions and layers as separate concepts. Erik puts forth a few rules about abstractions 1) they contain bugs, 2) they reduce performance and 3) they increase complexity. Dealing with each point in turn:

1) Abstractions contain bugs. An abstraction is a mathematical concept, it is what it is. The implementation can easily contain bugs, either because it is a poor fit to the problem or because some of the underlying logic is actually broken. Abstractions fit or they don't. A badly fitting abstraction is not the abstraction's fault, it is the implementation. Being more specific though, all code, abstract or not contains bugs. While a specific sequence of steps done in a brute force manner may be easier to fix if there are bugs discovered, in general abstractions contain orders of magnitude less code and because the behavior is more generalized, the testing of the code is denser. As such, while abstract code is harder to fix, it is less likely to contain bugs as the code ages. Specifically for working tested abstract code, it will contain less bugs than some brute force version, but both will contain bugs. The denser usage and testing will bring problems to the surface much faster. Layering is another important way of breaking off complexity, but you can just as easily do it with very specific brute force code as you can do it with abstract code, it is just a way of slicing and dicing the problem into smaller pieces.

2) Abstractions reduce performance. We all recognize that an expert hand-coding some assembler can produce faster-tighter code than a C compiler can. That is not, I think a by-product of the abstraction that C provides as a language, but simply because the work hasn't been done on a C compiler to better optimized the final code, yet. We've seen tremendous speed increases in Java virtual machines, but primarily because the semantic and syntactic freedom of Java is way less than C, so it appears to be easier to think of new and exciting ways to detect specific circumstances that can easily be optimized. Nothing about a correctly fitting abstraction will inherently interfere with the performance characteristics that some clever programmer with a lot of time cannot detect and counter-balance. If that is not true then the abstraction has collapsed down some critical variables into an ambiguity, which was still the essence of the problem, so it is badly fitting by definition.

Also, in general a compiler can produce code that performs better than the 'average' assembler programmer. The level of expertise for most programming languages spans a huge margin and the average of that margin over time will produce lots of un-optimal code. In the simple cases most compilers, with their inherent consistency will exceed the abilities of an average (or possibly somewhat less) programmer over much of the code they produce. Even the sloppiest of programmers will be better sometimes, but probably not overall.

Getting even more detailed, in order to optimize any piece of code one needs to abstract it to begin with. If we still have just a rigid set of instructions, there is actually no way to optimize them, they are exactly what is needed to get the work done and nothing more. If we go to a higher level of abstraction, then we can start to move many of the 'computational' operations forward, so that we can reuse the calculations for the optimization. Optimizations often come from getting more abstract and then moving more calculations forward and reusing the values. Caching, for example can be viewed that way. Continue this generalization of the problem until at some point the processing can't be reduced any more. This is a core technique for manually optimizing code. Abstractions, rather then reducing performance, are the reasons why we actually can optimize code.

As for layering, the overhead from creating layers in a system is most often minimal in comparison with the work that the system is actually doing. Assuming that you don't go to far, slicing and dicing the code into segments generally pays for itself, and often allows the replacement of some of the generalized segments with optimized abstract variants.

Applying abstractions that allow for optimizations at various layer can cause massive boosts in performance. I once optimized a program that was initially running in 2 hours, down to 1/2 hour with less than half the amount of code. Not only that, but the abstractions allowed for a huge generalized set of things to be calculated instead of just the specific one, and building the code took half as many man-hours as the original. Less code, faster performance, faster development and applicable to many problems instead of just one. That is the true power of abstraction.

3) Abstraction increases complexity. The point of an abstraction is to simplify something, and by virtual of doing that it does not increase complexity *beyond* the actual cost of the understanding and implementing the abstraction. If we replace some long set of repetitive instructions with a generalized abstract set, we will dramatically reduce the overall amount of code in the system. While each line of new code in itself is more complicated, the reduction in size reduces the overall complexity. Not only that, but the newer denser code is easier to test and more likely to be consistent. For any reasonable fitting abstraction, if it is implemented correctly, the system will have less complexity relative to the size of the problems it is solving. Often it will have dramatically less complexity, although on a line-by-line basis it may have been increased.

Also, we can utilize layering and encapsulation to essentially 'hide' complexity in various lower levels of the system. For example, if we break off half of a system and hide it in a black box, then the remaining half is simpler then the original and the half in the black box is simpler, but overall the whole: half + half + box is a little more complicated. If the block box only partially encapsulates, then the system is indeed worse because all of the details are now visible again at the top level. While encapsulating a layer does add some additional complexity, it hugely decreases the 'localized' complexity which allows it to counter-balance the localized complexity increases from implementing an abstraction.

We could never write our current level of programs in assembler, the complexity would overwhelm us. It is the abstractions we have found that give us the ability to build higher and higher solutions to our ever growing problems. Through them we can manage the increases in complexity. Through layering, we can distribute it into smaller parcels. Even if it is disappointing sometimes, we cannot deny that the sophistication of today's systems is orders of magnitude above and beyond those written fifty years ago, progress is evident and abstraction is clearly a core reason. But while we have some neat abstractions, we have only scratched the surface of those that are actually possible.


WHY IS THIS IMPORTANT?

While it is often overlooked, Computer Science is firmly rooted in mathematics. In its purest sense, software exists in an entirely precise and nearly perfect mathematical world. As we fill in more and more placeholders for the real world, we often tie down our software with the vagaries and messiness of our daily existence. Chaos and disorder play in reality, finding their way into the code. But intrinsically there is no entropy for mathematical formulas, they will stand unaltered throughout the ages.

That software straddles the real world while running in a purely mathematical one is important in understanding how much disorder and complexity are necessary in our solutions. We are tied so closely to reality, that software -- unlike mathematics -- actually rusts if it is not continually updated: http://people.lulu.com/blogs/view_post.php?post_id=28963

What this means is that while we cannot undo or avoid any of the inherent complexity of the problem domain for the tools we are building, we can find more 'abstract' ways to represent the information we are manipulating which allows us more flexibility in how we create these tools. Brute forcing the problem of creating code causes programmers to iterate out every single step of the problem they wish to solve, an easy but dangerous approach. If we fall back on more abstract and general representations for our code, not only can we optimize the speed of handling the problem, but we can also optimize the speed of creating a tool to handle the problem. Massively reducing the amount of code we need for the solution comes directly from utilizing abstractions.

It is only in leveraging our work in developing software that we can hope to build the full range of tools possible for a computer. If the alternative is pounding out every line of code that we need or will ever want, then our computer tools will not live up to their full potential for hundreds of years at least. We need massive effort through brute force to create and maintain reams of fragile code.

Our real troubles in development lay with our implementations and encapsulation. We go thorough a lot of work to hide the details, and then we go through a lot more to make them visible again. A small bit of insanity that we have been constantly plagued with.

Even more concerning, there have been many movements towards modelling the real world, or towards pounding out the systems with variations of brute force. In either case we are passing over the obvious: the data starts as an abstraction and we can take advantage of that in our implementations. As we generalize up the abstraction levels, we increase some of the localized complexity, but we leverage the code enough to make it worthwhile. With more generalized solutions we can solve larger problem domains with only marginally larger solutions. Abstraction is the key to leveraging our work.

Abstractions are a pure mathematical concept. They exist for all things that can be computerized and we've only really seen the tip of the iceberg as far as abstractions are concerned. It is a huge space, and there are many millions more 'abstract' ways of solving our current problems than are currently realized. Abstractions are the closest thing to a silver bullet for software development that we will ever get. We should take advantage of them wherever possible.