Architecture

Writing a small program, a few thousand lines, is not particularly difficult once you've mastered logic and algorithms. You focus on a very specific part of the problem, work out the details and then pound it into shape. As the size of the program grows you might keep on with the same approach; decompose the problem into smaller ones, then dump each answer somewhere arbitrary in the code base. At some point however, the amount of code that is built up begins to become the problem in itself. 

With a small program, if you make some sloppy mistakes it is fairly easy to find and fix. A couple of days at most. So the code can be disorganized. It can be highly redundant. Each instance of the same solution can have its own unique approach. You can get away with a lot of really bad practices. It can be a mess.

As the code base gets larger, the cost of changing it grows -- probably exponential -- until it is a huge task. Somewhere around the medium range, say 40,000 to 60,000 lines of code, just dumping in code anywhere builds up enough disorganization that it hinders further development. It only gets worse as more code is added. A few hundred thousand disorganized lines is painful, millions of lines of that is totally unmanageable.

Disorganized code can make it very hard to correct problems. Finding a bug in a small program isn't too hard once you've mastered debugging, but in a big code base organization is required to let you quickly jump to the right area. Without organization, it could take weeks, months or even years to find the problem, you have to search everywhere. 

When there is enough functionality and enough complexity, fixing the problems is a mess too. It starts to resemble stuffing straw into an old sack. You stuff it in one hole, only to watch it fall out from others. The changes cause unexpected side effects, which cause more changes, then more side-effects, etc. It becomes an endless and mostly hopeless task made worse by impending deadlines. 

A big mess can also be hard to extend. You don't want to disturb the existing functionality, so you copy and paste bits of code from all over into a new location, but the new redundacies will start to contridict each other quickly causing their own special bugs. Instead of making things better, you are probably just making them worse.

Big disorganized code bases are unstable and suffer from all sorts of horrible operational issues and usually have annoying interface problems, not to mention endless security issues. In most cases they look as ugly from the outside as they do internally. 

As the size of any system grows, the organization of the code quickly becomes the most significant problem in the development. Fortunately there is a well-known solution.

Architecture, for software, is a combination of the underlying technology choices and the higher level organization that encapsulates the many peices of a big system from each other. Mostly, due to resource limitations, it is arrived at through a long-term vision that is then mediated via compromises and politics. It isn't necessary for small programs, but from medium to large and all the way through to huge programs, it becomes increasingly significant to the stability, quality and extendability of the system. 

A good architecture draws 'lines' through the code, providing the high level encapsulation between components. It makes it easy to attribute bugs to specific parts and it allows one to place a scope on the impact of any code changes. As well it defines exactly where common extentions should be made to the code and lays out a direction for how to add brand new peices. 

An architecture can be documented, but if the lines are clear and consistent in the code base an experienced programmer can pick it up directly from reading the source. It might help to understand why things are organized in a particular manner, but not knowing that doesn't prevent someone from following the pattern of the existing implementation. 

One way to know there is an good architecture in place is to guess where some existing functionality should be, then verify that it is actually there and only there. If that is true for most of the functionality, then it is well-organized in a manner that can be understood.

Besides all its other qualities, a good architecture makes it easier for refactoring and for code reuse. If you need to make changes there are a limited number of places to change, and they are all in a consistent location close together. This allows you to either enhance the behaviour across the system, or route new functionality through one single interface. This of course provides a strategy for building a massive system over years and years through an endless huge number of versions. Each new iteration usually involves first refactoring things, then extending them with consitent organization. Done well, the first part can even be non-destructive and quickly testable, saving time and increasing the confidence level for the new code. 

One frequent concern with extending an architecture in this manner is the possibility of creating too many intermediate layers, but again the depth of the layers is an organizational issue itself. And endless number is disorganized by definition, so this meta-problem needs similar treatment to the base organization. In fact, for millions of lines of code, this sort of upper level organizational issue repeats itself over and over again, getting a bit higher each time. Finding a reasonable multi-level organization for millions of anything is of course a non-trivial problem that lies at the heart of architecure.

Small programs don't need an architecture. It doesn't hurt, but it is not necessary. Big programs abolosutely must have one that is clean and consistent. Otherwise they become unmanageable balls of mud. Architecture is a different set of skills than programming, but an architect should know both. They should have a lot of experience with programming first. They also need to be able to grasp the larger context and help focus the work of the programmers to avoid re-implementations. Efficiency for ongoing development is driven directly by the organization of the code base. Extending a well-architected program is considerable faster than just tacking on stuff to a ball of mud. The coding may take longer, but all of the other costs are minimized. The cost of not having an architecture is a steep decline in development speed over the lifetime of a project. That generally shows as either lower productivity or lower quailty or both.

If you want to design and build big, complex systems than the only way to get them to that size and keep them moving forward is to have and enforce an architecture. Disorganization may seem faster to some people, but that's only a limited short-term perspective.

Best Practices

One significant problem in software development is not being able to end an argument by pointing to an official reference. Veteran developers acquire considerable knowledge about ‘best practices’ in their careers, but there is no authoritative source for all of this learning. There is no way to know whether a style, technique, approach, algorithm, etc. is well-known, or just a quirk of a very small number of programmers.

I have heard a wide range of different things referred to as best practices, so it’s not unusual to have someone claim that their eclectic practice is more widely adapted than it is. In a sense there is no ‘normal’ in programming, there is such a wide diversification of knowledge and approaches, but there are clearly ways of working that consistently produce better results. Over time we should be converging on a stronger understanding, rather than just continually retrying every possible permutation.

Our not having a standard base of knowledge makes it easier for people from outside the industry to make “claims” of understanding how to develop software. If for instance you can’t point to a reference that says there should be separate development, test and production environments, then it is really hard to talk people out of just using one environment and hacking at it directly. A newbie manager can easily dismiss 3 environments as being too costly and there is no way to convince them otherwise. No doubt it is possible get do everything all on the same machine, it’s just that the chaos is going to extract a serious toll in time and quality, but to people unfamiliar with software development issues like ‘quality’ find that they are not easily digestible.

Another example is that I’ve seen essentials like source code control set up in all manner of weird arrangements, yet most of these variations provide ‘less’ support than the technology can really offer. A well-organized repository not only helps synchronise multiple people, but it also provides insurance for existing releases. Replicating a bug in development is a huge step in being able to fix it, and basing that work on the certainty that the source code is identical between the different environments is crucial.

Schemas in relational databases are another classic area where people easily and often deviate from reasonable usage, and either claim their missteps as known or dismiss the idea that there is only a small window of reasonable ways to set up databases. If you use an RDBMS correctly it is a strong, stable technology. If you don’t, then it becomes a black hole of problems. A normalized schema is easily sharable between different systems, while a quirky one is implicitly tied to a very specific code base. It makes little sense to utilize a sharable resource in a way that isn’t sharable.

Documentation and design are two other areas where people often have very eclectic practices. Given the increasing time-pressures of the industry, there is a wide range of approaches happening out there that swing from ‘none’ to ‘way over the top’, with a lot of developers believing that one extreme or the other is best. Neither too much or too little documentation serves the development, and often documentation isn’t really the end-product, but just necessary steps in a long chain of work that eventually culminates in a version of the system. A complete lack of design is a reliable way to create a ball of mud, but overdoing it can burn resources and lead to serious over-engineering.

Extreme positions are common elsewhere in software as well. I’ve always figured that in their zeal to over-simplify, many people have settled on their own unique minimal subset of black and white rules, but often the underlying problems are really trade-offs that require subtle balancing instead. I’ll often see people crediting K.I.S.S (keep it simple stupid) as the basis for some over-the-top complexity that is clearly counter-productive. They become so focused on simplifying some small aspect of the problem that they lose sight that they’ve made everything else worse.

Since I’ve moved around a lot I’ve encountered a great variety of good and insane opinions about software development. I think it would be helpful if we could consolidate the best of the good ones into some single point of reference. A book would be best, but a wiki might serve better. One single point of reference that can be quoted as needed. No doubt there will be some contradictions, but we should be able to categorize the different practices by family and history.

We do have to be concerned that software development is often hostage to what amounts to pop culture these days. New “trendy” ideas get injected, and it often takes time before people realize that they are essentially defective. My favorite example was Hungarian notation, which has hopefully vanished from most work by now. We need to distinguish between established best practices and upcoming ‘popular’ practices. The former have been around for a long time and have earned their respect. The latter may make it to ‘best’ someday, but they’re still so young that it is really hard to tell yet (and I think more of these new practices are deemed ineffective then promoted to ‘best’ status).

What would definitely help in software development is to be able to sit down with management or rogue programmers and be able to stop a wayward discussion early with a statement like “storing all of the fields in the database as text blobs is not considered by X to be a best practice..., so we’re not going to continue doing it that way”. With that ability, we’d at least be able to look at a code base or an existing project and get some idea of conformity. I would not expect everyone to build things the same way, but rather this would show up projects that deviated way too far to the extremes (and because of that are very likely to fail). After decades, I think it’s time to bring more of what we know together into a usable reference.

Architecture

When and why do you need an architecture?

If you are going to build something small or even medium in size, you can just wing it. Small stuff requires little architecture and medium systems most often crystallize along some crude architectural lines without much intervention. If you are going to build something large and you start without an architecture, what you will wind up with is a large series of disconnected and/or overlapping pieces that are impossible to stabilize. The project will die.

So what is an architecture?

In its most simplest explanation it is the overall organization of the underlying sections of code. For something to be ‘organized’ that means that there are a series of rules which categorize a specific layer to a ‘reasonable’ number of sub-pieces. Most often I prefer to refer to these as ‘lines’ and to leave the number of layers to be relative to the problem. An unreasonable number of categories is somewhat dependant, but lower numbers like 3 or 5 are nice, while larger ones like 20 or 30 are problematic.

How does one describe an architecture?

Like any good blueprint, an architectural description is a top-down listing of the lines and the layers. But since there are essentially two dimensions to an architecture there need to be two main roots to the design. The first is the underlying data in the system, while the second is the way the code is constructed to manipulate this data.

A top-down view of the data is essentially a hierarchical structure, starting with the major entities and gradually breaking them down into the details. The overall organization however needs be contained within a reasonable number of categories at each layer, so the major entities are most likely collected into a smaller set of categories at an abstract layer. That ‘organizational’ constraint flows all of the way down to the details. The data itself often cross-references other entities, but it does so internally (as data), thus it often be arranged in a hierarchy.

For the code the situation is similar. A hierarchical structure of categorizations forms the basis, however code is a little more difficult. To avoid redundancies many well-defined parts of the system will be shared at many layers, so here a hierarchy really fails to capture the expressive requirements of a well-designed system. Thus, although there is a root (a place to start), the lines in the code form a graph of categorizations, although these should still be presented as well-organized layers (unwinding this is part of the challenge).

So overall a well-documented architecture has two main sections that each descend down into the details. At each layer, there are a reasonable number of sub-pieces that are either specific or abstract (if necessary to be reasonable). A group of programmers should be able to walk their way through both parts of the document and use all of the contained information to solve all of the final coding-level problems.

If all is going according to plan, they still have significant problems to work on but all of the external decisions have been resolved and they understand how their individual pieces will interact with each other.

How does one create an architecture?

A software system needs a well-defined problem to solve. The underlying details, driven by this problem, come from analysis of the solution, which identifies the required data and the algorithms to operate on the data. There may also be analysis into the operation environment for the system and possibly the development environment as well. All of these details need to be sorted and arranged, then finally organized into layers. Although the final architecture is top-down, assembling the pieces in a bottom-up manner is often simpler and produces less redundancies. Generalization and abstraction are key to reducing the effort and getting it presentable. The architecture is ready when it’s depth matches the programming team’s capabilities.

Creatively Speaking

Somewhere in my youth, I came into possession of a very simplified model of the human brain. I'm not sure from where, it was so long ago its roots were lost in time.

The idea is simple, the brain consist of three, and only three things. First is memory. We can memorize stuff. By that, I mean that we take some fact, associate it with a number of keys and then, on demand, we can return the fact as needed. It is committed to memory. It becomes a piece of knowledge, something we know. I'll quietly ignore long-term vs. short-term memory for this model.

The next thing is understanding. That is, we understand something. For this, I mean that we can take some set of facts that we learned, and at the appropriate time use them for some piece of higher reasoning. If you've just memorized a fact, that's nice, but until you understand it, you can't really take advantage of it. In some sense, understanding is really just how that set of facts is linked to other sets of facts that can in some way drive or change your behavior. You 'know' what an stove burner is (fact), and you 'understand' that once its turned on, it is hot and will burn you.

The very last thing is creativity. This, conceptually is a little more complicated. If a fact is a thing you remember, and understanding is how it is linked, creativity in some offbeat way is how you can create 'dynamic' linkages between non-related facts or understandings. You might have a pencil for instance, and you know how to use it to draw shapes and textures, but at the time you are working you're feeling blue, which is some intangible emotion. If you want to use the pencil and your ability to draw to express your emotion state by drawing a dark moody image of yourself, you are taking your pencil, and your self-view, both facts, applying your understanding of drawing and then 'combining' that with your current emotional state to 'create' something that on its own would not come into existence for anyone else. You are creatively expressing your emotions through a self-portrait with a moody overtone. The work is not reproducible.

Creativity then is this ability to combine two entirely unrelated things, in arbitrary, random, or mostly irrational ways. The base things are either facts or understandings.


CONSEQUENCES

A consequence of this simplified model is that the abilities of a human on one level are quite simple. We build up knowledge -- linking it to other bits -- in ways that we find useful. Indirectly, as if it were a computer program run amok, one can see creativity as some type of inherent 'flaw' in our brains. After all, connecting two things that "shouldn't" be connected is not rational or deterministic. It's like a computer programming randomly changing the data, it is outside of the norm. The brain shorts out, or something.

From a natural point of view, this model allows one to see human progress often as this long history of people gradually creating new ideas and memes by either observing them in nature or randomly combining together un-related bits. In a direct sense, all of our of knowledge beyond what is quantifiable, is accidental. We stumble into discovering it.

That's a little hard for many people to swallow, so it likely shows that this model is far to simplified to be absolutely correct, but still quite interesting. To get to the really high concepts, they have to be based on lower ones, that somewhere down the way have to be based on observable things, or creative leaps.

I like this model because it provides nice tidy explanations for many things. For instance, why somebody with 'photographic' memory does so well in their early university courses, but then has trouble later on; they never really understood the original stuff, just memorized it. The latter courses require understanding as well as memorization. You can't build on a foundation that you don't have.

Intelligence is not just our ability to remember facts, understanding and creativity play a huge role as well. So that means a brilliant person is one, then who has a lot of facts, and understands them well. While a genius is an intensely creative person, who many not actually sit on that big a reservoir of understandings or facts; but they can just combine what little they have into interesting new bits. There are a lot of brilliant people, while creativity, is a rarer thing.

This model harmonizes 'intelligence' and shows that one cannot easily compare their three values against other people's three values. With three dimensions, you cannot easily rank your intelligence. Often you can meet people who can quote tremendous amounts of knowledge, but seem to have trouble really putting the pieces together. Your typical absent minded professor may be incredible forgetful, but full of deep understandings. It is a great perspective on our abilities.


COMPUTER SCIENCE

Now, all of that being interesting, what does it have to do with computer science? I read a excellent write up recently of someone talking about how they believe that math is actually creative. It is know as Lockhart's Lament and it is at:

http://www.maa.org/devlin/LockhartsLament.pdf

This to me this is a hugely misunderstood characterization of what it means to be creative, and of what mathematics is, and why it is so special. In a way, relating it back towards creativity takes away from the true underlying awe and beauty of mathematics. It trivializes it.

In a very real sense, mathematics is an infinite set of axioms over an abstract space. It does not exist in our real world and because of its rigour, which could not exist either in this world, it is purely abstract. Math is sort of this pure crystallize abstract structure that we can perceive if we concentrate on it long enough. It exists, but without substance. A truly amazing thing.

As such, all math exists and has always existed. It always will, with or without us.

The branches, axioms, primitives etc. of the various different types of math are our expressions of the that abstract foundation. It is out there, right now. All of our existing math -- a drop in the overall bucket -- and all of the stuff we still do not know. Mathematicians set out on a quest to 'find' new mathematics. They find it, prove it is rigorous, and then explain it to the rest of us.

In that sense, mathematics as we know it, is a huge set of facts and a significant amount of understanding about those facts. In its abstractness, it is consistent, and contained. It is not contaminated by the real world, where entropy and chaos dominate. It exists in a pure state, perfect, rigorous and eternal.

Knowledge and understanding then easily define our existing view of mathematics. They exist, so there is little that is inherently creative in their existence. Learning mathematics is not a creative exercise. You study the existing works, memorize the facts and work on problems until the understanding of the facts kicks in. If you are being led down the garden path, then as exciting as it is, learning about some new set of axioms and how to use them really isn't your creative spark, is it?

If learning mathematics is not particularly creative, then the big question must be one about 'finding' the ideas initially; whether or not that is a really 'creative' endeavour?


DOWN ANOTHER HOLE

To digress just a little bit. Software development is in its very own way a form of expressing a set of instructions to a computer that in its abstractness and rigour are similar to the act of creating a mathematical proof. Expression, be it about what you had for lunch, the weather or some specific algorithm for searching is a form of communication. The underlying topic may change, but choosing English, French, C or Java for your expression is still a way of communicating something. For English, it is non-rigorous and it goes to another human being. For C, it is very rigorous and it goes to a computer, but if both are about algorithms, then they are just different ways of expressing the same underlying thing. We only choose to express vastly different things in the different mediums.

There is, of course, a question about the expression of intangible quantities. It is one of those that Richard Gabriel in his book "Patterns of Software" says architect Christopher Alexander called: "the quality without a name". We have to categorize the world into those things that are determinable, tangible in some way, and those that are not. Tangible usually means, some form of physical existence. For the intangible, mostly it comes to us, personally in some what that is spiritual to some degree. By that I do not evoke any higher meaning, other than to describe 'spiritual' as a mood that is specified by humans. It is one of our many emotions. In Gabriel's book he talks about the feelings one gets when they go into these huge architectural wonders, often churches, but not always. It moves people in some intangible way. That unnameable quality then to my mind is just another intangible human emotion. Perhaps awe, or something similar.

That, fortunately makes for a nice and tidy taxonomy, since I've often described fine art -- for example -- as having an embedded emotional context, which is what separates it from merely being graphic design. If it stirs you in some way, it is art. If it is just pretty, then it is only graphic design. The same is true for music, it moves you or it is 'pop' music.

What we can perceive with our intelligence is tangible, and what we can experience with our emotions is intangible.

And so, in my earlier example with the drawing, the self portrait combines two very distinct things, one quite tangible, the representation of a human -- with the other, which is intangible -- the emotion of feeling somewhat blue. The creative 'short' or spark that drives that is not natural to most people, so although afterwards we (some of us) can see what the artist was getting at, and are moved by it. Although left in the same circumstances, we -- not being that specific artist -- would not have been creative in that same way.


MATHEMATICS

Coming back to mathematics: we know that the math is out there, and that it is abstract, yet not intangible. It is a thing, and if you search hard enough, eventually you will come upon it. I am not saying that in any way it is easy, or that it would take humanity anything less than thousands of years of culture before they might stumble on a complexity concept like negative numbers, which is exactly what happened. It also took a long time for us to figure out zero, complex numbers, calculus, etc. Some humans clearly can get there faster, but the road that needs to be traveled needs to be traveled by all, collectively.

We could not have complex branches of mathematics for instance without having an understanding of negative numbers. The intuitive leap that one and only one person can make, can only be so far. We are limited by our life spans. The imaginary of standing on shoulders of giants is exactly correct if you realize that the giants themselves are only just other people standing on other people's shoulders. And on it goes.

We get left then with the understanding that finding new forms of mathematics is a search, and it is one that comes from a limited distance from our last existing searches. With each new generation we can cover new ground, but only just barely. In that, although for one human, they learn the existing knowledge and then the existing understandings, they can only start searching from that point on. To them it is an open un-explored field with hidden surprises, to humanity however, it is just part of an ever going permutation through all of the localized space relative to the last set of solved problems. So, in the end, the best that a mathematician can do -- in that sense -- is to find or get to their discovery before the hoard of people following them do. Speed is important, but so is being in the right place at the right time, as the field is constantly filling up with others overturning the same rocks and prodding the same bushes too. If you don't discover it, they eventually will.

All of this really comes back to whether or not the idea of someone prodding around in a big field looking for interesting bits of mathematics is in fact being creative or not. And, to some very large degree, if mathematicians are creative in what they find, then by virtual of their similarity, computer scientists -- at least for the very first time they are writing something really unknown -- are equally as creative.

As if you needed more twists, we still need to try and relate this situation back to something more physical.

The simplest mapping would be to look at a graphic artist. They are by virtual of their training able to produce things that are visually appealing. Yet, as we are awash in pretty graphic design, you know that their work is not stirring in any fashion.

It looks nice, but it doesn't not give rise to emotions. That's true because while they have taken their facts, and taken their understanding -- in this case abilities and skills at things like drawing or matching pleasing colors -- they have not crossed the gulf and gone on to combine these with some intangible emotional context or any other thing that is out of the ordinary. Their skills may be beyond what some in the 17th century would have been able to do -- appearing back then to be 'art' -- but here and now, in this time they are applying the things they learned in school to a specific end. Understanding, yes, creativity, probably not.


BACK AROUND AGAIN

So going back to our mathematicians, as they ponder and search throughout the field, are they really being creative, or are they not just applying the skills thy were taught towards the goal of identifying previously unknown bits of mathematics? There may, I admit be some truly creative spark there that helps particularly gifted mathematicians to get to a specific point in the field faster than others and allows them to identify their thing rapidly. How else, can you explain Einstein? Although his leaps where related to a specific science, not general abstract mathematics, the distance that he appears to have travelled vs. his contemporaries at the time seems to be enormous. But again, what he found was there plainly for all to see, if they wanted to, or were willing to look. Relativity always was, and always will be. You just have to want to see it. But still, the leap was big and it took twenty years for his fellows to catch up with him and get to that particular place.

And so, for me, I'd have to say that most of the leaps that are going in mathematics are not leaps, but a natural progression of steps. And that these steps are an expected consequence of the knowledge and understanding that come from studying the field and really understanding it. That sometimes, but very rarely, a few make creative'ish giant leaps, but in general people are just working towards the goals of their field. If any one famous mathematician had died early, it may have taken us longer to get to their understanding, but we would still get there. The results might be slightly different, possibly composed of a different set of primitives, but it will still be functionally equivalent. As mathematicians create proofs, so too do computer programmers create code. It is exceptionally less complex, and far more mundane, but still shares the same type of relationship.

The code exists, we just have to find that sequence of instructions that expresses the solution to the problem in the most elegant fashion. So, we have to ask again: is programming really creative? The answer would have to be rarely. Only at that moment where we are combining two unrelated things in a truly unique way are we actually allowing our brains to take creative leaps. Otherwise we are just applying our understanding of how to code.

If your just doing another day's worth of problem solving, then those are exactly the skills programmer's should have and were taught. Software design has creative moments, and often the design is left unspecified so that some of the effort is required by the programmers, but mostly the business domain, technical domain and best practices remove a significant amount of the degrees of freedom. The interfaces share the same creative tones as graphic design. Most of it is really the application of an understanding.

Is software development art? Never. There is virtually no place in software for the expression of intangible emotions, I can barely imagine what that might look like. Tools are just tools. But, to slightly confuse the issue, the content of the programming could be art. That is, you could create a masterpiece of art using code as the means of expression. Games designers and many of the early pioneers of tools like flash succeeded or came close.

But really, a tool that moves you to tears is somewhat counter-productive, yes? It would be difficult to be both stirring and functional. Although, I do have to admit that some big company software products have caused me to cry, but really, that is only due to dealing with their crap, and my inability to be able to find a better replacement. The code did not stir the emotions, the angst of dealing with it did.

The Essence of Analysis

In any venture if you do not start off on the right foot, your chances of success are dramatically diminished. So it is easy to understand that the farther away we are from our initial target, the longer it will take to get there. If we have no idea where we are even going, then we won't even know if we have ever arrived. Some things, just are.

Many software developers understand that design is a highly critical part of getting the software right. You can't just magically iterate your way into the perfect system without first understanding what the system is supposed to do. Formal, or on the back of napkin, developers need a target to work towards. However, it is often the case that even the best of all designs still cannot help if the even earlier stages are fatal. Design is not actually the first step.

The very first thing for any building is to have a reason to build it. A building without occupants, a book without readers, or a software product that does not solve any real problems are all examples of things that got lost in their very first stage: analysis.


SOME EARLIER THOUGHTS

A while back, I wrote down some ideas about what I thought were the essential development problems:

http://theprogrammersparadox.blogspot.com/2008/01/essential-develoment-problems.html

I identified four big issues a) perception, b) discipline, c) generalization and d) analysis. Of these, analysis is the one that I have least talked about, primarily because without resorting to domain specific examples -- generally proprietary in the software world -- talking about it is difficult.

But analysis is clearly one of the keys to getting the right thing built at the right time. Turning a list of functions into executing code takes effort, but turning a vague understanding of someone's process into a list of functions takes a genuine understanding. It is a very difficult, very overlooked and a very uncommon skill set.


BASIC DEFINITIONS

In analysis, only one group of people is really important: the user of the software. All of the other 'stakeholders' have some effect in helping or derailing the project, but your quality of work comes from only pleasing one group. For this entry I'll focus only on them, leaving the others as just background obstacles to be overcome.

All that software can do is to help a user manipulate a growing pile of data. Whether it is visualization, tracking or editing, software is incredibly simple; each function just helps play with the information in some way. The key to building good software, then is to obviously tie this back to something useful in the user's life. Software that helps solves specific problems is software that is desirable. It extends out the user's ability to manage their information efficiently and effectively. Software that violates that covenant just irritates it users. Realistically most software is a mixture of useful and irritating features. Sometimes on purpose, often not.

Without a doubt, the most common approach to software development is a combination of hanging around on an ivory tower and collectively guessing at the sorts of problems that the user might want solved. If this sounds haughty and aloof, it generally is, which is why a great deal of software lacks a considerable amount of empathy for its users. Missing empathy equates to sticky, hard to use features that beg to be rewritten. It doesn't need to be like that, but that's the industry norm.

If you stand back a bit, you can see that software solves two basic problems: one from the business domain, the other from the technical. The technical domain is the packaging that is required by the programmers to get the business solution to the right people. The users for example may want to keep track of their financial information; the database, GUI, programming language, etc. are all just technical solutions that help the user access the financial tracking capabilities that they need. From a user's perspective, they are essentially noise that has to be put up with, in order to accomplish their goals. User's don't, and shouldn't care about the technical issues.

Programmers, on the other hand, love solving technical problems which are easier and more straight-forward. As such they try to take those abilities learned from creating technical solutions and throw them at the business problems. Generally to very painful results. Technical problems are consistent, simple and rational. Business problems, at least from the perspective of the programmers are messy, irrational crazy things that make no sense. The type of problems that scare most developers. Trying to wrap the irrational in a simple consistent model is a hopeless task.

Because of its messiness most programmer's prefer working on technical issues. If you need any real proof for that, a quick examination of the OpenSource movement shows that for the most part, the projects -- particularly the successful ones -- have been entirely spawned around and dedicated to technical problems. Volunteer programmers want to tackle easy fun problems, so they are naturally drawn to the technical ones. Why tackle the painful stuff for free?

Not surprisingly, the bulk of the world's programming problems are not technical. They are domain specific, which is why commercial development and consulting are still significant industries, even in the face of so many programmings seemingly giving away their craft. Given the difference between the two types of problems and the obvious lack of appeal for the ugly business code, it is more than likely software development will continue to be a well-paying profession for some time.


CONSULT THE EXPERTS

A computer programmer is someone who can assemble a complex set of instructions intended to implement some functionality in a software program. A software developer, on the other hand, is someone who can analyse real world issues, isolate the problems and then conceive, design, implement, test and deploy a working solution to help with those problems. The difference is clearly the scope of the work being performed. There are lots of computer programmers, but very few real software developers. You need to master the technology first, and then grow your abilities to start tackling the really hard problems.

The users are the expert at understanding their own problems, but an experience software developer is the expert at turning that into some workable solution. Naturally because domain problems are extremely messy, deep and irrational, years of experience with a specific domain are important. You can't solve problems if you don't really understand them.

While the developers need to understand the domain in which they are working, they are not by definition the experts. They don't have to know it all, or are necessarily functional in it. They really only need to see it from the perspective of the computer software:

- the data, its structure, frequency, quality, etc.
- the processes, their timings, interactions, differences, etc.
- the players, their roles, responsibilities, expectations, etc.

A computer takes a deterministic static view of the data, so all that the software developer really needs to understand is that viewpoint. But, that viewpoint must, to an incredible level of detail, be understand really thoroughly. The success of software rests on its implementation of the details.


FINDING THE PROBLEMS

The first mistake most developers make is trying to solve the wrong problems. Computers can't change things, they can only act as extension or tool for the users. In that sense, any software development is an automation project, one that is geared towards a specific pile of data. Thinking your shifting their paradigm, rocking their world, or changing the way they work will ultimately fall flat. At very most, someone farther upstream might be able to eliminate some lower-down position, with help from the computer, but at the very end, it all needs to be anchored back to a person, and they, at least in their capacity of using the software, aren't going to be pure management (which is a reporting only position).

So, software problems, are simple automation projects that involve building up piles of data. That leads to a fundamental property:

- data has to be captured or entered into the software.

It sounds silly, but in large projects people often fail to trace the data back to its source, a simple exercise that easily reveals many of the worst detail problems with the foundations. If you trace back the data to its source, you generally have a broader understanding of the domain. Even in a big warehousing project, all of the data starts somewhere, be it an an interface for service people, some type of capture scanning, or some feed from some other database. The other database had to gets its data from somewhere. So in general, most data was entered directly from manual forms on a screen. If you dig deep enough you'll find the real source, and thus the real problems. For each piece of data, you have to know the quality and frequency, that is vital.


UNDERSTANDING AND STRUCTURE ARE KEY

Bad assumptions, and misunderstanding data are very common problems that propagate through out development projects. I've speculated that there is essentially one great superstructure underneath in some of my blog entries (Age of Clarity, The Science of Information), but whether or not you believe that, getting a real working proper model of the underlying data is a hugely problematic issue. In many industries, even after thirty years, huge elements of the underlying data are frequently misunderstood. Guessing -- which is the most common approach to understanding data -- is dangerous, particularly if those poor initial pot-shots get permanently enshrined into the schema for reasons of backward compatibility.

So many software programs start off with a slightly-off underlying model, and are never able to truly fix their original problems. Seeing the danger of that, it is important to simplify, but not to miss any obvious chunks. Because of the limits of our current technology, it is extremely hard to 'uncomplicate' a schema once it is in production.

Clearly, the best most reasonable approach is to seek out the users on their own ground and have them explain their problems to you in their own language. This initial 'incomprehensible' babble, is only so, when you don't understand their domain. As such, any communication impedance mismatches, or user complaints, most often point to significant domain issues that you need to understand and resolve. Programmers are quick at dismissing their users as 'stupid and ugly', while software developers know to look deeper. We build for them, and if we don't take the time to see their point of view, that is reflected in our work. With understanding and experience, the things that you initially took for trivial turn out to have significant value.

There are of course, times when the users are just letting off steam, but unless you know that for sure -- not as an assumption -- then it is best to assume there is something tangible, even if it is slight, underneath that is worth considering.


THE USER IS NOT THE DATA

In that data has an underlying structure, that understanding and interaction is rarely what the user is after. For example, we want to interact with our computers in a desktop metaphor, while underneath they need to deal with files, folders and links, and under that: file formats, data structures, bytes and bits.

An incredibly common mistake, particularly when it comes to code generators is to assume that the user perspective, and the data perspective are identical. The data contains inherent ugliness that the computer should shield the user from. In that sense, in a classic system, the database schema will contain a level of detail that is completely uninteresting to the user. Their application summarizes, stores context, and essentially hides away a great deal of that detail. And, the user perspective is irrational, there is no formulaic way to tie it back to the data model. It will be similar, but it will always be very different in irrational ways.

Harmonizing that translation between the user perspective and the true data perspective is another great problem. Overtime people have understood that it was there, but all too often they have attributed it to the wrong causes, like assuming it was physical storage issues, or alternative viewpoints.

I guess for simplification reasons, we would really like to have one single consistent model for all of our data. And having one model that we could magically use to generate all of the code is the next logical conclusions. But, as many programmers know, there are a huge number of transformations happening in any well-written software. Now, we just do the work, and stick it were it is convenient, but the duality between the structure of the data, and the user view, points to an area where we really should be able to separate out the different models in the architecture and formalize (to some degree) the transformations back and forth between these different perspectives.

With further consideration, you can see this as a strong reason why many attempts at re-use fail so badly. The developers, in their quest for one overall model, mix the application-specific structure with the underlying real structure tying the resulting code absolutely to the one and only one application for which it mostly matches.

Beyond the technical problems of implementation, this has a big impact on analysis. The analyst needs to work with the users to see their perspective, but they also need to get down to the real underlying data perspective as well, something that the users may or may not be aware of. The user champions, then are often only really experts at half of the details. The analyst still needs to find and understand the other half. That is a huge problem, but manageable if you are aware of its existence.


CONSOLIDATING THE KNOWLEDGE

So, great you've successfully gone out and found a hundred or so really important 'functions' that would make your user's lives easier as they accumulate their growing pile of data. Randomly slapping these 'things' into a GUI is a bad idea. Sure it is easy to just extend the system by attaching a 'bag' onto the side of it with some disconnected functionality, but that is the type of analysis and design that makes all of us hate those 'big company' programmers.

If you've really come to understand your user's needs, then you should understand how to integrate that back into the existing solution without just gluing on more disconnected features. It is this consolidation of the feature set, that is often extremely difficult. But finding a way to accomplish this, not only makes for a better tool, it also significantly cuts down on testing. E.g. testing one slightly enhanced application is less work than testing two disjoint ones (although most companies fail to recognize this, and only increase their testing resources by a fraction of what is really needed, so they get two apps tested to a lessor degree than the original one).

Mostly, we design our tools to have some consistent overall metaphor, or style of working. Extending that, while maintaining its consistency is crucial. At times, the right answer may to actually throw away all of the old perspective and go up to the next level for a totally new one. Keeping everything simple and consistent, is extremely hard, and extremely dependent on the underlying domain and tool set.

Backward compatibility is important to some degree, but that should not be used as an excuse to not properly integrate functionality into an application. The tools should get easier to use as the coverage of the functionality grows, something that is clearly not happening with a lot of modern software.

Consolidation is as hard as the initial analysis, and is considerably more important. We've reached the stage were we can bang out some pretty sophisticated software products, but we have not gone over to being able to take a large amount of functionality and make it useful. This is obviously somewhere significant where Computer Science needs to grow up.


THE LAST WORDS

Bad, poor, or missing analysis is the start of a doomed project. You can't sit in a cubicle, miles away from your users and expect to be able to write useful software for them. Where you succeed with this approach, it is only and absolutely luck, and nothing else. It is easy to see why so much of our software is so defective.

In five years of constant programming, most people who stay at it will master to some degree the ability of structuring sets of instructions for the computer to execute. These people are computer programmers, and mostly should get some help from others to determine what the instruction set should accomplish.

Our industry standard is to recognize a five year career as being a senior, but while that might be true of someone attempting to master implementing specific functionality into a particular technology, that isn't even remotely true of someone trying to learn how to analyse a problem domain and use that information to build tools. There is far more beyond just belting out sets of instruction that is required to build things that are usable.

The secret then, if there is one to be known and understood, is to master the various techniques and technologies of software development, but never ever, even for a moment 'assume' that you have completely mastered the problem domain issues. Well, once you have re-written the same system three times, you're probably on your way towards mastering that specific issue, but jump problems, domains or industries and wam! your back to ground zero again. We have to be very careful to understand the difference between what we know, and what we assume we know. Building a simple system to assemble a big pile of data, does not make us really understand how that data works in the world around us.

If you can accept this, then you can prepare for a significant number of serious problems in the development leaking in from your lack of knowledge of the problem domain. Once you are past your own ego, you are a little more likely to accept the problems and their irrational nature for what they really are: the things that your software needs to solve to become better. Building software is easy, knowing what to build is the hard problem. All too often our industry wants to make the former sound harder and ignore the latter.

From scratch or extending a system, an experienced software developer makes a huge difference. Luck is the only thing that replaces a lack of analysis. The software industry's high failure rate comes from its inability to recognize the important of hiring developers with senior analytical skills to properly lead development projects. Analysis is a significant skill set. It is unique and not related to writing code. It is not just taking notes from user conversations and building up a list of requirements. It is not writing down stories, or guessing at how people might use things. It is a full in-depth understanding of the user's needs and the underlying data, and how to tie these two disparate things together into tool that is usable.

Even after nearly twenty years and many different software development domains, I am still often surprised by the 'depth' of complexity that some domain specific information is hiding. That is to say, that with each new problem I attempt to analyse, it is far deeper, far more complex and far more irrational than I'd ever really expect, and I've dealt with some pretty deep, complex and irrational problems already. But, I have certainly seen, again and again, that if you get into the user's perspective and build them a tool that really works, while capturing (but hiding) the true complexity of the data, you've come very close to really mastering software development.

If you get the analysis correct, then all you have to do it build it :-)

In Object-Orientation

"Familiarity breeds contempt" is a common cliche. It sums up an overall attitude, but for technology I don't think it is that simple.

I am certainly aware that the more we work with something, the more we come to really understand its weaknesses. To the same degree, we tend to overlook the most familiar flaws just because we don't want to admit their existence. We tend towards self-imposed blindness, right up to the instant before we replace the old with the new. It was perfect last week; this week it is legacy.

Software development has once again come to that state where we are searching for the next great thing. We are at a familiar set of cross-roads, one that will possibly take us to the next level.

In the past, as we have evolved, we have also dumped out "the baby with the bath-water", so to speak. We'd go two steps forward, and then one and three-quarters of a step back. Progress has been slow and difficult. Each new generation of programmers has rebuilt the same underlying foundations, often with only the slightest of improvements, and most of the old familiar problems.

To get around this, I think we need to be more honest about our technologies. Sure, they sometimes work and have great attributes, but ultimately we should not be afraid to explore their dark sides as well. All things have flaws, and technology often has more than most. In that, nothing should really be unassailable. If we close our minds, and pretend that it is perfect, we'll make very little progress, if any.

For this blog entry, I wanted to look at the Object-Oriented family of programming philosophies. They have a long and distinguished history, and have become a significant programming paradigm. Often, possibly because of their history, most younger programmers just assume that they work for everything, and that they are the only valid approach to programming. They clearly excel in some areas, but they are not as well rounded as most people want to believe.


EVOLUTION

Although it is hard to establish, the principles of Object-Oriented (OO) programming appear to be based around Abstract Data Types (ADT). The formalization of ADTs comes in and around the the genesis of the first Object-Oriented programming language Simula, in 1962. In particular, The Art of Computer Programming, by Donald Knuth which explores the then commonly known data structures, was also started in 1962 although the first volume wasn't published until 1968. Certainly, whatever the initial relationship between the two, they are very close, yet they are very different.

Abstract Data Types as a movement is about using data structures as the basis of the code. Around these data structures, the programmer writes a series of primitive, atomic functions that are restricted to just accessing the structure. This is very similar to Object-Orientation, except that in ADTs it is a "philosophy of implementation" -- completely language independent -- while in OO it is a fundamental abstraction buried deep within the programming language.

One can easily see that OO is the extension of the ADT ideas into the syntax and semantics of the programming languages. The main difference being that ADTs are a style of programming, while OO is a family of language abstractions.

Presumably, by embedding the ideas into the language, it makes it more difficult for the programmers to create unstructured spaghetti code. The language pushes the programmer towards writing better structured code, making it more awkward to do the wrong thing. The compiler or interpreter assists the programmer in preventing bad structure.

While ADTs are similar to OO -- you set up a data-type and then build some access functions to it -- at a higher level there is no specific philosophy. The philosophy only covers the low-level data structures, but absolutely nothing is said about the rest of the code. Practice, however is to use structured code to layout each of the high-level algorithms that are used to drive the functionality.

This means there is a huge difference between ADTs and OO. Outside of the data structures, you can fall back into pure procedural style programming in ADTs. The result is that ADT style programming looks similar Object-Oriented at the low-level, but is really structured as 'algorithms' at the high-level. A well-structured ADT program consists of a number of data-structures, the access functions and a series of higher-level algorithms that work the overall logic of the system. Since data drives most of the logic in most programs, the non-data parts of the code are usually control loops, interfaces to dispatch functionality or glue code to interfaces between different control loops. Which ever way, they can be coded in the simplest most obvious fashion, since there are no structural requirements.

That ambiguity in defining the style of the code at a high level in ADTs is very important. This means that there is a natural separation between the data and the algorithms, and they get coded slightly differently. But we'll get back to this later.


CODING SECRETS

Software development often hinges on simplifications, so it is not surprising that we really only want one consistent abstraction in our programming languages.

Because of this, we spend a lot of time arguing about which approach is better: a fixed language, that is strongly-typed or one that is loosely-typed. We spend a lot time arguing about syntax, and a lot of time arguing about the basic semantics. Mostly we spend a lot time arguing about whether or not the language should be flexible and loose, or restricted and strict. It is a central issue in most programming discussions.

If you come an impasse enough times, sooner or later you need to examine why you keep returning to the same spot. Truthfully, we work in two different levels with our implementations. At the higher level, the instructions are more important. At the low level it is the data.

At the low level we want to make sure the data that we are assembling is 'exactly' what we want.

When I was building a PDF rendering engine in Perl, for example, I added in an extra layer of complexity. The engine was designed to build up a complicated page as a data-structure and then traverse it, printing each element out to a file. This type of processing is really simple in a strongly typed language, but can get really messy in a loosely typed one. To get around Perl's loose semantics I wrapped each piece of data in a small hash table with an explicit type. With the addition of a couple of really simple access functions, this had the great quality of emulating a strongly typed syntax, both making the code easier to write, but also guaranteeing less errors. It was a very simple structuring that also made it really easy to extend the original code.

At the higher level we want to focus on the instructions and their order, data is not important. Batch languages and text processing tools are usually loosely typed because it makes more sense. The data is really minimally important, but the order of the functions is critical.

As another example, I was building a dynamic interface engine in Java. In this case, I wasn't really interested in what the data was, only that at the last moment it was getting converted into whatever data-type I needed for display or storage. This type of problem is usually trivial in a loosely-typed language, so in Java, I added in a class to 'partially' loosely type the data. Effectively, I was concerned with 'removing' via a single call, any of the attributes of the type of the data. It does not matter what the data is, only what it should be. Again it is a simple structuring, but one that effectively eliminated a huge amount of nearly redundant code and convoluted error handling.

With enough coding experience in different languages, you come to realize that being strict at a low-level is a great quality, but being flexible at the high level is also important. It is easy enough to get around the bias of the language, but it does count as some additional complexity that might prove confusing to other developers. It is no wonder that the various arguments about which type of language is superior are never conclusively put to rest, it is more than a simple trade-off. There is an inherent duality related to the depth of the programming. The arguments for and against, then are not really subjective as much as they are just trying to compare two completely different things.

Well-rounded implementations need both strong-typing and loose-typing. It matches how we see the solutions.


INTERNAL MODELS

Another fundamental problem we face, comes from not realizing that there are actually two different models of the solution at work in most pieces of software.

The way a user needs to work in the problem domain is not a simple transformation from the way the data is naturally structured. The two models often don't even have a simple one-to-one mapping, instead the user model is an internal abstraction that is convenient for the users, while the data model is a detail-oriented format that needs to be consistent and strictly checked.

A classic example is from a discussion between Jim Coplien and Bob Martin:

http://www.infoq.com/interviews/coplien-martin-tdd

Specifically Jim said:

"It's not like the savings account is some money sitting on the shelf on a bank somewhere, even though that is the user perspective, and you've just got to know that there are these relatively intricate structures in the foundations of a banking system to support the tax people and the actuaries and all these other folks, that you can't get to in an incremental way."

In that part of the conversation Jim talked about how we see our interaction with the bank as a saving account, but underneath due to regulatory concerns the real model is far more complex. This example does a great job of outlining the 'users' perspective of the software, as opposed to the underlying structure of the data. Depending on the usage, a bank's customers deal with the bank tellers operating the software. The customer's view of their accounts is the same one that the system's users -- the tellers -- need, even if underneath, the administrators and regulators in the bank have completely different perspectives on the data.

While these views need to be tied to each other, they need not be simple or even rational. The user's model of the system is from their own perspective, and they expect -- rightly so -- that the computer should simplify their view-point. The computer is a powerful tool, and one of its strengths is to be able to 'transform' the model into something simpler allowing the user to more easily interact with it, and then transform it back to something that can be stored and later mined.

It is a very common problem for many systems to have the software developers insist that there is only one 'true' way to look at the data. The way it needs to be structured in the database. It is not uncommon to see forms tools, generators or other coding philosophies that are built directly on the underlying data model.

This makes it possible to generate code and it cuts down on the required work, but rarely is the data model structured in the same way as the user needs. The results are expected. The user's can't internally map the data to their perspective, so the applications are extremely awkward.

A single model type of design works only for very simple applications where there is little mapping between the user model and the data model. Mostly that is sample applications and very simple functions. While there are some applications of this sort, most domain specific systems require a significant degree of complex mapping, often maintaining state, something the computer can and should be able to do.

The essence of this, is that the 'users' have a model that is independent of the 'data' model. E.g. The interface translates between the real underlying structure of the data, and the perspective that the user 'needs' to complete their work. More importantly the models are not one-to-one, or we would be able to build effective translators. This duality and inconsistent mapping is why code generators and forms tools don't work. You can't build a usable application from the data model, because it is only loosely tied to the user model.


OBJECT ORIENTED

So what does this have to do with Object-Oriented programming?

The structure of an Object-Oriented language makes it relatively easy to model the structure of the data in the system. At the lower data level, the semantics of the language allow it to be used to build complex systems. You create all the objects that are needed to model the data, or the user abstraction. Object-Oriented languages are at their height when they are encoding easily structured objects into the language.

The classic example is a GUI system where the objects in the system are mapped directly to the visual objects on the screen, that type of mapping means that when there are problems it is really easy to go back to the screen and find the errors. The key here, is that one-to-one link between the visual elements and the objects. The point of going to a lot of extra effort, is to make it easy to find and fix problems. You pay for the extra effort in structuring the OO code by getting reduced effort in debugging it.

At the higher level, all of the OO languages totally fall apart. We add mass amounts of the artificial complexity into the works just to be able to structure the problem in an Object-Oriented manner. Ultimately this makes the programs fragile and prone to breakage. Ideas such as inversion of control, and dependency injection are counter-intuitive to the actual coding problems. They become too many moving parts.

Now, instead of the problem being obvious, you may need to spend a significant amount of time in a debugger tracing through essentially arbitrarily structured objects. Constructs like Design Patterns help, but the essence of the implementation has still moved far away from the real underlying problem. The developer is quickly buried in a heap of technical complexity and abstractions.

If what you wanted was a simple control loop, a traversal or some hooks into a series of functions, it would be nice if the simplicity of the requirement actually matched the simplicity of the code.

Even more disconcerting, we must leave our primary language to deal with building and packaging problems. We use various other languages to really deal with the problems at the highest level, splitting the development problem domain into pieces. Fragmentation just adds more complexity.

Our build, package and deploy scripts depend on shell, ant or make. We could write the build mechanics in our higher level language, but writing that type of building code in Java for example, is so much more awkward than writing it in ant. If we see it that clearly for those higher-level problems, does it not seem as obvious for the higher-level in the rest of the system?

Another critical problem with the Object-Oriented model is the confusion between the user model, and data model and the persistent storage. We keep hoping that we can knock out a significant amount of the code, if can simplify this all down to one unique thing. The Object-Oriented approach pushes us towards only one single representation of the data in the system. A noble goal, but only if it works.

The problem is that in heading in that direction, we tend to get stingy with the changes. E.g the first version of the application gets built with one big internal model. It works, but the user's quickly find it awkward and start making changes. At different depths in the code, the two models start to appear, but they are so overlapped it is impossible to separate them. As integration concerns grow, because more people want access to the data, the persistence model starts to drift away as well.

Pretty soon there are three distinct ways of looking at the underlying business logic, but nobody has realized it. Instead the code gets patched over and over again, often toggling between leaning towards one model, and then back again towards another. Creating an endless amount of work. And allowing the inconsistencies tp create an endless amount of bugs.

If the architecture identifies and encapsulates the different models from each other, the overall system is more stable. In fact the code is far easier to write. The only big questions comes about with deciding the real differences between the user model and the data model. When a conflict is discovered, how does one actually know that the models should differ from each other or that both are wrong?


SUMMARY

Ironically, even though they are probably older and less popular, ADTs were better equipped for handling many implementation problems. Not because of what they are, but because of what they are not. Their main weakness was that they were not enforced by the language, so it was easy to ignore them. On the plus side, they solved the same low-level problem that Object-Oriented did, but allowed for better structuring at the higher level. It was just that it was up to the self-discipline of the programmer to implement it correctly.

I built a huge commercial system in Perl, which has rudimentary Object-Oriented support. At first I tried to utilize the OO features, but rapidly I fell back into a pure ADT style. The reason was that I found the code was way simpler. In OO we spend too much time 'jamming' the code into the 'right' way. That mismatch makes for fragile code with lots of bugs. If you want to increase your quality, you have to start with trying to get the 'computer' to do more work for you, and then you have to make it more obvious were the problems lay. If you can scan the code and see the bugs, it is far easier to work with.

We keep looking of the 'perfect' language. The one-size-fits-all idea that covers over both the high and low level aspects of programming. Not surprisingly, because they are essentially two different problems, we've not been able to find something that covers the problem entirely. Instead of admitting to our problems, we prefer to get caught up in endless arguments about which partial solution is better than the others.

The good qualities of ADTs ended up going into the Object-Oriented language design, but the cost of getting one consistent way of doing things was a paradigm where it is very easy to end up with very fragile high-level construction. Perhaps that's why C++ was so widely adopted. While it allowed OO design, it could still function as a non-OO language allowing a more natural way to express the solution. Used with caution, this allows programmers to avoid forcing the code into an artificial mechanism, not that anyprogrammers who have relied on this would ever admit it.

The lesson here I guess is that the good things that make Object-Oriented popular, are also the same things that make it convoluted to some degree. If we know we have this duality, then for the next language we design for the masses, we should account for this. It is not particularly complicated to go back to ADTs and use them to redefine another newer paradigm, this time making it different at the low and high levels. Our own need for one consistent approach seems to be driving our technologies into becoming excessively complicated. We simplify for the wrong variables.

We don't want to pick the 'right' way to do it anymore, we want to pick the one that means we are most likely to get the tool built. Is a single consistent type mechanism better than actually making it easier to get working code?

Fundamental Coding Issues

Everybody loves a short summary, one that can easily compress a complicated idea into a simple concept. They are hard to find, but always very useful.

In software development there are many complex issues surrounding our computer language 'code' and the various conventions we use for programming. There are lots of different attributes that a program can have beyond just working, but ultimately there are only a few things that easily define a great program. With that in mind, I find the following summary quite reasonable:

The secret to great code is to get the smallest program possible without resorting to being clever, while generalizing to as large a problem space as possible given the time constraints. Make all of the broad strokes explicit, while making all of the details implicit. Get this right, keep it clean and consistent, and you've got elegance.

It is simple, as it should be, but still the underlying parts require deeper explanation.

THE SMALLEST CODE BASE

Software developers don't like to admit it, but the amount of the code in a system is a hugely significant issue for all sorts of reasons.

Programming is a massive amount of work. If you look at the various parts of development: analysis, implementation, testing and deployment, the second one -- implementation, which is the act of writing the code to implement the solution -- doesn't appear all that large in the overall process. It is only one of four basic steps. However, despite appearances, the size of the code drives issues with all of the other work, including the analysis. The code is the anchor for everything else; bigger code means more work and bigger problems.

The problem starts with having to convert your understanding of the user's need into some set of interfaces. Complicated problems produce complicated interfaces which feed on themselves. It is far more work to add a new function to Microsoft Word, for example, then it is to add it to some really simple interface. The analysis doesn't just cover the problem space, it also includes how to adapt the proposed solution to a specific tool. Adding a function to a small tool is way less work than adding it into some huge existing monolith. Because of this,the analysis changes depending on the target infrastructure. More code means more work integrating any new functionality.

The size of the code itself causes its own problems. If your intended solution requires 150,000 lines of code you'd be looking at a small team of programmers for over a year. If your intended solution requires 1,000,000 lines of code it will take a huge team many years to build it. Once it is built, if there is a problem with the 150,000 lines of code, refactoring chunks of it is significant, but not dangerous to the project. With 1,000,000 lines you are committed to your code base for good or bad. It, like the Titanic is slow to turn, so if there are any obstacles along the way such as icebergs you are in serious trouble. Why use 1,000,000 lines if 150,000 will do?

With each line of code, you commit to something you've got to maintain throughout history. It requires work to keep it updated, it even requires work to delete it. The more code there is, the more work that is required to just figure out the available options. Big code bases are unwieldy beasts that are expensive and difficult to handle. Developers often try to get away with just bolting a bit more code onto the side, but that rather obvious tactic is limited and ugly.

In all code there are always latent unwanted behaviors (bugs), that includes mature code that has been running in production for years. These 'problems' sit there like little land mines waiting for unsuspecting people to pass by and trigger some functionality. Testing is the act of mine-sweeping, where the testers spend as much time as they have, trying to detect as many of these problems as possible. Most current test processes are horribly inefficient, they end up spending way too much time on the easy issues and way too little time on the hard ones. Once the time is up, the code is released in whatever state. So often you'll find that cord works well for its simple common usages, but becomes increasingly unstable the more you try to push its envelop. Not surprisingly, that matches the testing.

Testing follows some type of inverse square law, e.g. it likely takes four times as much effort to test twice as much code. By committing to a bigger code base you are committing to a huge increase in the amount of testing, or as more often the case, you are actually diminishing your existing testing by a significant degree. So often, the code base doubles but the testing resources remain the same, only now they are 25% as effective.

With the increase in testing requirements being ignored, most big software packages get more operational problems and issues. For all software, there is a considerable support cost even if it is just an installer, an operator, a simple help desk and some beepers for the programmers. For large commercial projects support can include an entire division of the company.

Running code is hugely expensive, most programmers don't seem to understand this. They just hand off their work and don't think about the consequences. There are the fixed costs, but also those driven because of behavioral problems. A particularly bad bug in a commercial setting could cost millions of dollars and really hurt the reputation of a company. Even in an in-house setting, the code going down could delay other significant events costing money or bad publicity. The more useful the code, the more expense the failures.

Bigger code 'will' fail more often. From experience it is clear that with twice as much code, comes twice as many bugs. Programmers have a nearly consistent rate of adding bugs to their code, that is mostly independent of the actual programmer or testing. Better programmers often have less bugs per line of code, but they still have them, and because they tend to work on the more complicated sections of the system or write more code, it is not uncommon for their bug count to be higher. It stands to reason, if there is twice as much code, there are at least twice as many bugs, so the odds of getting a bug is twice as likely.

Finally, bigger code bases also mean bigger algorithms, and a lot more of them. The more complex code is harder to use and has more bugs, but it also means more documentation work. Usually the 'rules' for any of the functional behavior of the system start to become really 'sophisticated'. Well, sophisticated isn't the right word, overcomplicated is probably more appropriate. Once the functionality bloats, it takes 'essays' to even explain the simplest usage of the system, the support costs go through the roof. A very simple program that does one thing that is strictly following the conventions in an obvious way, probably doesn't need any help or tutorials. Once you grow to do fancier tasks with lots of customization, then online help becomes a necessity. Open up the system to allow the users to adapt it to themselves then you need lots pf tutorials, courses and books. A whole industry springs forth from really complex software. Photoshop is the classic example, with a phenomenal secondary industry devoted to helping people utilize the internals by translating between language of the programmers, and the language of the users (such as "how do I remove red-eye from this photo?").


NOT CLEVER

Given all of the above problems with big code bases, programmers shouldn't rush out to compact their code into the tiniest possible pieces. Size is an issue, but readable code is more important. If you get too clever and really tightly pack your code it can be impossible to understand. Clever is bad. Very bad.

The obfuscated C code contest is a great example of really clever ways to dramatically alter or reduce the size of the code. While it is entertaining, in practice it is extremely dangerous. Clever code packs way too much complexity into too small a package. If we wrote things once and only once then never touched it, that would be fine, but the lifespan of code is huge. In its life, there are always periods were people make fast judgment calls on the functioning of the code. Code needs a constant amount of never-ending fixing and updating. Stress is a part of software because the amount of work is always larger than the resources. Clever code just sets itself up to cause problems in the future. It is yet another land mine waiting to happen, as if it were just a bug of some sort.

You get away with some clever trick of the language or some other weird way of getting your results, it will be easily missed by someone else. That code is dangerous, and definitely not elegant.

Since code is just complicated by its very nature. All problem spaces have huge amounts of complexity, but we really want to lay out each and every line of code in the simplest, most straight-forwardly reusable manner possible. That also includes, not commenting too much, as well as too little. Taking away the readability of code is always asking for trouble. If you can give it to someone with a very light coding background and they get 'it' immediately then it is probably close enough to elegant. Programming students, for example should be able to easily understand the nature and purpose of the code. You shouldn't need an extensive background to read good code, although you definitely need one to write it.

The formatting, names of the variables, comments, and all of the syntactic attributes are very important in producing something that is easy to read. In big languages such Perl, showing a tremendous amount of discipline in 'not' using some of the more esoteric features of the language is the industrial strength way of coding. In Java, not following weak design pattern and bean conventions will produce things that are more readable and easily understood. Obscuring the underlying functioning of the code by focusing on its structure, just makes it harder to understand. The magic comes from generalizing it, not from being clever.

Some of the 'movements' in programming over the years have been counter-productive because they get too obsessed about the 'right way' to realize that they are doing extreme damage to the 'readability' of their code base; a factor that is far more important than being right. If it is readable and consistent it can't be too far away from elegant.


LARGEST PROBLEM SPACE

When you are solving a problem, the range of possible solutions extends from pounding out each and every instruction using a 'brute-force' approach, to writing some extremely generalized configurable program to solve a huge number of similar problems. At the one end of this spectrum, using brute-force, there is a tremendous amount of work in typing and maintaining a very fragile and stiff code base. The solution is static and brittle, needing lots of fixes. While it is probably huge, each sub-section of it is very straight-forward as it is just a long sequence of instructions to follow, e.g. get this file, open it, read the contents, put them in this structure, add these numbers, do these manipulations, save them in this format, etc. Most computer languages provide some higher level of abstraction, so at least each instruction is millions of lines of assembler, but it is still rigid and explicit.

Adding more degrees of freedom to the instructions, generalizing or making it dynamic, means that the resulting code can be used to solve more and more similar problems. The size of the problem space opens up and with additional configuration information; the usage of the code becomes huge.

As we shift the dynamic behavior away from the static lines of code, we have to provide some 'meta-data' in order for the generalized version of the code to work on a specific problem. In a very philosophical sense, the underlying details must always exist, no matter how general the solution. When we generalize however, we shift the details from being statically embedded into the code in a fragile manner, to being, either implicit, or explicitly held in the meta-data effecting the behavior.

Like energy, the primary problem domain details* can neither be created nor destroyed. You just shift them around. They just get shifted from being explicitly embedded into the code to existing somewhere else, either implicitly or in the configuration data.

*We can and do create massive amounts of artificial complexity, which creates artificial details, which can be refactored out of the code. If you can delete details without altering the functionality, then it was clearly artificial.

Way way off to the very end of the spectrum, one might imagine some very complicated all-purpose general piece of code the can do everything, but funny enough, that 'code' exists and is the computer itself. It is the ultimate general solution.

Building and maintaining a system is a huge amount of work that is often underestimated. Usually wherever a specific software based tool can be used to solve a problem, there is an abundance of similar problems that also need to be solved. Programmers love to just bite off a piece and chew on that, ignoring the whole space, but it is far more effective to solve a collection of problems all at once. All it takes is the ability to step back and look at the specific problems in their larger context.

Which of the business rules seem to bend, and how many other problems in the company have the same feel to them? The larger the problem space, the cheaper the solution. If you have ten departments in a company that all need customized phone books, solving each one by itself is considerably more work than solving them all together. If you have one group that needs approvals for their documents, that problem spans a huge number of different groups. They will all benefit by a common solution.

Generalizing makes the solution cheaper, and the reduces the overall work. Also, strangely enough, generalized code is always smaller than brute force. That means that the long term costs of maintaining a code base are cheaper as well. The size issue on its own is significant enough that in a long run perspective it is always worth generalizing to bring down the size of the code, even if there are no available similar problems. Generalizing can reduce the code base enough to get it to fit in the available development window. The technique can be applied as a measure to control the costs of the project and build more efficiently.

It also helps in maintaining consistency. If there is one routine that is responsible for rendering multiple screens in a system, but virtue of its usage it enforces consistency.


TIME CONSTRAINTS

Everybody underestimates the amount of time it takes to build software. They also underestimate the amount of effort it takes to keep it running. Like an iceberg, only a small portion of the code is actually visible to the users so they see it as a rather small malleable thing that can be easily changed. Any significant set of functionality gets locked in by its own size and complexity.

If you make frequent quick changes to your code base you'll quickly find out that you are destabilizing it. The more fast hacks you add, the worse the code becomes. There is always an expensive trade-off to be made between getting it done quickly and getting it done correctly. Many projects make the wrong choices, although the damage often takes years before it fully shows. Getting out one good release is easy, getting one out time and time again is difficult.

Without a doubt, time is the biggest problem encountered in software. There is never enough. That means that any 'technique' that helps reduce the time taken to do some work is probably good if it helps both in the short run and the long run. Any technique that adds extra work is probably bad. We should always keep in mind that sometimes you need to add a little extra effort in the short run to save time in the long run.

For instance, keeping the code clean and neat makes it easy to add stuff later. A well-maintained project takes more time, but is worth it. Sloppy is the friend of complexity.

Optimization is always about doing something extra now, that results in a gain later because the results get re-used over and over. Clean code saves time in understanding and modifying it. A little bit of extra work and discipline that pay off. Design saves time from being lost later. With a solid design you can fit right into the code you need, you don't have to waste a lot of time wandering around trying to guess at what might work.

Time is an all important quantity in software development. Wasting it is a bad idea. Some of the newer programming development techniques seek to make 'games' out of programming. It is an immature way of trying to remove some of the natural tediousness away from coding. We want to build things and get to the end as fast as possible. Playing around is avoiding the issue.

If you want to build great tools, at times it will be painful, it comes with the territory. No job is 100% entertaining, they all have their down-sides, that is why it is called work and we have to be paid to do it. Nothing wrong with hobbyist programmer's playing games and competing, but its is not appropriate for the work place.

Testing is one area that people waste massive amounts of time without getting any additional benefit. Again, some of the newer testing techniques add extra work, but in exchange they claim to reduce the amount of bugs. If that works it is great, but you have to be sceptical of most claims. Component testing thoroughly is good, but if you cannot assure the interaction of the components with each other, then there is always some minimal level of final testing that still needs to be done.

It is immutable, and as such is not possible to be optimized away. If you must test the final product as a whole coherent piece, than you cannot skip those tests no matter how much work you have done on the sub-components. If you are not skipping the tests, then you are not saving any time. If you down-grade the final tests to be less, then you upgrade the risks of allowing problems through. Of course this is true as a basic principle: in-stream testing in any process is there to reduce the amount of bouncing around between states, but it does significantly bump up the amount of testing work without increasing the quality. If the bouncing between states is not significant, then reducing it doesn't add much extra value. It doesn't negate any of the final testing, which still needs to be done.

In same way that one cannot solve the halting problem, the amount of testing required to achieve 100% certainty that there are no bugs is 'infinite'. It is an easy proof, once you accept the possibility that there is some sequence of input that causes the software to get into a state that can break it. Given that all non-trivial software maintains some form of internal-state, then to achieve absolute certainty that there are no bugs you have have to test every combination of possible input. Since there is no limit to the size of the input, the number of possible test scenarios is infinite, and it would take forever to create and apply them. Given our restrictions on time, unless we do our development on the edge of a black hole, there will always be some possibly of bugs with any release.

You may put in significant effort for a series of releases to really produce stellar quality, but you always need to remember that software development projects never really end. Sooner or later a bug is going to escape. In practice it is always sooner, and it always happens way more than most programmers anticipate. Although one person called it 'defeatist', it is a really good idea to plan for sending out patches to fix the code. Just assume there will be bugs and deal with it. If you build this into distribution and deployment, then when it is necessary it is just part of the system, otherwise the 'crisis' will eat up huge amounts of time.

Unexpected, 'expected' events cause delays, morale issues and scheduling conflicts. When we continually see the same problems happen over and over again, we need to accept them as part of process, rather than try to ignore them or rail against how unfair they are. If we anticipate problems with the deployment of the software we can build in the mechanisms to make this dealing with the problem easier. Taking a broad approach to development to include the development, testing and operations into the problem domain is the best way to build practical solutions that can withstand significant shifts in their environments.

The time issues never go away and ignoring them only makes it worse and more disruptive.


BROAD STROKES AND DETAILS

There are lots of arguments between strongly typed and loosely typed languages. Different camps of programmers feel that one or the other is the perfect approach to the problem. After lots of experience on both sides, I've come to realize that some problems are best handled with strongly typed approaches -- while others are best handled with very loose ones. The difference comes down to whether or not you need to be pedantic about the details.

For some types of solutions, the underlying correctness of the data is the key to making it work. In those cases you want to build up internal structures while programming that are very specific, and during the process you want to insure that the correct structure is actually built.

If your writing some code to write out a specific file format, for example, you'd like the internal structure to mirror the file format. In that way, depending on the format the structure and syntax are highly restricted. As the various calls go about in the system building up the structure, there is also code to make sure that the structure stays consistent and correct. With many types of errors, the closer the program stops after the first error, the easier it is to diagnose. When running in a development environment, a program that is strongly typed and checking its data can stop immediately the moment it deviates from the prescribed plan. That makes it really easy to find and fix the problems which also helps to minimize testing.

For some solutions, if you are performing a large number of steps and you want the program to be as tolerant as possible, then loosely typed is way better. It doesn't matter for example, how the data got into the program, instead it matters how it will be transformed. High-level programs, scripting, data filters, and any other program that needs to deal with a wide range of unexpected inputs fit well into this type of circumstance. The range of the data is massive, but the transformations are pretty trivial. If the program stopped each and every time the input was an unknown combination, the code would be so fragile that it would be useless. Loosely typing data under this circumstance means that the importance is on the set of instructions, they need to be completed, the data is only secondary. Scripting in particular requires this.

This dichotomy is true for all software. For some goals the data is the most important thing and it needs to be structured correctly. For some goals, it is the sequence of instructions and the final output that is significant. It doesn't matter how it got there.

So for example, we can use a typed language like Java to perform the algorithmic combinations, but a basically untyped tool like ant to insure that the code was built and deployed correctly. That the shell scripting languages in Unix are mostly loosely typed is no accident. The two approaches are needed for two very different problems.

It is also true that within all systems, the sequence of instructions at the higher level is more important, and the structure of data at the low level is important. The nice part is realizing that the depth of the code makes a big difference. If you are building a complex system, the broad strokes of the code should be loosely typed, they are more flexible and less rigid that way, while the detailed calculations should be strongly typed because the accuracy for the data is often the key. Loose typing at the higher level also helps in decoupling the architecture and splitting off the presentation from the underlying data. All these things work together at the different levels within any system.

Clearly any new language that covers the whole domain of building and deploying complex systems will cover the whole spectrum and be both strongly and loosely typed. Working both into the same syntax will be one of those key things that will bump us forward in complexity handling.


ELEGANCE

Clean and consistent code is a great secret in getting things launched, but it is exceptionally difficult to get a group of programmers to follow some underlying conventions. There are cultural issues that make it nearly impossible for big teams to synchronize their working practices. That could be why the initial version of most of commercial software is built initially by little teams and later handed off to big teams for maintenance. We still don't know how to coordinate our efforts correctly on a large scale. Our most common official methodologies are horrible.

Elegance, just for the sake of elegance is never a great idea. Elegance because it makes you job easier and it means that the software works better is a great idea. It is even better when it takes away a lot of the stresses associated with messy programming habits. It becomes a means to an end.

We build and maintain tools to help our users solve their problems, which are most generally in playing with their ever-increasing piles of data. The most important thing is that the tools we build actually solve the problems for the users. Everybody getting their input into the design, programmers having fun while coding, and processes being left wide open and 'casual' may amuse some people while they are working, but they do not get the basic tasks completed any faster. Spending time to understand the user's real needs, keeping the code clean and consistent, using real graphic design for the interface, and providing simple tools that are easy to understand and effective are some of the things that make the users come to appreciate (or not) the underlying software. Computers can make people's lives easier or they can make them harder, the difference is up to the abilities of the programmers involved with the code.

In the end, a short simple solution that works simply and consistently is as good as it gets. Bad, overly complicated bloated software with every function imaginable isn't good, and it isn't much of an accomplishment. All the fancy graphics, dancing baloney and crammed in information can't hide a badly written program. It's not the technology, it's what you do with it that matters.