Duality

There are two very similar ways of looking at software systems. 

The most common one is to see it as a lot of code that moves data around. Code is the primary concept. Its dual, however, is that you see it as data, with the code just assisting in getting it from place to place. Code is a secondary issue.

They may seem similar, and it's easy to miss the difference between the two, but from a higher level the second perspective is a lot simpler and a lot more powerful.

When we first learn to program, we are taught to start assembling larger and larger code fragments. First, it is small examples of branches and loops, putting some code into functions, calling other chunks of code to get them to do stuff. That, rather directly, establishes that ‘the code’ is super important. We get caught up in syntax issues, different languages, and various IDEs. Early coding tasks are to ‘write some code that ...’, interviews are uber-focused on the code too. Everything is about ‘code’.

Data is usually introduced as a secondary issue, but most often it is somewhat trivial and primitive. If we take a data structures course, the actual data is even abstracted away, we’re just left with abstract relationships like lists and trees.

That carries on through most programmers' careers. Their work assignments are often crafted as producing some code to support a feature. In many shops, the code is started first, then later they realize that there were some details about the data that were missed. If there is a bug to be fixed, it is because some code is missing or calculating the wrong values.

So its code, code, code, all of the way down. Data Is an afterthought.

The point of taking a data structures course is lost on most people. Sure, the underlying data is abstracted away but it's not because it doesn’t matter. It's the exact opposite. Data structures are a pretty complete means of decomposition. That is, you can take most large and complex programs and rephrase them as a set of data structures. Most programs are just a bunch of primitive operations happening on a fairly small set of structures like lists, trees, stacks, queues, etc. If those structural operations are pulled out and reused, the resulting code is way smaller, and intrinsically has less bugs. That’s why Donald Knuth collected them all together in the Art of Programming, that is why they keep getting taught in classes. They are the ‘power tools’ of programming, but to get them to work you have to flip your perspective on the system.

Data structures aren’t the only opposite approach. The general ideas around them got formalized and explicitly wired into languages as Object-Oriented programming. In non-OO languages, the programmers had to set up the structures themselves and keep any primitives nearby. With Objects that become a part of the language syntax. Objects bind code directly to the data not because it is fun, but so that objects that are structurally similar can be encapsulated and reused and they can point, polymorphically to other types. It is exactly the same as basic data structures, just formalized into the language.

It’s also why people writing a lot of code in OO that is focused on doing super long lists of instructions end up getting into such a mess with Object-Oriented languages. Retrofitting a brute force procedural style into objects is basically going against the grain of the language. Objects as mega-instructions clash with other such objects, which prevent reuse, are hard to understand collectively and are prone to integration bugs. It makes the code awkward, which keeps getting worse as the system gets larger.

While data structures are very powerful, they are just the tip of the iceberg and came of age in an era when data was really rare and hard to get. That all changed when computers became ubiquitous and networked. Now data is plentiful, everywhere, and often of extremely poor quality. Data was always important, but it's getting more important as we collect a lot more of it and want to draw wisdom out of what we have.

So the alternative perspective is to see a system, just by its data, and how that data flows around. 

For data to be valuable, it has to stay around for a long time, which happens when we persist it into a database. In all ways, the schema defines the possible scope and capabilities of a system. If you haven’t persisted the data, it isn’t available for usage. If you saved some of it, in an awkward format, that will percolate upwards through all of the other functionality you use it for. If the data is available, guaranteed to be correct, then most of the features that require it are fairly simple. If its hard to write some code, its often because the incoming data is a mess.

All data starts somewhere in the real world. That may seem like a controversial statement since data like log files originate in response only to changes in the digital realm, but if you accept that those come from the behavior of the underlying hardware then it makes more sense. Besides operational data, the rest of it is entered from end-users, administrators, programmers, or third-party organizations. It starts in the real world, and really only has value in the real world. On top of this raw information, we can derive other useful relationships but it all has to start somewhere.

What becomes important then is how the data flows from one location to another. For example, it may have started as an observation by a person. They used some interface to get it into persistence. Later it might be pulled from storage and used to augment some other data. That new data is shown in another interface or packaged together and sent to a remote location. Maybe in exchange, more data flows into the system from that remote site. 

If you just look at the data and ignore the code, threads, processes, etc. most systems are not particularly complex. They act as a hub to collect and distribute data to a whole bunch of different sources, people, or other machines. 

What’s needed then, to build a system that manages that data is many fragments of code that can move, decorate, or translate the data as it circulates around. If you minimize the fiddling done with that code as it travels around, you’ve optimized a large portion of the system without even realizing it. The closer those data formats are in the widgets, middleware, and persistence, the less work that is needed when it is in motion. Moving data is always expensive.

That perspective puts data first. It is more important than the code, and the idea is to minimize what the code is doing to it whenever possible. There still might be memoization or other macro optimizations that are possible, but those can be seen as refinements.

What becomes important then is keeping dependent pieces of data together and managing the structural relationships between these ‘entities’. This brings us right back to data-structures. They deal with the relationships between these larger chunks and can be built to have indirect references to the underlying entities. Why? Because the structural relationships are smaller and more common. If you get them perfected for one type of entity, you can reuse them for all types of entities. That then just shifts the problem down to picking a good set of entities and relationships that best fit the data as it originated in the real world, or basically ‘modeling’.

Now the data perspective doesn’t magically eliminate performance problems. Systems aren’t intrinsically scalable, handling huge loads has to be explicitly engineered into the system in order for it to work correctly. But seeing the system as flowing data does make it a whole lot easier to lay out an industrial scale architecture. 

Take caching for example. From a code perspective, programmers often just allocate a chunk of memory, set it up as a hash table, then do some type of lookup first to get the value. That seems to make caching easy, but eventually, the real problems show their ugly heads. If you see caching as a form of memoization where we keep a smaller pool of data closer to the action, then what is obvious is how to decide what’s in that pool and what's not in it. Caching stale data is bad, and also letting the cache grow to be the same size as the persistence is rather pointless. But on top of that, if the data may be in two places at the same time, what happens when you need to update it? The code perspective of caching forces programmers to think about memory, while the data perspective forces them to think about the quality of the data. It makes it easier to see the whole picture, which makes it easier to get the implementation correct. Once you realize that removing stuff from the cache and keeping it in sync during writes are the real problems, figuring out what code and architecture are needed to make this happen is a whole lot easier.

The same is true for putting queues between systems to resolve speed differences and with syncing external information for read-only internal copies. Pretty much all of the big enterprise issues get a whole lot less challenging. The difficult problems shift away from some tangled mess of code, to really keeping the whole thing organized and operating correctly. 

It also applies to the really big systems as well. It’s hard to figure out how to massively parallelize systems until you realize that it is just an issue about dependencies. Where data is independent, it can be safely split across threads, processes, or machines. If it's not independent, then there will be problems. So, instead of being a complex coding problem, it is really a complex data modeling one. What underlying model do we need that both approximates the real-world issues, but also guarantees some usable independences that can be parallelized? If you structure the data correctly, the rest is just spending the time to write out the code.

As an industry, we usually start by teaching new programmers to code. That’s okay, but we often fail to teach them how to flip to this dual perspective. Instead, we leave them dangling in the wind, trying to crank out code from what is clearly a much harder perspective. Sure, some people get data structure courses or even computer theory, but then they go out into the industry and none of that sticks. To make it worse, people tag it to ‘abstraction’ and ‘overcomplexity’ and keep trying to insist that writing more overly simple code, faster, is somehow going to make it better. It even percolated back into our bad interview styles. 

It permeates everything, making most serious programming tasks a whole lot harder than they need to be. If you’ve ever suspected that there was an ‘easier’ way to build systems, then you were right. If you forget about the code, and focus on the data, getting that set up correctly, then most of the coding is easy and straightforward.

Theory and Practice

Nearly three decades ago, when I started university all I really wanted to learn was the magic of programming. But my course load included plenty of mathematics and computer theory courses, as well as crazy electives. “What does all this have to do with programming?” I often complained. At first I just wished they’d drop the courses from the curriculum and give me more intensive programming assignments. That’s what I thought I needed to know. In time I realized that most of it was quite useful.

Theory is the backbone of software development work. For a lot of programming tasks you can ignore the theory and just scratch out your own eclectic way of handling the problem, but a strong theoretical background not only makes the work easier it also is more likely to withstand the rigors of the real world. Too often I’ve seen programmers roll their own dysfunctional code to a theoretical problem without first getting a true appreciation of the underlying knowledge. What most often happens is that they flail away at the code, unable to get it to be stable enough to work. If they understood the theory however, not only is the code shorter, but they’d spend way less time banging at it. It makes it easier. Thus for some types of programming, understanding the underlying theory is mandatory. Yes, it’s a small minority of the time, but it’s often the core of the system, where even littlest of problems can be hugely time intensive.

The best known theoretical problem is the ‘halting problem’. Loosely stated, it is impossible to write some code that can determine if some other code will converge on an answer or run forever (however one can write an estimation that works with a finite subset within a Turing Machine and that seems doable).  

In its native form the halting problem isn’t crossed often in practice, but we do see it in other ways. First is that an unbounded loop could run forever. An unbounded recursion can run forever as well. Thus in practice we really don’t want code that is ever unbounded -- infinite loops annoy users and waste resources -- at some point the code has to process a finite set of discrete objects and then terminate. If that isn’t possible, then some protective form of constraint is necessary (although the size should be easily configurable at operational time).

The second way we see it is that we can’t always write code to understand what code is trying to do. In an offbeat way, that limits the types of tools we can use in automation. It would be nice for instance if we could write something that would list out the stack for all possible exceptions in the code with respect to input, but that would require the lister to ‘intelligently’ understand the code enough to know the behavior. We could approx that, but the lack of accuracy might negate the value of the tool.

Another interesting theoretical problem is the Two Generals Problem. This is really just a coordination issue between any two independent entities (computers, threads, processes, etc.). There is no known way to reliability get 100% communication if the entities are independent. You can reduce the window of problems down to a tiny number of instructions, but you can never remove it entirely. With modern computers we can do billions of things within fractions of a second, so even a tiny 2 ms window could result in bugs occurring monthly in a system with a massive number of transactions. Thus what seems like an unlikely occurrence can often turn into a recurring nuisance that irritates everyone.

Locking is closely related to the Two Generals Problem. I’ve seen more bugs in locking than in any other area of modern programming (dangling pointers in C were extremely common in the mid 90s but modern languages mitigated that). It’s not hard to write code to lock resources, but it is very easy to get it wrong. At its heart, it really falls back to a simple principle: to get reliable locking you need a ‘test-and-set’ primitive. That is, in one single uninterrupted single-threaded protected operation, you need to test a variable and set it to ‘taken’ or return that is it unavailable. Once you have that primitive, you can build all other locking mechanisms on top of it. If it’s not atomic however, there will always be a window of failure. That links back to the Two Generals Problem quite nicely, since where it becomes an issue is when you can’t have access to an atomic ‘test-and-set’ primitive (and thus there will always be problems).

Parsing is one of those areas where people often tread carelessly without a theoretical background, and it always ends badly. If you understand the theory and have read works like The Red Dragon Book then belting out a parser is basically a time problem. You just decide what the ‘language’ requires such as LR(1), and how big the language is and then you do the appropriate work, which more often than not is either a recursive descent parser or a table driven one (using tools like lex/yacc or antlr). There are messy bits of course, particularly if you are trying to draft your own new language, but the space is well explored and well documented. In practice however what you see is a lot of crude split/join based top-down disasters, with the occasional regular expression disaster thrown in for fun. Both of those techniques can work with really simple grammars, but then fail miserably when applied to more complex ones. Thus being able to parse a CSV file, does mean you know how to parse something more complex. Bad parsing usually is a huge time sink, and if it’s way off then the only reasonable option is to rewrite it properly. Sometimes it’s just not fixable.

One of my favorite theoretical problems is the rather well-known P vs NP problem. While the verdict is still outstanding on the relationship, it has a huge implication for code optimizations. For people unfamiliar with ‘complexity’, it is really a question of growth. If you have an algorithm that takes 3 seconds to run with 200 inputs, what happens when you give it 400 inputs? With a simple linear algorithm it takes 6 seconds to run. Some algorithms perform worse, so they may take 9 secs (3^2 -- three squared) to run, or even 64 seconds (4^3 -- four to the power of three). We can take any algorithm and calculate its ‘computational complexity’ which will tell us exactly how the time grows with respect to the size of the input. We usually categorize this by the dominant operators so O(1) is a constant growth, O(n) is growing linearly by the size of the input, O(n^c) is growing by a constant exponent (polynomial time) and O(c^n) has the size of the input as the exponent (exponential time). The P in the equation is a reference to polynomial time, while NP is rather loosely any growth such as exponential that is larger (I know, that is a gross oversimplification of NP, but it serves well enough to explain that it references problems that are larger, without getting into what constrains NP itself).

Growth is a really important factor when it comes to designing systems that run efficiently. Ultimately what we’d like is to build is a well-behaved system that runs in testing on a subset of the data, and then to know when it goes into production that the performance characteristics have not changed. The system shouldn’t suddenly grind to a halt when it is being accessed by a real number of users, with a real amount of data. What we’ve learned over the years is that it is really easy to write code where this will happen, so often to get the big industrial stuff working, we have to spend a significant amount of time optimizing the code to perform properly. The work a system has to do is fixed, so the best we can do is find approaches to preserve and reuse the work (memoization) as much as possible. Optimizing code, after its been shown to work, is often crucial to achieving the requirements.

What P != NP is really saying in practice is that there is a very strong bound on just exactly how optimized the code can really be. If it’s not true then there would be no possible way you could take an exponential problem and find clever tricks to get it to run in polynomial time. You can always optimize code, but there might be a physical bound on exactly how fast you can get it. A lot of this work was best explored with respect to sorting and searching, but for large systems it is essential to really understand it if you are going to get good results.

if it were true however, amongst many other implications, that would mean that we are able to calculate some pretty incredible stuff. Moore’s law has always been giving us more hardware to play with, but users have kept pace and are continually asking for processing beyond our current limits. Without that fixed boundary as a limitation, we could write systems that make our modern behemoth's look crude and flaky, and it would require a tiny fraction of the huge effort we put in right now to build them (also it would take a lot of fun out of mathematics according to Gödel).

Memoization as a technique is best known from ‘caching’. Somewhere along the way, caching became the over-popular silver bullet for all performance problems. Caching in essence is simple, but there is significant more depth there than most people realize, and as such it is not uncommon to see systems that are deploying erratic caching to harmful effect. Instead of magically fixing the performance problems, they manage to make them worse and provide a slew of inconsistencies in the results. So you get really stale data, or a collection of data with parts out of sync, slower performance, rampant memory leaks, or just sudden scary freezes in the code that seem unexplainable. Caching, like memory management, threads and pointers is one of those places where ignoring the underlying known concepts is most likely to result in pain, rather than a successful piece of code.

I’m sure there are plenty of other examples. Often when I split programming between ‘systems programming’ and ‘applications programming’ what I am really referring too is that the systems variety requires a decent understanding of the underlying theories. Applications programming needs an understanding of the domain problems, but they can often be documented and passed on to the programmer. For the systems work, the programmer has to really understand what they are writing, for if they don’t, the chances of just randomly striking it lucky and getting the code to work are are nearly infinitesimal. Thus, as I found out over the years, all of those early theory courses that they made me take are actually crucial to being able to build big industrial strength systems. You can always build on someone else’s knowledge, which is fine, but if you dare tread into any deep work, then you need to take it very seriously and do the appropriate homework. I’ve seen a lot of programmers fail to grok that and suffer horribly for their hubris.

Creatively Speaking

Somewhere in my youth, I came into possession of a very simplified model of the human brain. I'm not sure from where, it was so long ago its roots were lost in time.

The idea is simple, the brain consist of three, and only three things. First is memory. We can memorize stuff. By that, I mean that we take some fact, associate it with a number of keys and then, on demand, we can return the fact as needed. It is committed to memory. It becomes a piece of knowledge, something we know. I'll quietly ignore long-term vs. short-term memory for this model.

The next thing is understanding. That is, we understand something. For this, I mean that we can take some set of facts that we learned, and at the appropriate time use them for some piece of higher reasoning. If you've just memorized a fact, that's nice, but until you understand it, you can't really take advantage of it. In some sense, understanding is really just how that set of facts is linked to other sets of facts that can in some way drive or change your behavior. You 'know' what an stove burner is (fact), and you 'understand' that once its turned on, it is hot and will burn you.

The very last thing is creativity. This, conceptually is a little more complicated. If a fact is a thing you remember, and understanding is how it is linked, creativity in some offbeat way is how you can create 'dynamic' linkages between non-related facts or understandings. You might have a pencil for instance, and you know how to use it to draw shapes and textures, but at the time you are working you're feeling blue, which is some intangible emotion. If you want to use the pencil and your ability to draw to express your emotion state by drawing a dark moody image of yourself, you are taking your pencil, and your self-view, both facts, applying your understanding of drawing and then 'combining' that with your current emotional state to 'create' something that on its own would not come into existence for anyone else. You are creatively expressing your emotions through a self-portrait with a moody overtone. The work is not reproducible.

Creativity then is this ability to combine two entirely unrelated things, in arbitrary, random, or mostly irrational ways. The base things are either facts or understandings.


CONSEQUENCES

A consequence of this simplified model is that the abilities of a human on one level are quite simple. We build up knowledge -- linking it to other bits -- in ways that we find useful. Indirectly, as if it were a computer program run amok, one can see creativity as some type of inherent 'flaw' in our brains. After all, connecting two things that "shouldn't" be connected is not rational or deterministic. It's like a computer programming randomly changing the data, it is outside of the norm. The brain shorts out, or something.

From a natural point of view, this model allows one to see human progress often as this long history of people gradually creating new ideas and memes by either observing them in nature or randomly combining together un-related bits. In a direct sense, all of our of knowledge beyond what is quantifiable, is accidental. We stumble into discovering it.

That's a little hard for many people to swallow, so it likely shows that this model is far to simplified to be absolutely correct, but still quite interesting. To get to the really high concepts, they have to be based on lower ones, that somewhere down the way have to be based on observable things, or creative leaps.

I like this model because it provides nice tidy explanations for many things. For instance, why somebody with 'photographic' memory does so well in their early university courses, but then has trouble later on; they never really understood the original stuff, just memorized it. The latter courses require understanding as well as memorization. You can't build on a foundation that you don't have.

Intelligence is not just our ability to remember facts, understanding and creativity play a huge role as well. So that means a brilliant person is one, then who has a lot of facts, and understands them well. While a genius is an intensely creative person, who many not actually sit on that big a reservoir of understandings or facts; but they can just combine what little they have into interesting new bits. There are a lot of brilliant people, while creativity, is a rarer thing.

This model harmonizes 'intelligence' and shows that one cannot easily compare their three values against other people's three values. With three dimensions, you cannot easily rank your intelligence. Often you can meet people who can quote tremendous amounts of knowledge, but seem to have trouble really putting the pieces together. Your typical absent minded professor may be incredible forgetful, but full of deep understandings. It is a great perspective on our abilities.


COMPUTER SCIENCE

Now, all of that being interesting, what does it have to do with computer science? I read a excellent write up recently of someone talking about how they believe that math is actually creative. It is know as Lockhart's Lament and it is at:

http://www.maa.org/devlin/LockhartsLament.pdf

This to me this is a hugely misunderstood characterization of what it means to be creative, and of what mathematics is, and why it is so special. In a way, relating it back towards creativity takes away from the true underlying awe and beauty of mathematics. It trivializes it.

In a very real sense, mathematics is an infinite set of axioms over an abstract space. It does not exist in our real world and because of its rigour, which could not exist either in this world, it is purely abstract. Math is sort of this pure crystallize abstract structure that we can perceive if we concentrate on it long enough. It exists, but without substance. A truly amazing thing.

As such, all math exists and has always existed. It always will, with or without us.

The branches, axioms, primitives etc. of the various different types of math are our expressions of the that abstract foundation. It is out there, right now. All of our existing math -- a drop in the overall bucket -- and all of the stuff we still do not know. Mathematicians set out on a quest to 'find' new mathematics. They find it, prove it is rigorous, and then explain it to the rest of us.

In that sense, mathematics as we know it, is a huge set of facts and a significant amount of understanding about those facts. In its abstractness, it is consistent, and contained. It is not contaminated by the real world, where entropy and chaos dominate. It exists in a pure state, perfect, rigorous and eternal.

Knowledge and understanding then easily define our existing view of mathematics. They exist, so there is little that is inherently creative in their existence. Learning mathematics is not a creative exercise. You study the existing works, memorize the facts and work on problems until the understanding of the facts kicks in. If you are being led down the garden path, then as exciting as it is, learning about some new set of axioms and how to use them really isn't your creative spark, is it?

If learning mathematics is not particularly creative, then the big question must be one about 'finding' the ideas initially; whether or not that is a really 'creative' endeavour?


DOWN ANOTHER HOLE

To digress just a little bit. Software development is in its very own way a form of expressing a set of instructions to a computer that in its abstractness and rigour are similar to the act of creating a mathematical proof. Expression, be it about what you had for lunch, the weather or some specific algorithm for searching is a form of communication. The underlying topic may change, but choosing English, French, C or Java for your expression is still a way of communicating something. For English, it is non-rigorous and it goes to another human being. For C, it is very rigorous and it goes to a computer, but if both are about algorithms, then they are just different ways of expressing the same underlying thing. We only choose to express vastly different things in the different mediums.

There is, of course, a question about the expression of intangible quantities. It is one of those that Richard Gabriel in his book "Patterns of Software" says architect Christopher Alexander called: "the quality without a name". We have to categorize the world into those things that are determinable, tangible in some way, and those that are not. Tangible usually means, some form of physical existence. For the intangible, mostly it comes to us, personally in some what that is spiritual to some degree. By that I do not evoke any higher meaning, other than to describe 'spiritual' as a mood that is specified by humans. It is one of our many emotions. In Gabriel's book he talks about the feelings one gets when they go into these huge architectural wonders, often churches, but not always. It moves people in some intangible way. That unnameable quality then to my mind is just another intangible human emotion. Perhaps awe, or something similar.

That, fortunately makes for a nice and tidy taxonomy, since I've often described fine art -- for example -- as having an embedded emotional context, which is what separates it from merely being graphic design. If it stirs you in some way, it is art. If it is just pretty, then it is only graphic design. The same is true for music, it moves you or it is 'pop' music.

What we can perceive with our intelligence is tangible, and what we can experience with our emotions is intangible.

And so, in my earlier example with the drawing, the self portrait combines two very distinct things, one quite tangible, the representation of a human -- with the other, which is intangible -- the emotion of feeling somewhat blue. The creative 'short' or spark that drives that is not natural to most people, so although afterwards we (some of us) can see what the artist was getting at, and are moved by it. Although left in the same circumstances, we -- not being that specific artist -- would not have been creative in that same way.


MATHEMATICS

Coming back to mathematics: we know that the math is out there, and that it is abstract, yet not intangible. It is a thing, and if you search hard enough, eventually you will come upon it. I am not saying that in any way it is easy, or that it would take humanity anything less than thousands of years of culture before they might stumble on a complexity concept like negative numbers, which is exactly what happened. It also took a long time for us to figure out zero, complex numbers, calculus, etc. Some humans clearly can get there faster, but the road that needs to be traveled needs to be traveled by all, collectively.

We could not have complex branches of mathematics for instance without having an understanding of negative numbers. The intuitive leap that one and only one person can make, can only be so far. We are limited by our life spans. The imaginary of standing on shoulders of giants is exactly correct if you realize that the giants themselves are only just other people standing on other people's shoulders. And on it goes.

We get left then with the understanding that finding new forms of mathematics is a search, and it is one that comes from a limited distance from our last existing searches. With each new generation we can cover new ground, but only just barely. In that, although for one human, they learn the existing knowledge and then the existing understandings, they can only start searching from that point on. To them it is an open un-explored field with hidden surprises, to humanity however, it is just part of an ever going permutation through all of the localized space relative to the last set of solved problems. So, in the end, the best that a mathematician can do -- in that sense -- is to find or get to their discovery before the hoard of people following them do. Speed is important, but so is being in the right place at the right time, as the field is constantly filling up with others overturning the same rocks and prodding the same bushes too. If you don't discover it, they eventually will.

All of this really comes back to whether or not the idea of someone prodding around in a big field looking for interesting bits of mathematics is in fact being creative or not. And, to some very large degree, if mathematicians are creative in what they find, then by virtual of their similarity, computer scientists -- at least for the very first time they are writing something really unknown -- are equally as creative.

As if you needed more twists, we still need to try and relate this situation back to something more physical.

The simplest mapping would be to look at a graphic artist. They are by virtual of their training able to produce things that are visually appealing. Yet, as we are awash in pretty graphic design, you know that their work is not stirring in any fashion.

It looks nice, but it doesn't not give rise to emotions. That's true because while they have taken their facts, and taken their understanding -- in this case abilities and skills at things like drawing or matching pleasing colors -- they have not crossed the gulf and gone on to combine these with some intangible emotional context or any other thing that is out of the ordinary. Their skills may be beyond what some in the 17th century would have been able to do -- appearing back then to be 'art' -- but here and now, in this time they are applying the things they learned in school to a specific end. Understanding, yes, creativity, probably not.


BACK AROUND AGAIN

So going back to our mathematicians, as they ponder and search throughout the field, are they really being creative, or are they not just applying the skills thy were taught towards the goal of identifying previously unknown bits of mathematics? There may, I admit be some truly creative spark there that helps particularly gifted mathematicians to get to a specific point in the field faster than others and allows them to identify their thing rapidly. How else, can you explain Einstein? Although his leaps where related to a specific science, not general abstract mathematics, the distance that he appears to have travelled vs. his contemporaries at the time seems to be enormous. But again, what he found was there plainly for all to see, if they wanted to, or were willing to look. Relativity always was, and always will be. You just have to want to see it. But still, the leap was big and it took twenty years for his fellows to catch up with him and get to that particular place.

And so, for me, I'd have to say that most of the leaps that are going in mathematics are not leaps, but a natural progression of steps. And that these steps are an expected consequence of the knowledge and understanding that come from studying the field and really understanding it. That sometimes, but very rarely, a few make creative'ish giant leaps, but in general people are just working towards the goals of their field. If any one famous mathematician had died early, it may have taken us longer to get to their understanding, but we would still get there. The results might be slightly different, possibly composed of a different set of primitives, but it will still be functionally equivalent. As mathematicians create proofs, so too do computer programmers create code. It is exceptionally less complex, and far more mundane, but still shares the same type of relationship.

The code exists, we just have to find that sequence of instructions that expresses the solution to the problem in the most elegant fashion. So, we have to ask again: is programming really creative? The answer would have to be rarely. Only at that moment where we are combining two unrelated things in a truly unique way are we actually allowing our brains to take creative leaps. Otherwise we are just applying our understanding of how to code.

If your just doing another day's worth of problem solving, then those are exactly the skills programmer's should have and were taught. Software design has creative moments, and often the design is left unspecified so that some of the effort is required by the programmers, but mostly the business domain, technical domain and best practices remove a significant amount of the degrees of freedom. The interfaces share the same creative tones as graphic design. Most of it is really the application of an understanding.

Is software development art? Never. There is virtually no place in software for the expression of intangible emotions, I can barely imagine what that might look like. Tools are just tools. But, to slightly confuse the issue, the content of the programming could be art. That is, you could create a masterpiece of art using code as the means of expression. Games designers and many of the early pioneers of tools like flash succeeded or came close.

But really, a tool that moves you to tears is somewhat counter-productive, yes? It would be difficult to be both stirring and functional. Although, I do have to admit that some big company software products have caused me to cry, but really, that is only due to dealing with their crap, and my inability to be able to find a better replacement. The code did not stir the emotions, the angst of dealing with it did.

The Essence of Analysis

In any venture if you do not start off on the right foot, your chances of success are dramatically diminished. So it is easy to understand that the farther away we are from our initial target, the longer it will take to get there. If we have no idea where we are even going, then we won't even know if we have ever arrived. Some things, just are.

Many software developers understand that design is a highly critical part of getting the software right. You can't just magically iterate your way into the perfect system without first understanding what the system is supposed to do. Formal, or on the back of napkin, developers need a target to work towards. However, it is often the case that even the best of all designs still cannot help if the even earlier stages are fatal. Design is not actually the first step.

The very first thing for any building is to have a reason to build it. A building without occupants, a book without readers, or a software product that does not solve any real problems are all examples of things that got lost in their very first stage: analysis.


SOME EARLIER THOUGHTS

A while back, I wrote down some ideas about what I thought were the essential development problems:

http://theprogrammersparadox.blogspot.com/2008/01/essential-develoment-problems.html

I identified four big issues a) perception, b) discipline, c) generalization and d) analysis. Of these, analysis is the one that I have least talked about, primarily because without resorting to domain specific examples -- generally proprietary in the software world -- talking about it is difficult.

But analysis is clearly one of the keys to getting the right thing built at the right time. Turning a list of functions into executing code takes effort, but turning a vague understanding of someone's process into a list of functions takes a genuine understanding. It is a very difficult, very overlooked and a very uncommon skill set.


BASIC DEFINITIONS

In analysis, only one group of people is really important: the user of the software. All of the other 'stakeholders' have some effect in helping or derailing the project, but your quality of work comes from only pleasing one group. For this entry I'll focus only on them, leaving the others as just background obstacles to be overcome.

All that software can do is to help a user manipulate a growing pile of data. Whether it is visualization, tracking or editing, software is incredibly simple; each function just helps play with the information in some way. The key to building good software, then is to obviously tie this back to something useful in the user's life. Software that helps solves specific problems is software that is desirable. It extends out the user's ability to manage their information efficiently and effectively. Software that violates that covenant just irritates it users. Realistically most software is a mixture of useful and irritating features. Sometimes on purpose, often not.

Without a doubt, the most common approach to software development is a combination of hanging around on an ivory tower and collectively guessing at the sorts of problems that the user might want solved. If this sounds haughty and aloof, it generally is, which is why a great deal of software lacks a considerable amount of empathy for its users. Missing empathy equates to sticky, hard to use features that beg to be rewritten. It doesn't need to be like that, but that's the industry norm.

If you stand back a bit, you can see that software solves two basic problems: one from the business domain, the other from the technical. The technical domain is the packaging that is required by the programmers to get the business solution to the right people. The users for example may want to keep track of their financial information; the database, GUI, programming language, etc. are all just technical solutions that help the user access the financial tracking capabilities that they need. From a user's perspective, they are essentially noise that has to be put up with, in order to accomplish their goals. User's don't, and shouldn't care about the technical issues.

Programmers, on the other hand, love solving technical problems which are easier and more straight-forward. As such they try to take those abilities learned from creating technical solutions and throw them at the business problems. Generally to very painful results. Technical problems are consistent, simple and rational. Business problems, at least from the perspective of the programmers are messy, irrational crazy things that make no sense. The type of problems that scare most developers. Trying to wrap the irrational in a simple consistent model is a hopeless task.

Because of its messiness most programmer's prefer working on technical issues. If you need any real proof for that, a quick examination of the OpenSource movement shows that for the most part, the projects -- particularly the successful ones -- have been entirely spawned around and dedicated to technical problems. Volunteer programmers want to tackle easy fun problems, so they are naturally drawn to the technical ones. Why tackle the painful stuff for free?

Not surprisingly, the bulk of the world's programming problems are not technical. They are domain specific, which is why commercial development and consulting are still significant industries, even in the face of so many programmings seemingly giving away their craft. Given the difference between the two types of problems and the obvious lack of appeal for the ugly business code, it is more than likely software development will continue to be a well-paying profession for some time.


CONSULT THE EXPERTS

A computer programmer is someone who can assemble a complex set of instructions intended to implement some functionality in a software program. A software developer, on the other hand, is someone who can analyse real world issues, isolate the problems and then conceive, design, implement, test and deploy a working solution to help with those problems. The difference is clearly the scope of the work being performed. There are lots of computer programmers, but very few real software developers. You need to master the technology first, and then grow your abilities to start tackling the really hard problems.

The users are the expert at understanding their own problems, but an experience software developer is the expert at turning that into some workable solution. Naturally because domain problems are extremely messy, deep and irrational, years of experience with a specific domain are important. You can't solve problems if you don't really understand them.

While the developers need to understand the domain in which they are working, they are not by definition the experts. They don't have to know it all, or are necessarily functional in it. They really only need to see it from the perspective of the computer software:

- the data, its structure, frequency, quality, etc.
- the processes, their timings, interactions, differences, etc.
- the players, their roles, responsibilities, expectations, etc.

A computer takes a deterministic static view of the data, so all that the software developer really needs to understand is that viewpoint. But, that viewpoint must, to an incredible level of detail, be understand really thoroughly. The success of software rests on its implementation of the details.


FINDING THE PROBLEMS

The first mistake most developers make is trying to solve the wrong problems. Computers can't change things, they can only act as extension or tool for the users. In that sense, any software development is an automation project, one that is geared towards a specific pile of data. Thinking your shifting their paradigm, rocking their world, or changing the way they work will ultimately fall flat. At very most, someone farther upstream might be able to eliminate some lower-down position, with help from the computer, but at the very end, it all needs to be anchored back to a person, and they, at least in their capacity of using the software, aren't going to be pure management (which is a reporting only position).

So, software problems, are simple automation projects that involve building up piles of data. That leads to a fundamental property:

- data has to be captured or entered into the software.

It sounds silly, but in large projects people often fail to trace the data back to its source, a simple exercise that easily reveals many of the worst detail problems with the foundations. If you trace back the data to its source, you generally have a broader understanding of the domain. Even in a big warehousing project, all of the data starts somewhere, be it an an interface for service people, some type of capture scanning, or some feed from some other database. The other database had to gets its data from somewhere. So in general, most data was entered directly from manual forms on a screen. If you dig deep enough you'll find the real source, and thus the real problems. For each piece of data, you have to know the quality and frequency, that is vital.


UNDERSTANDING AND STRUCTURE ARE KEY

Bad assumptions, and misunderstanding data are very common problems that propagate through out development projects. I've speculated that there is essentially one great superstructure underneath in some of my blog entries (Age of Clarity, The Science of Information), but whether or not you believe that, getting a real working proper model of the underlying data is a hugely problematic issue. In many industries, even after thirty years, huge elements of the underlying data are frequently misunderstood. Guessing -- which is the most common approach to understanding data -- is dangerous, particularly if those poor initial pot-shots get permanently enshrined into the schema for reasons of backward compatibility.

So many software programs start off with a slightly-off underlying model, and are never able to truly fix their original problems. Seeing the danger of that, it is important to simplify, but not to miss any obvious chunks. Because of the limits of our current technology, it is extremely hard to 'uncomplicate' a schema once it is in production.

Clearly, the best most reasonable approach is to seek out the users on their own ground and have them explain their problems to you in their own language. This initial 'incomprehensible' babble, is only so, when you don't understand their domain. As such, any communication impedance mismatches, or user complaints, most often point to significant domain issues that you need to understand and resolve. Programmers are quick at dismissing their users as 'stupid and ugly', while software developers know to look deeper. We build for them, and if we don't take the time to see their point of view, that is reflected in our work. With understanding and experience, the things that you initially took for trivial turn out to have significant value.

There are of course, times when the users are just letting off steam, but unless you know that for sure -- not as an assumption -- then it is best to assume there is something tangible, even if it is slight, underneath that is worth considering.


THE USER IS NOT THE DATA

In that data has an underlying structure, that understanding and interaction is rarely what the user is after. For example, we want to interact with our computers in a desktop metaphor, while underneath they need to deal with files, folders and links, and under that: file formats, data structures, bytes and bits.

An incredibly common mistake, particularly when it comes to code generators is to assume that the user perspective, and the data perspective are identical. The data contains inherent ugliness that the computer should shield the user from. In that sense, in a classic system, the database schema will contain a level of detail that is completely uninteresting to the user. Their application summarizes, stores context, and essentially hides away a great deal of that detail. And, the user perspective is irrational, there is no formulaic way to tie it back to the data model. It will be similar, but it will always be very different in irrational ways.

Harmonizing that translation between the user perspective and the true data perspective is another great problem. Overtime people have understood that it was there, but all too often they have attributed it to the wrong causes, like assuming it was physical storage issues, or alternative viewpoints.

I guess for simplification reasons, we would really like to have one single consistent model for all of our data. And having one model that we could magically use to generate all of the code is the next logical conclusions. But, as many programmers know, there are a huge number of transformations happening in any well-written software. Now, we just do the work, and stick it were it is convenient, but the duality between the structure of the data, and the user view, points to an area where we really should be able to separate out the different models in the architecture and formalize (to some degree) the transformations back and forth between these different perspectives.

With further consideration, you can see this as a strong reason why many attempts at re-use fail so badly. The developers, in their quest for one overall model, mix the application-specific structure with the underlying real structure tying the resulting code absolutely to the one and only one application for which it mostly matches.

Beyond the technical problems of implementation, this has a big impact on analysis. The analyst needs to work with the users to see their perspective, but they also need to get down to the real underlying data perspective as well, something that the users may or may not be aware of. The user champions, then are often only really experts at half of the details. The analyst still needs to find and understand the other half. That is a huge problem, but manageable if you are aware of its existence.


CONSOLIDATING THE KNOWLEDGE

So, great you've successfully gone out and found a hundred or so really important 'functions' that would make your user's lives easier as they accumulate their growing pile of data. Randomly slapping these 'things' into a GUI is a bad idea. Sure it is easy to just extend the system by attaching a 'bag' onto the side of it with some disconnected functionality, but that is the type of analysis and design that makes all of us hate those 'big company' programmers.

If you've really come to understand your user's needs, then you should understand how to integrate that back into the existing solution without just gluing on more disconnected features. It is this consolidation of the feature set, that is often extremely difficult. But finding a way to accomplish this, not only makes for a better tool, it also significantly cuts down on testing. E.g. testing one slightly enhanced application is less work than testing two disjoint ones (although most companies fail to recognize this, and only increase their testing resources by a fraction of what is really needed, so they get two apps tested to a lessor degree than the original one).

Mostly, we design our tools to have some consistent overall metaphor, or style of working. Extending that, while maintaining its consistency is crucial. At times, the right answer may to actually throw away all of the old perspective and go up to the next level for a totally new one. Keeping everything simple and consistent, is extremely hard, and extremely dependent on the underlying domain and tool set.

Backward compatibility is important to some degree, but that should not be used as an excuse to not properly integrate functionality into an application. The tools should get easier to use as the coverage of the functionality grows, something that is clearly not happening with a lot of modern software.

Consolidation is as hard as the initial analysis, and is considerably more important. We've reached the stage were we can bang out some pretty sophisticated software products, but we have not gone over to being able to take a large amount of functionality and make it useful. This is obviously somewhere significant where Computer Science needs to grow up.


THE LAST WORDS

Bad, poor, or missing analysis is the start of a doomed project. You can't sit in a cubicle, miles away from your users and expect to be able to write useful software for them. Where you succeed with this approach, it is only and absolutely luck, and nothing else. It is easy to see why so much of our software is so defective.

In five years of constant programming, most people who stay at it will master to some degree the ability of structuring sets of instructions for the computer to execute. These people are computer programmers, and mostly should get some help from others to determine what the instruction set should accomplish.

Our industry standard is to recognize a five year career as being a senior, but while that might be true of someone attempting to master implementing specific functionality into a particular technology, that isn't even remotely true of someone trying to learn how to analyse a problem domain and use that information to build tools. There is far more beyond just belting out sets of instruction that is required to build things that are usable.

The secret then, if there is one to be known and understood, is to master the various techniques and technologies of software development, but never ever, even for a moment 'assume' that you have completely mastered the problem domain issues. Well, once you have re-written the same system three times, you're probably on your way towards mastering that specific issue, but jump problems, domains or industries and wam! your back to ground zero again. We have to be very careful to understand the difference between what we know, and what we assume we know. Building a simple system to assemble a big pile of data, does not make us really understand how that data works in the world around us.

If you can accept this, then you can prepare for a significant number of serious problems in the development leaking in from your lack of knowledge of the problem domain. Once you are past your own ego, you are a little more likely to accept the problems and their irrational nature for what they really are: the things that your software needs to solve to become better. Building software is easy, knowing what to build is the hard problem. All too often our industry wants to make the former sound harder and ignore the latter.

From scratch or extending a system, an experienced software developer makes a huge difference. Luck is the only thing that replaces a lack of analysis. The software industry's high failure rate comes from its inability to recognize the important of hiring developers with senior analytical skills to properly lead development projects. Analysis is a significant skill set. It is unique and not related to writing code. It is not just taking notes from user conversations and building up a list of requirements. It is not writing down stories, or guessing at how people might use things. It is a full in-depth understanding of the user's needs and the underlying data, and how to tie these two disparate things together into tool that is usable.

Even after nearly twenty years and many different software development domains, I am still often surprised by the 'depth' of complexity that some domain specific information is hiding. That is to say, that with each new problem I attempt to analyse, it is far deeper, far more complex and far more irrational than I'd ever really expect, and I've dealt with some pretty deep, complex and irrational problems already. But, I have certainly seen, again and again, that if you get into the user's perspective and build them a tool that really works, while capturing (but hiding) the true complexity of the data, you've come very close to really mastering software development.

If you get the analysis correct, then all you have to do it build it :-)

In Object-Orientation

"Familiarity breeds contempt" is a common cliche. It sums up an overall attitude, but for technology I don't think it is that simple.

I am certainly aware that the more we work with something, the more we come to really understand its weaknesses. To the same degree, we tend to overlook the most familiar flaws just because we don't want to admit their existence. We tend towards self-imposed blindness, right up to the instant before we replace the old with the new. It was perfect last week; this week it is legacy.

Software development has once again come to that state where we are searching for the next great thing. We are at a familiar set of cross-roads, one that will possibly take us to the next level.

In the past, as we have evolved, we have also dumped out "the baby with the bath-water", so to speak. We'd go two steps forward, and then one and three-quarters of a step back. Progress has been slow and difficult. Each new generation of programmers has rebuilt the same underlying foundations, often with only the slightest of improvements, and most of the old familiar problems.

To get around this, I think we need to be more honest about our technologies. Sure, they sometimes work and have great attributes, but ultimately we should not be afraid to explore their dark sides as well. All things have flaws, and technology often has more than most. In that, nothing should really be unassailable. If we close our minds, and pretend that it is perfect, we'll make very little progress, if any.

For this blog entry, I wanted to look at the Object-Oriented family of programming philosophies. They have a long and distinguished history, and have become a significant programming paradigm. Often, possibly because of their history, most younger programmers just assume that they work for everything, and that they are the only valid approach to programming. They clearly excel in some areas, but they are not as well rounded as most people want to believe.


EVOLUTION

Although it is hard to establish, the principles of Object-Oriented (OO) programming appear to be based around Abstract Data Types (ADT). The formalization of ADTs comes in and around the the genesis of the first Object-Oriented programming language Simula, in 1962. In particular, The Art of Computer Programming, by Donald Knuth which explores the then commonly known data structures, was also started in 1962 although the first volume wasn't published until 1968. Certainly, whatever the initial relationship between the two, they are very close, yet they are very different.

Abstract Data Types as a movement is about using data structures as the basis of the code. Around these data structures, the programmer writes a series of primitive, atomic functions that are restricted to just accessing the structure. This is very similar to Object-Orientation, except that in ADTs it is a "philosophy of implementation" -- completely language independent -- while in OO it is a fundamental abstraction buried deep within the programming language.

One can easily see that OO is the extension of the ADT ideas into the syntax and semantics of the programming languages. The main difference being that ADTs are a style of programming, while OO is a family of language abstractions.

Presumably, by embedding the ideas into the language, it makes it more difficult for the programmers to create unstructured spaghetti code. The language pushes the programmer towards writing better structured code, making it more awkward to do the wrong thing. The compiler or interpreter assists the programmer in preventing bad structure.

While ADTs are similar to OO -- you set up a data-type and then build some access functions to it -- at a higher level there is no specific philosophy. The philosophy only covers the low-level data structures, but absolutely nothing is said about the rest of the code. Practice, however is to use structured code to layout each of the high-level algorithms that are used to drive the functionality.

This means there is a huge difference between ADTs and OO. Outside of the data structures, you can fall back into pure procedural style programming in ADTs. The result is that ADT style programming looks similar Object-Oriented at the low-level, but is really structured as 'algorithms' at the high-level. A well-structured ADT program consists of a number of data-structures, the access functions and a series of higher-level algorithms that work the overall logic of the system. Since data drives most of the logic in most programs, the non-data parts of the code are usually control loops, interfaces to dispatch functionality or glue code to interfaces between different control loops. Which ever way, they can be coded in the simplest most obvious fashion, since there are no structural requirements.

That ambiguity in defining the style of the code at a high level in ADTs is very important. This means that there is a natural separation between the data and the algorithms, and they get coded slightly differently. But we'll get back to this later.


CODING SECRETS

Software development often hinges on simplifications, so it is not surprising that we really only want one consistent abstraction in our programming languages.

Because of this, we spend a lot of time arguing about which approach is better: a fixed language, that is strongly-typed or one that is loosely-typed. We spend a lot time arguing about syntax, and a lot of time arguing about the basic semantics. Mostly we spend a lot time arguing about whether or not the language should be flexible and loose, or restricted and strict. It is a central issue in most programming discussions.

If you come an impasse enough times, sooner or later you need to examine why you keep returning to the same spot. Truthfully, we work in two different levels with our implementations. At the higher level, the instructions are more important. At the low level it is the data.

At the low level we want to make sure the data that we are assembling is 'exactly' what we want.

When I was building a PDF rendering engine in Perl, for example, I added in an extra layer of complexity. The engine was designed to build up a complicated page as a data-structure and then traverse it, printing each element out to a file. This type of processing is really simple in a strongly typed language, but can get really messy in a loosely typed one. To get around Perl's loose semantics I wrapped each piece of data in a small hash table with an explicit type. With the addition of a couple of really simple access functions, this had the great quality of emulating a strongly typed syntax, both making the code easier to write, but also guaranteeing less errors. It was a very simple structuring that also made it really easy to extend the original code.

At the higher level we want to focus on the instructions and their order, data is not important. Batch languages and text processing tools are usually loosely typed because it makes more sense. The data is really minimally important, but the order of the functions is critical.

As another example, I was building a dynamic interface engine in Java. In this case, I wasn't really interested in what the data was, only that at the last moment it was getting converted into whatever data-type I needed for display or storage. This type of problem is usually trivial in a loosely-typed language, so in Java, I added in a class to 'partially' loosely type the data. Effectively, I was concerned with 'removing' via a single call, any of the attributes of the type of the data. It does not matter what the data is, only what it should be. Again it is a simple structuring, but one that effectively eliminated a huge amount of nearly redundant code and convoluted error handling.

With enough coding experience in different languages, you come to realize that being strict at a low-level is a great quality, but being flexible at the high level is also important. It is easy enough to get around the bias of the language, but it does count as some additional complexity that might prove confusing to other developers. It is no wonder that the various arguments about which type of language is superior are never conclusively put to rest, it is more than a simple trade-off. There is an inherent duality related to the depth of the programming. The arguments for and against, then are not really subjective as much as they are just trying to compare two completely different things.

Well-rounded implementations need both strong-typing and loose-typing. It matches how we see the solutions.


INTERNAL MODELS

Another fundamental problem we face, comes from not realizing that there are actually two different models of the solution at work in most pieces of software.

The way a user needs to work in the problem domain is not a simple transformation from the way the data is naturally structured. The two models often don't even have a simple one-to-one mapping, instead the user model is an internal abstraction that is convenient for the users, while the data model is a detail-oriented format that needs to be consistent and strictly checked.

A classic example is from a discussion between Jim Coplien and Bob Martin:

http://www.infoq.com/interviews/coplien-martin-tdd

Specifically Jim said:

"It's not like the savings account is some money sitting on the shelf on a bank somewhere, even though that is the user perspective, and you've just got to know that there are these relatively intricate structures in the foundations of a banking system to support the tax people and the actuaries and all these other folks, that you can't get to in an incremental way."

In that part of the conversation Jim talked about how we see our interaction with the bank as a saving account, but underneath due to regulatory concerns the real model is far more complex. This example does a great job of outlining the 'users' perspective of the software, as opposed to the underlying structure of the data. Depending on the usage, a bank's customers deal with the bank tellers operating the software. The customer's view of their accounts is the same one that the system's users -- the tellers -- need, even if underneath, the administrators and regulators in the bank have completely different perspectives on the data.

While these views need to be tied to each other, they need not be simple or even rational. The user's model of the system is from their own perspective, and they expect -- rightly so -- that the computer should simplify their view-point. The computer is a powerful tool, and one of its strengths is to be able to 'transform' the model into something simpler allowing the user to more easily interact with it, and then transform it back to something that can be stored and later mined.

It is a very common problem for many systems to have the software developers insist that there is only one 'true' way to look at the data. The way it needs to be structured in the database. It is not uncommon to see forms tools, generators or other coding philosophies that are built directly on the underlying data model.

This makes it possible to generate code and it cuts down on the required work, but rarely is the data model structured in the same way as the user needs. The results are expected. The user's can't internally map the data to their perspective, so the applications are extremely awkward.

A single model type of design works only for very simple applications where there is little mapping between the user model and the data model. Mostly that is sample applications and very simple functions. While there are some applications of this sort, most domain specific systems require a significant degree of complex mapping, often maintaining state, something the computer can and should be able to do.

The essence of this, is that the 'users' have a model that is independent of the 'data' model. E.g. The interface translates between the real underlying structure of the data, and the perspective that the user 'needs' to complete their work. More importantly the models are not one-to-one, or we would be able to build effective translators. This duality and inconsistent mapping is why code generators and forms tools don't work. You can't build a usable application from the data model, because it is only loosely tied to the user model.


OBJECT ORIENTED

So what does this have to do with Object-Oriented programming?

The structure of an Object-Oriented language makes it relatively easy to model the structure of the data in the system. At the lower data level, the semantics of the language allow it to be used to build complex systems. You create all the objects that are needed to model the data, or the user abstraction. Object-Oriented languages are at their height when they are encoding easily structured objects into the language.

The classic example is a GUI system where the objects in the system are mapped directly to the visual objects on the screen, that type of mapping means that when there are problems it is really easy to go back to the screen and find the errors. The key here, is that one-to-one link between the visual elements and the objects. The point of going to a lot of extra effort, is to make it easy to find and fix problems. You pay for the extra effort in structuring the OO code by getting reduced effort in debugging it.

At the higher level, all of the OO languages totally fall apart. We add mass amounts of the artificial complexity into the works just to be able to structure the problem in an Object-Oriented manner. Ultimately this makes the programs fragile and prone to breakage. Ideas such as inversion of control, and dependency injection are counter-intuitive to the actual coding problems. They become too many moving parts.

Now, instead of the problem being obvious, you may need to spend a significant amount of time in a debugger tracing through essentially arbitrarily structured objects. Constructs like Design Patterns help, but the essence of the implementation has still moved far away from the real underlying problem. The developer is quickly buried in a heap of technical complexity and abstractions.

If what you wanted was a simple control loop, a traversal or some hooks into a series of functions, it would be nice if the simplicity of the requirement actually matched the simplicity of the code.

Even more disconcerting, we must leave our primary language to deal with building and packaging problems. We use various other languages to really deal with the problems at the highest level, splitting the development problem domain into pieces. Fragmentation just adds more complexity.

Our build, package and deploy scripts depend on shell, ant or make. We could write the build mechanics in our higher level language, but writing that type of building code in Java for example, is so much more awkward than writing it in ant. If we see it that clearly for those higher-level problems, does it not seem as obvious for the higher-level in the rest of the system?

Another critical problem with the Object-Oriented model is the confusion between the user model, and data model and the persistent storage. We keep hoping that we can knock out a significant amount of the code, if can simplify this all down to one unique thing. The Object-Oriented approach pushes us towards only one single representation of the data in the system. A noble goal, but only if it works.

The problem is that in heading in that direction, we tend to get stingy with the changes. E.g the first version of the application gets built with one big internal model. It works, but the user's quickly find it awkward and start making changes. At different depths in the code, the two models start to appear, but they are so overlapped it is impossible to separate them. As integration concerns grow, because more people want access to the data, the persistence model starts to drift away as well.

Pretty soon there are three distinct ways of looking at the underlying business logic, but nobody has realized it. Instead the code gets patched over and over again, often toggling between leaning towards one model, and then back again towards another. Creating an endless amount of work. And allowing the inconsistencies tp create an endless amount of bugs.

If the architecture identifies and encapsulates the different models from each other, the overall system is more stable. In fact the code is far easier to write. The only big questions comes about with deciding the real differences between the user model and the data model. When a conflict is discovered, how does one actually know that the models should differ from each other or that both are wrong?


SUMMARY

Ironically, even though they are probably older and less popular, ADTs were better equipped for handling many implementation problems. Not because of what they are, but because of what they are not. Their main weakness was that they were not enforced by the language, so it was easy to ignore them. On the plus side, they solved the same low-level problem that Object-Oriented did, but allowed for better structuring at the higher level. It was just that it was up to the self-discipline of the programmer to implement it correctly.

I built a huge commercial system in Perl, which has rudimentary Object-Oriented support. At first I tried to utilize the OO features, but rapidly I fell back into a pure ADT style. The reason was that I found the code was way simpler. In OO we spend too much time 'jamming' the code into the 'right' way. That mismatch makes for fragile code with lots of bugs. If you want to increase your quality, you have to start with trying to get the 'computer' to do more work for you, and then you have to make it more obvious were the problems lay. If you can scan the code and see the bugs, it is far easier to work with.

We keep looking of the 'perfect' language. The one-size-fits-all idea that covers over both the high and low level aspects of programming. Not surprisingly, because they are essentially two different problems, we've not been able to find something that covers the problem entirely. Instead of admitting to our problems, we prefer to get caught up in endless arguments about which partial solution is better than the others.

The good qualities of ADTs ended up going into the Object-Oriented language design, but the cost of getting one consistent way of doing things was a paradigm where it is very easy to end up with very fragile high-level construction. Perhaps that's why C++ was so widely adopted. While it allowed OO design, it could still function as a non-OO language allowing a more natural way to express the solution. Used with caution, this allows programmers to avoid forcing the code into an artificial mechanism, not that anyprogrammers who have relied on this would ever admit it.

The lesson here I guess is that the good things that make Object-Oriented popular, are also the same things that make it convoluted to some degree. If we know we have this duality, then for the next language we design for the masses, we should account for this. It is not particularly complicated to go back to ADTs and use them to redefine another newer paradigm, this time making it different at the low and high levels. Our own need for one consistent approach seems to be driving our technologies into becoming excessively complicated. We simplify for the wrong variables.

We don't want to pick the 'right' way to do it anymore, we want to pick the one that means we are most likely to get the tool built. Is a single consistent type mechanism better than actually making it easier to get working code?

Fundamental Coding Issues

Everybody loves a short summary, one that can easily compress a complicated idea into a simple concept. They are hard to find, but always very useful.

In software development there are many complex issues surrounding our computer language 'code' and the various conventions we use for programming. There are lots of different attributes that a program can have beyond just working, but ultimately there are only a few things that easily define a great program. With that in mind, I find the following summary quite reasonable:

The secret to great code is to get the smallest program possible without resorting to being clever, while generalizing to as large a problem space as possible given the time constraints. Make all of the broad strokes explicit, while making all of the details implicit. Get this right, keep it clean and consistent, and you've got elegance.

It is simple, as it should be, but still the underlying parts require deeper explanation.

THE SMALLEST CODE BASE

Software developers don't like to admit it, but the amount of the code in a system is a hugely significant issue for all sorts of reasons.

Programming is a massive amount of work. If you look at the various parts of development: analysis, implementation, testing and deployment, the second one -- implementation, which is the act of writing the code to implement the solution -- doesn't appear all that large in the overall process. It is only one of four basic steps. However, despite appearances, the size of the code drives issues with all of the other work, including the analysis. The code is the anchor for everything else; bigger code means more work and bigger problems.

The problem starts with having to convert your understanding of the user's need into some set of interfaces. Complicated problems produce complicated interfaces which feed on themselves. It is far more work to add a new function to Microsoft Word, for example, then it is to add it to some really simple interface. The analysis doesn't just cover the problem space, it also includes how to adapt the proposed solution to a specific tool. Adding a function to a small tool is way less work than adding it into some huge existing monolith. Because of this,the analysis changes depending on the target infrastructure. More code means more work integrating any new functionality.

The size of the code itself causes its own problems. If your intended solution requires 150,000 lines of code you'd be looking at a small team of programmers for over a year. If your intended solution requires 1,000,000 lines of code it will take a huge team many years to build it. Once it is built, if there is a problem with the 150,000 lines of code, refactoring chunks of it is significant, but not dangerous to the project. With 1,000,000 lines you are committed to your code base for good or bad. It, like the Titanic is slow to turn, so if there are any obstacles along the way such as icebergs you are in serious trouble. Why use 1,000,000 lines if 150,000 will do?

With each line of code, you commit to something you've got to maintain throughout history. It requires work to keep it updated, it even requires work to delete it. The more code there is, the more work that is required to just figure out the available options. Big code bases are unwieldy beasts that are expensive and difficult to handle. Developers often try to get away with just bolting a bit more code onto the side, but that rather obvious tactic is limited and ugly.

In all code there are always latent unwanted behaviors (bugs), that includes mature code that has been running in production for years. These 'problems' sit there like little land mines waiting for unsuspecting people to pass by and trigger some functionality. Testing is the act of mine-sweeping, where the testers spend as much time as they have, trying to detect as many of these problems as possible. Most current test processes are horribly inefficient, they end up spending way too much time on the easy issues and way too little time on the hard ones. Once the time is up, the code is released in whatever state. So often you'll find that cord works well for its simple common usages, but becomes increasingly unstable the more you try to push its envelop. Not surprisingly, that matches the testing.

Testing follows some type of inverse square law, e.g. it likely takes four times as much effort to test twice as much code. By committing to a bigger code base you are committing to a huge increase in the amount of testing, or as more often the case, you are actually diminishing your existing testing by a significant degree. So often, the code base doubles but the testing resources remain the same, only now they are 25% as effective.

With the increase in testing requirements being ignored, most big software packages get more operational problems and issues. For all software, there is a considerable support cost even if it is just an installer, an operator, a simple help desk and some beepers for the programmers. For large commercial projects support can include an entire division of the company.

Running code is hugely expensive, most programmers don't seem to understand this. They just hand off their work and don't think about the consequences. There are the fixed costs, but also those driven because of behavioral problems. A particularly bad bug in a commercial setting could cost millions of dollars and really hurt the reputation of a company. Even in an in-house setting, the code going down could delay other significant events costing money or bad publicity. The more useful the code, the more expense the failures.

Bigger code 'will' fail more often. From experience it is clear that with twice as much code, comes twice as many bugs. Programmers have a nearly consistent rate of adding bugs to their code, that is mostly independent of the actual programmer or testing. Better programmers often have less bugs per line of code, but they still have them, and because they tend to work on the more complicated sections of the system or write more code, it is not uncommon for their bug count to be higher. It stands to reason, if there is twice as much code, there are at least twice as many bugs, so the odds of getting a bug is twice as likely.

Finally, bigger code bases also mean bigger algorithms, and a lot more of them. The more complex code is harder to use and has more bugs, but it also means more documentation work. Usually the 'rules' for any of the functional behavior of the system start to become really 'sophisticated'. Well, sophisticated isn't the right word, overcomplicated is probably more appropriate. Once the functionality bloats, it takes 'essays' to even explain the simplest usage of the system, the support costs go through the roof. A very simple program that does one thing that is strictly following the conventions in an obvious way, probably doesn't need any help or tutorials. Once you grow to do fancier tasks with lots of customization, then online help becomes a necessity. Open up the system to allow the users to adapt it to themselves then you need lots pf tutorials, courses and books. A whole industry springs forth from really complex software. Photoshop is the classic example, with a phenomenal secondary industry devoted to helping people utilize the internals by translating between language of the programmers, and the language of the users (such as "how do I remove red-eye from this photo?").


NOT CLEVER

Given all of the above problems with big code bases, programmers shouldn't rush out to compact their code into the tiniest possible pieces. Size is an issue, but readable code is more important. If you get too clever and really tightly pack your code it can be impossible to understand. Clever is bad. Very bad.

The obfuscated C code contest is a great example of really clever ways to dramatically alter or reduce the size of the code. While it is entertaining, in practice it is extremely dangerous. Clever code packs way too much complexity into too small a package. If we wrote things once and only once then never touched it, that would be fine, but the lifespan of code is huge. In its life, there are always periods were people make fast judgment calls on the functioning of the code. Code needs a constant amount of never-ending fixing and updating. Stress is a part of software because the amount of work is always larger than the resources. Clever code just sets itself up to cause problems in the future. It is yet another land mine waiting to happen, as if it were just a bug of some sort.

You get away with some clever trick of the language or some other weird way of getting your results, it will be easily missed by someone else. That code is dangerous, and definitely not elegant.

Since code is just complicated by its very nature. All problem spaces have huge amounts of complexity, but we really want to lay out each and every line of code in the simplest, most straight-forwardly reusable manner possible. That also includes, not commenting too much, as well as too little. Taking away the readability of code is always asking for trouble. If you can give it to someone with a very light coding background and they get 'it' immediately then it is probably close enough to elegant. Programming students, for example should be able to easily understand the nature and purpose of the code. You shouldn't need an extensive background to read good code, although you definitely need one to write it.

The formatting, names of the variables, comments, and all of the syntactic attributes are very important in producing something that is easy to read. In big languages such Perl, showing a tremendous amount of discipline in 'not' using some of the more esoteric features of the language is the industrial strength way of coding. In Java, not following weak design pattern and bean conventions will produce things that are more readable and easily understood. Obscuring the underlying functioning of the code by focusing on its structure, just makes it harder to understand. The magic comes from generalizing it, not from being clever.

Some of the 'movements' in programming over the years have been counter-productive because they get too obsessed about the 'right way' to realize that they are doing extreme damage to the 'readability' of their code base; a factor that is far more important than being right. If it is readable and consistent it can't be too far away from elegant.


LARGEST PROBLEM SPACE

When you are solving a problem, the range of possible solutions extends from pounding out each and every instruction using a 'brute-force' approach, to writing some extremely generalized configurable program to solve a huge number of similar problems. At the one end of this spectrum, using brute-force, there is a tremendous amount of work in typing and maintaining a very fragile and stiff code base. The solution is static and brittle, needing lots of fixes. While it is probably huge, each sub-section of it is very straight-forward as it is just a long sequence of instructions to follow, e.g. get this file, open it, read the contents, put them in this structure, add these numbers, do these manipulations, save them in this format, etc. Most computer languages provide some higher level of abstraction, so at least each instruction is millions of lines of assembler, but it is still rigid and explicit.

Adding more degrees of freedom to the instructions, generalizing or making it dynamic, means that the resulting code can be used to solve more and more similar problems. The size of the problem space opens up and with additional configuration information; the usage of the code becomes huge.

As we shift the dynamic behavior away from the static lines of code, we have to provide some 'meta-data' in order for the generalized version of the code to work on a specific problem. In a very philosophical sense, the underlying details must always exist, no matter how general the solution. When we generalize however, we shift the details from being statically embedded into the code in a fragile manner, to being, either implicit, or explicitly held in the meta-data effecting the behavior.

Like energy, the primary problem domain details* can neither be created nor destroyed. You just shift them around. They just get shifted from being explicitly embedded into the code to existing somewhere else, either implicitly or in the configuration data.

*We can and do create massive amounts of artificial complexity, which creates artificial details, which can be refactored out of the code. If you can delete details without altering the functionality, then it was clearly artificial.

Way way off to the very end of the spectrum, one might imagine some very complicated all-purpose general piece of code the can do everything, but funny enough, that 'code' exists and is the computer itself. It is the ultimate general solution.

Building and maintaining a system is a huge amount of work that is often underestimated. Usually wherever a specific software based tool can be used to solve a problem, there is an abundance of similar problems that also need to be solved. Programmers love to just bite off a piece and chew on that, ignoring the whole space, but it is far more effective to solve a collection of problems all at once. All it takes is the ability to step back and look at the specific problems in their larger context.

Which of the business rules seem to bend, and how many other problems in the company have the same feel to them? The larger the problem space, the cheaper the solution. If you have ten departments in a company that all need customized phone books, solving each one by itself is considerably more work than solving them all together. If you have one group that needs approvals for their documents, that problem spans a huge number of different groups. They will all benefit by a common solution.

Generalizing makes the solution cheaper, and the reduces the overall work. Also, strangely enough, generalized code is always smaller than brute force. That means that the long term costs of maintaining a code base are cheaper as well. The size issue on its own is significant enough that in a long run perspective it is always worth generalizing to bring down the size of the code, even if there are no available similar problems. Generalizing can reduce the code base enough to get it to fit in the available development window. The technique can be applied as a measure to control the costs of the project and build more efficiently.

It also helps in maintaining consistency. If there is one routine that is responsible for rendering multiple screens in a system, but virtue of its usage it enforces consistency.


TIME CONSTRAINTS

Everybody underestimates the amount of time it takes to build software. They also underestimate the amount of effort it takes to keep it running. Like an iceberg, only a small portion of the code is actually visible to the users so they see it as a rather small malleable thing that can be easily changed. Any significant set of functionality gets locked in by its own size and complexity.

If you make frequent quick changes to your code base you'll quickly find out that you are destabilizing it. The more fast hacks you add, the worse the code becomes. There is always an expensive trade-off to be made between getting it done quickly and getting it done correctly. Many projects make the wrong choices, although the damage often takes years before it fully shows. Getting out one good release is easy, getting one out time and time again is difficult.

Without a doubt, time is the biggest problem encountered in software. There is never enough. That means that any 'technique' that helps reduce the time taken to do some work is probably good if it helps both in the short run and the long run. Any technique that adds extra work is probably bad. We should always keep in mind that sometimes you need to add a little extra effort in the short run to save time in the long run.

For instance, keeping the code clean and neat makes it easy to add stuff later. A well-maintained project takes more time, but is worth it. Sloppy is the friend of complexity.

Optimization is always about doing something extra now, that results in a gain later because the results get re-used over and over. Clean code saves time in understanding and modifying it. A little bit of extra work and discipline that pay off. Design saves time from being lost later. With a solid design you can fit right into the code you need, you don't have to waste a lot of time wandering around trying to guess at what might work.

Time is an all important quantity in software development. Wasting it is a bad idea. Some of the newer programming development techniques seek to make 'games' out of programming. It is an immature way of trying to remove some of the natural tediousness away from coding. We want to build things and get to the end as fast as possible. Playing around is avoiding the issue.

If you want to build great tools, at times it will be painful, it comes with the territory. No job is 100% entertaining, they all have their down-sides, that is why it is called work and we have to be paid to do it. Nothing wrong with hobbyist programmer's playing games and competing, but its is not appropriate for the work place.

Testing is one area that people waste massive amounts of time without getting any additional benefit. Again, some of the newer testing techniques add extra work, but in exchange they claim to reduce the amount of bugs. If that works it is great, but you have to be sceptical of most claims. Component testing thoroughly is good, but if you cannot assure the interaction of the components with each other, then there is always some minimal level of final testing that still needs to be done.

It is immutable, and as such is not possible to be optimized away. If you must test the final product as a whole coherent piece, than you cannot skip those tests no matter how much work you have done on the sub-components. If you are not skipping the tests, then you are not saving any time. If you down-grade the final tests to be less, then you upgrade the risks of allowing problems through. Of course this is true as a basic principle: in-stream testing in any process is there to reduce the amount of bouncing around between states, but it does significantly bump up the amount of testing work without increasing the quality. If the bouncing between states is not significant, then reducing it doesn't add much extra value. It doesn't negate any of the final testing, which still needs to be done.

In same way that one cannot solve the halting problem, the amount of testing required to achieve 100% certainty that there are no bugs is 'infinite'. It is an easy proof, once you accept the possibility that there is some sequence of input that causes the software to get into a state that can break it. Given that all non-trivial software maintains some form of internal-state, then to achieve absolute certainty that there are no bugs you have have to test every combination of possible input. Since there is no limit to the size of the input, the number of possible test scenarios is infinite, and it would take forever to create and apply them. Given our restrictions on time, unless we do our development on the edge of a black hole, there will always be some possibly of bugs with any release.

You may put in significant effort for a series of releases to really produce stellar quality, but you always need to remember that software development projects never really end. Sooner or later a bug is going to escape. In practice it is always sooner, and it always happens way more than most programmers anticipate. Although one person called it 'defeatist', it is a really good idea to plan for sending out patches to fix the code. Just assume there will be bugs and deal with it. If you build this into distribution and deployment, then when it is necessary it is just part of the system, otherwise the 'crisis' will eat up huge amounts of time.

Unexpected, 'expected' events cause delays, morale issues and scheduling conflicts. When we continually see the same problems happen over and over again, we need to accept them as part of process, rather than try to ignore them or rail against how unfair they are. If we anticipate problems with the deployment of the software we can build in the mechanisms to make this dealing with the problem easier. Taking a broad approach to development to include the development, testing and operations into the problem domain is the best way to build practical solutions that can withstand significant shifts in their environments.

The time issues never go away and ignoring them only makes it worse and more disruptive.


BROAD STROKES AND DETAILS

There are lots of arguments between strongly typed and loosely typed languages. Different camps of programmers feel that one or the other is the perfect approach to the problem. After lots of experience on both sides, I've come to realize that some problems are best handled with strongly typed approaches -- while others are best handled with very loose ones. The difference comes down to whether or not you need to be pedantic about the details.

For some types of solutions, the underlying correctness of the data is the key to making it work. In those cases you want to build up internal structures while programming that are very specific, and during the process you want to insure that the correct structure is actually built.

If your writing some code to write out a specific file format, for example, you'd like the internal structure to mirror the file format. In that way, depending on the format the structure and syntax are highly restricted. As the various calls go about in the system building up the structure, there is also code to make sure that the structure stays consistent and correct. With many types of errors, the closer the program stops after the first error, the easier it is to diagnose. When running in a development environment, a program that is strongly typed and checking its data can stop immediately the moment it deviates from the prescribed plan. That makes it really easy to find and fix the problems which also helps to minimize testing.

For some solutions, if you are performing a large number of steps and you want the program to be as tolerant as possible, then loosely typed is way better. It doesn't matter for example, how the data got into the program, instead it matters how it will be transformed. High-level programs, scripting, data filters, and any other program that needs to deal with a wide range of unexpected inputs fit well into this type of circumstance. The range of the data is massive, but the transformations are pretty trivial. If the program stopped each and every time the input was an unknown combination, the code would be so fragile that it would be useless. Loosely typing data under this circumstance means that the importance is on the set of instructions, they need to be completed, the data is only secondary. Scripting in particular requires this.

This dichotomy is true for all software. For some goals the data is the most important thing and it needs to be structured correctly. For some goals, it is the sequence of instructions and the final output that is significant. It doesn't matter how it got there.

So for example, we can use a typed language like Java to perform the algorithmic combinations, but a basically untyped tool like ant to insure that the code was built and deployed correctly. That the shell scripting languages in Unix are mostly loosely typed is no accident. The two approaches are needed for two very different problems.

It is also true that within all systems, the sequence of instructions at the higher level is more important, and the structure of data at the low level is important. The nice part is realizing that the depth of the code makes a big difference. If you are building a complex system, the broad strokes of the code should be loosely typed, they are more flexible and less rigid that way, while the detailed calculations should be strongly typed because the accuracy for the data is often the key. Loose typing at the higher level also helps in decoupling the architecture and splitting off the presentation from the underlying data. All these things work together at the different levels within any system.

Clearly any new language that covers the whole domain of building and deploying complex systems will cover the whole spectrum and be both strongly and loosely typed. Working both into the same syntax will be one of those key things that will bump us forward in complexity handling.


ELEGANCE

Clean and consistent code is a great secret in getting things launched, but it is exceptionally difficult to get a group of programmers to follow some underlying conventions. There are cultural issues that make it nearly impossible for big teams to synchronize their working practices. That could be why the initial version of most of commercial software is built initially by little teams and later handed off to big teams for maintenance. We still don't know how to coordinate our efforts correctly on a large scale. Our most common official methodologies are horrible.

Elegance, just for the sake of elegance is never a great idea. Elegance because it makes you job easier and it means that the software works better is a great idea. It is even better when it takes away a lot of the stresses associated with messy programming habits. It becomes a means to an end.

We build and maintain tools to help our users solve their problems, which are most generally in playing with their ever-increasing piles of data. The most important thing is that the tools we build actually solve the problems for the users. Everybody getting their input into the design, programmers having fun while coding, and processes being left wide open and 'casual' may amuse some people while they are working, but they do not get the basic tasks completed any faster. Spending time to understand the user's real needs, keeping the code clean and consistent, using real graphic design for the interface, and providing simple tools that are easy to understand and effective are some of the things that make the users come to appreciate (or not) the underlying software. Computers can make people's lives easier or they can make them harder, the difference is up to the abilities of the programmers involved with the code.

In the end, a short simple solution that works simply and consistently is as good as it gets. Bad, overly complicated bloated software with every function imaginable isn't good, and it isn't much of an accomplishment. All the fancy graphics, dancing baloney and crammed in information can't hide a badly written program. It's not the technology, it's what you do with it that matters.