Get Serious

It is time we get serious about building software systems.

Small programs used for entertainment can be hastily written, but they do not scale up for industrial usage. Large organizations need massive amounts of computation that they can rely upon, both to keep up with the ever-changing times but also to master the dragons of complexity that we have unleashed.

So far, in software development, all we've been doing is idly tossing code at a sea of independent silos, while the more questionable people in our industry keep making fabulous claims about how their stuff, this time, will really fix everybody else's problems. The thing is, that each time we dump in more disjoint technology we're just making our problems worse, not better. Another silo is not a positive accomplishment.

It would be really nice if there was some new "magical" technology on the horizon which would rectify the mistakes of the past, but so far I haven't seen anything that is substantially different from our earlier offerings. If we don't change the focus we'll just perpetually re-invent the same whole house of cards that is already collapsing under the weight of our dysfunction.

To hopefully get us out of this self-imposed 'vicious cycle' we need a reasonable set of goals. This would allow us to make better choices along the way and give us some tangible indication of progress. With that in mind, I am suggesting seven basic requirements, or constraints, all driven from outside the technologies, that if met, would better position any new solutions to avoid repeating the past insanities.

Seven constraints for better software systems:

1. There is one and only one system per organization.

2. All of the data is in one logical data persistence location, and all of it is accessible from one general interface. It has global ACID properties as a single system (even if it is distributed across a massive number of machines).

3. There is one all covering general user interface, for everything. There can also be specialized interfaces for particular roles, but all of that functionality is available from the general interface.

4. Data and code partitioning is malleable, on-the-fly (access, security, visibility of both data and code).

5. Functionality is loosely bound to all of the user interfaces, all versions of all functionality are always available. Users can choose which version they want (prefer).

6. Almost everything works at near real time (fast enough to be usable and interactive). For any really slow computations, there is one simple mechanism to schedule work to be completed and users notified. The user can decide to have immediate or delayed access to any and all functionality.

7. Data and code organization is completely handled by the system itself, not by the users. They can name things in any way they want, but the things themselves are organized by the computer and thus are always up-to-date.

Each one of these constraints fits together to encapsulate behaviours that address well-known problems.

Having only one system per organization ensures that the data and functionality aren't siloed. It is potentially usable by all. Organizations are fluid arrangements, so this also means that you can easily split and merge systems. 'Easily' means that it doesn't take years or months. There is only one operations dept, that can get the full range of experience required from dealing with just one system. Expertise can be built up, not thinly spread across dozens of incompatible systems.

There must be one overall, consistent, fast and easy way to access absolutely every piece of data in the system through one programmatic interface (including logging and errors). Underneath, the data is likely distributed, shared and partitioned, but those physical attributes are encapsulated away. Thus at the data level there are no silos. This type of persistence solution, as will become apparent in later constraints, may share the same concepts of some of the modern technologies, but it won't really come into existence until we address it specifically. It will have ACID properties. It will be dependable.

Absolutely every single piece of new or old functionality should be accessible, fairly easily, from one complete overall general user interface. A general interface however would never be optimal for highly specialized users with deeper or more complex needs, so there would also be several more specific interfaces tuned to specific users, groups or roles. The overall interface insures that at the very least, all of the code and all of the data is always accessible to the application support roles, without having to do specific training on specialized interfaces. It might be slower to navigate around, but the consistency and accompanying organization would mean that someone, with the assistance of some domain experience, could reasonably view data, edit data or run some computation. This insures that shifts in the organization, and quirks of the specialized interfaces don't represent risks, but rather that administration might be slower at times.

Organizations change, code tends to be static. That causes an unstoppable amount of rust. Security, sub-groups and any number of access issues crop up. A system should be easily reconfigured in far less than a day, so that it can keep pace with the organization's structure. Re-orgs should not be hampered by technology; they may be unpleasant, but form an important part of preventing stagnation. The system needs to deal with what happens in real life, not just take a convenient way around it.

The programmers should be consistently building functionality, not playing hurry up and wait with the users. Some users will be early adopters, many won't. As such the users themselves should have some control on how they move forward utilizing any new functionality, which will continue to grow along with the organization (at a near exponential rate sometimes). Once tested the functionality should be available, but the older functionality must also continue to work correctly. This is a thorny problem, but it can be handled by decoupling the structure of data away from the data itself, in the same manner that we already decoupled the indexing of data away. In that sense, to make this works demands a primary structure and many secondary ones, all of which are usable by the code. The data should be restructurable on the fly, so that an early version of some functionality that depends on a given structure works whether it is primary or secondary. Also for any set of related data, some it can have one primary structure, while some of it can have an older one. This makes it easy to schedule batch processes to slowly convert the database over time, rather than having to commit to large downtimes. Given that the underpinnings can naturally support many versions, the users should have the ability to explicitly control which versions are bound to their interfaces. That is, one user might set their functionA to version 1.2 while another sets it to 3.4, and both should be able to access and add in similar data at the same time. At the same time the programmers could have moved on to implementing version 5.1. With that achieved the system really does decouple the differences in what the users are doing, from the software construction process.

Processing speed is important, but it is also important for the users to maintain control over their expected results. If some computation is slow, scheduling it to be completed in the background should be trivial and convenient. Any sort of uncertainties in time need to be removed to allow for the users to make better plans with regard to the computations. They need to be informed, kept abreast of the progress. We know that no matter how fast the hardware gets, there will always be slow computations, but so far we've been ignoring this.

The last constraint is one that I've been pondering for a while now. Allowing many users to organize assets gradually over time always seems to result in significant disorganization. Computers as a tool are most valuable when they automatically organize stuff for us. It is an incredibly complex issue, but I think it can be tackled. The idea would be similar to my earlier library example from my post in January called "Organization". As the sizes of the assets, code and data, get larger, the computer would automatically add in newer navigational indexing schemes, allowing the users to meaningfully name them, but not to create new arbitrary half-completed clumps of their own. Mostly this means sets, groups, hierarchies and indexing, on any number of dimensions with some limited combination of subsets. As the underlying size grows, the system would introduce new forms of organization, insuring that all of them are always 'complete' at all times.

The underlying essence of any trustable, effective system is a single well-defined architecture spanning the full organization that allows the data, its structure, and any necessary code to keep pace with the growth and changes that are natural. This constrained flexibility is necessary for any organization that wants to fully utilize the data that it has been collecting. We have most elements for this type of system buried in our technologies, but to my knowledge nothing this comprehensive actually exists, although many a software developer has dreamed of it late at night. There is nothing in this description that is known to be impossible, although some constraints directly confront difficult problems that we have been avoiding so far. It is possible to build something that matches these constraints, but it is also possible that we won't achieve this sort of system for many decades to come; both the software industry and our programming culture have other agendas. Still, modern organizations are crying out to stop the insanity and there has been a long history of people promoting non-existent solutions to these problems. The demand is there, the seeds of the technology are also there, we just haven't oriented ourselves in the right direction yet.

Flashlights

Imagine being lost in a wilderness of perpetual darkness. All you have is a small flashlight. The bumpy landscape is covered with all sorts of interesting, yet complex structures; some large, some tiny, some benign, some extraordinarily dangerous. As you thread your way around these obstacles, you can only see a temporary two-dimensional projection of the structures as illuminated by your flashlight's beam. You do your best to remember what you can about the landscape, but it's the limits of your own memory that throttle your ability to move about. If you try to move too fast, but your memory lets you down, the results can be fatal. So, you have to shine the light ahead first, then carefully maneuver.

You're not alone in this wilderness, there are lots of other people carefully making their way through this landscape as well. You can see their lights flashing here and there, but unless they are really nearby, their beams don't assist you, they are merely a curiosity. You're still on your own, most of the journey.

What could assist you is turning on the overhead lights. But unfortunately for you, there is no such thing. The only light comes from flashlights, nothing else.

The next best thing would be to somehow keep the glow of the surfaces going for a lot longer than just their appearance in a beam. That is, for quite a while after being illuminated by a flashlight, if the surface continued to emit light, with enough people shining their lights around, you could continuously highlight the structures well enough to allow everyone to move quickly and safely throughout the obstacles. All that would be needed is some form of light-absorbing paint that glows afterward, that was fairly long-lasting. In that sense, a technology that could collect a bit of the light, and then make that available for some time.

Knowledge, in a sense, is a collection of some pretty intricate structures. Given how much there is to know and how limited humans are, it is easy to draw similarities between what people collectively understand and a vast, mostly dark, landscape. Knowledge is of course not tangible, but it's not hard to see how having more of it would significantly make it easier to move around in this world. At all levels, we are constantly making decisions, but we only make them based on what we ourselves know -- what we can see in the flashlight beam directly in front of us. That then transpires into either very slow going, a lot of very poor choices or a rather significant reliance on luck, the latter of which is in shorter supply as the world gets more complex. What we need, is to aid ourselves but also everyone else trapped in this same landscape, with some type of light-emitting paint.

But knowledge isn't tangible, so our paint wouldn't be either. Fortunately, we have stumbled upon a technology that could, within the confines of this analogy, fit well into our need to conserve and share our world. Computers may be void of intelligence, but they have an intrinsic ability to remember stuff and to hopefully regurgitate it back at the right moment when it is useful. In a sense, they capture the light and continue to glow long afterward.

If we want to move around more conveniently than it is imperative that we capture and retain as much as we can. We need to chart out the briefly illuminated world into a form that is readily accessible to its inhabitants. A massive map that accurately reflects the structures of what we all individually have encountered; the knowledge that we have seen. The more light shone on this endeavor, the more accurate it will become. Once that map is widely available to all, the pace of our traversals will improve.

Getting back to the analogy, a little bit of glowing paint might help a select few, but the chains of darkness don't get fully broken until everything has been coated with paint. That type of massive effort isn't going to be accomplished by a small number of people, rather it takes everyone chipping in to ensure the benefits for all; leaving no dark zones left to trap the weary.

That being said, if we want to remove our current physical limitations and really improve our fate as a species, we have to leverage what we know collectively to make our world easily navigatable. We have the technology, now we just have to utilize it.

Special Case Considered Harmful

Without a doubt, the all-time quickest way to implement any new variability in an existing code base is to create a new branch with if/else statements, copy & paste the original code into its new block, and then modify the copy. You don't have to spend any time thinking. It's quick. It's initially painless. And it is one of the most popular ways to create code.

As a result of this, we often see programs that consist of layer upon layer of nested branches; sometimes four levels deep, sometimes worse. Advanced programmers avoid this by hiding these special cases in a mass of methods or functions.

Although the code itself gets larger, the testing of conditionals is usually cheap so there aren't any real performance penalties associated with this type of code. It may not be optimal, but it's not really slow either.

Testing does require more effort; forgetting to check the functionality of a newly added special case can easily mean that it gets released accidentally. If the branches are deeply nested, it can take considerable effort to unwind all of them them into a complete list of test cases. Most often that's not actually done, leading to bugs.

Extending the code is where the worst hit is taken. A few branches are not hard to deal with, but as they quickly build up, it becomes harder and harder to guess their behaviour just by reading the code. With lots of test cases necessary to cover the whole range of behaviour, it also becomes harder to understand when using a debugger. Thus you can't read it and it's no easy task to walk it either.

As the difficulties mount, the uncertainty grows -- probably exponentially -- until any new change to the code has a much better chance of being wrong then it does of working properly. At this point, since the code can no longer be safely changed, other worse options are usually selected.

At the root of this very common problem are special cases. If there is only one way in the code to handle the all of the data, then everything goes through the same logic and there is no need for any branches. The code is almost trivial.

Still, most people find it easier to solve big problems by breaking off a subset and then handling it in a 'special' way.

It's taught to us offhandedly in that we learn to solve problems by decomposing them. That's fine, except when that decomposition cuts across deeply related problems. Breaking up everything into special cases might be an effective way to understand how to go about solving the pieces, but the second half of that approach -- that isn't taught -- is to bring it all together again for the whole solution.

Special cases are often highly favored in many domains, thus the domain experts and analysts are always a significant source of them entering into the project. Big systems get built by going after the low hanging fruit first, then picking off the lessor cases one by one. This insures that progress is visible right from on onset of the project.

Thus the proliferation of special cases is fast, easy and we're often led to believe that it is the correct way to handle complex problems. That this is then mapped directly into deeply nested code or a quagmire of functions is not really a surprise. But it doesn't have to be that way.

One of the most visible aspects when reading any elegant code is that it does not have either an excessive number of branches nor too many loops. The code is organized, the flow through it is clear and the special cases few or non-existent. Clarity comes from the level just above the instructions, done well it keeps the logic from spewing off into a myriad of spaghetti-like flows.

From those rare examples, we can easily conclude that minimizing branches is both good and definitely worthwhile if we are intent on having the code continue to expand. The follow up questions then are "how?" and "what's the minimum number of branches?"

Long ago, when I first started programming, there was always a lot of effort put into simplifying coding logic. The earlier languages offered little guidance in structure, so the programmers had to spent more time back then, trying to work out appropriate ways to keep it from getting out of control. Object Oriented programming, and indirectly design patterns, embedded some of that organization directly into the languages themselves, but at the expense of the newer generations of programmers no longer understanding why well-structured code was important. The emphasis shifted to gluing stuff together as fast as possible. That might work for consolidating clumps of existing technologies, but it's an unmanageable nightmare when applied to a sea of special cases coming in from the domain side.

What's necessary is to avoid implementing special cases wherever and whenever possible. And to cleanup some of the existing mess, it is critical to merge back any existing special cases into the mainstream. Fortunately special cases are easy to spot since they exist as branches, and they are easy to remove, you just get rid of the if/else statements and merge the code together.

This does require some thought and practice, but it is not rocket science and it becomes easier to do as you gain more experience.

In the really easy cases, you can drop a branch when the difference between the two underlying blocks of code is only a variable or two. That is, you create new variables, assign them values at the beginning and then perform the sequence of actions, passing in the variables where appropriate. In the worst case, you might have to go back aways through the code, adding back the variables right up until the data loaded. I'll get back to that approach later.

For some cases, the difference is actually contained in a larger set of variables. This is handled the same way as individual variables, but with a structure or object instead of a single variable. Keeping related variables together and building up complex structures to pass around is a strong approach to organizing any code.

To get rid of some cases, the lists of executed instructions are slightly different in each case. There are some subtle ways of handling this, but they usually involve either table-driven programming or an intermediate data-structure. The idea is that you build up data that lists the instructions, varying by each case, as you go. Once the structure is complete, you walk through it executing everything. Simple lists fit into statically created tables, while more complex logic can be dynamically created as lists, trees or even DAGs. You'll find this type of code at the heart of interpreters, compilers, editors, graphics programming and some really beautiful domain applications. The if statements are implicitly hidden because of the way the structure was constructed. This is a powerful way to eliminate special cases and to control complexity, while still making it almost trivial to extend the code. For static tables you just add in new rows, for intermediate representations you just add more functionality to manipulate the structure.

That of course leads to employing polymorphism in OO languages. The special cases disappear because everything is handled by overrides of basic primitive methods that are specific to each class. That is, the blocks in the if/else statements form the methods implemented in the objects. Then you get a 1:1 relationship between the smallest blocks needed and the primitives necessary to solve the problem. Set up correctly, expanding a polymorphic hierarchy requires as little thinking as creating deeply nested code (which generally means you have to be careful) but the end result is still organized. To go a bit farther, you can even reuse common code blocks via inheritance, which saves on typing and testing.

In particullary messy systems where there are an explosive number of permutations that all might need to be handled someday, polymorphism is a really strong approach. Even before OO we'd implement the same mechanics in C by using table-driven function pointers that implemented the primitives. Then to change processing for different sets of data or configuration all you needed to do was update the table with a new set of function pointers, thus making it easy and reliable to extend to new cases. Setting up that type of organization doesn't take much time, but it saves massive amounts later as the code grows.

Sometimes the special cases are very very hard to remove. For that we have to go up to abstraction, and then manufacture virtual elements to smooth out the logic. This is a tricky technique to explain in general terms but the basic idea is to add in things that doesn't really exist, but if they did then there wouldn't be any special cases remaining. A very simple example of it exists in the "The Art of Computer Programming" where Donald E. Knuth documents a couple of  algorithms for linked lists that always have a head node (circular and doubly linked). That is, the list is never empty, even if there is nothing in it. This simple trick reduces the logic down to something quite manageable and provides several neat little micro optimizations. It's a very small "space vs. time" trade-off that's unfortunately been mostly forgotten.

I've had to use this abstraction technique many times in the past to simplify difficult, yet performance dependent logical flows. The different cases are tantalizingly close to each other, but they don't actually fit together. It can take a bit of creativity and patience, but eventually you stumble onto some virtual piece that ties it all into one. The key is that the code actually gets simpler, if it doesn't then you need to toss it immediately and try again. Almost always it has taken a couple of iterations to get right, but in the long run it has always been worth it.

If you look carefully, you'll see this technique in all sorts of interesting places, but because it rounds out the code so nicely it is easy to miss. Still, if you really want to write something sophisticated, it is an indispensable part of the your toolkit.

Sometimes branches are just unavoidable, but that doesn't mean they have to gum up the code. Like limiting scope in variables, you can push a branch down to the absolute lowest point in the code. It wraps only the absolute minimum instructions that differ. Doing that is often enough to get it encapsulated away from the rest of the complexity. Another strong way to handle stubborn branches is to move them to the earliest part of the code where the data comes in. It has always been a good idea to only convert and verify data 'once' upon entry from an external source, thus avoiding needless fiddling and redundant checks. This is also a great place to add in variables that avoid branching. Placed there, the branch can set up a variable that flows cleanly through the rest of the main logic.

Most special cases are unnecessary. There are the very few rare ones that are completely unavoidable, but long before accepting any new special cases into that category all effort should be made force fit them into the mainstream properly. Special cases match the way people think and they get created quite naturally from various domains, programmers and the analysis process. That is to be expected, so it is up to the system's designers and its programmers to merge them back into as few cases as possible. Although it's easy to just whack out more special cases in a code base, it is one of the fastest and most dangerous forms of technical debt. Intrinsically each one is bump up in disorganization. It's also mindless, and in software anytime something isn't thought about explicitly, it usually manifests itself into a problem. We got into software development because we like to solve problems by thinking deeply about them, oddly trying not to think deeply about them is counter to what most people claim they enjoy.

Shades of Grey

Every profession changes its practitioners. How could it not? They'll spend a big part of their life approaching problems from a given angle, applying knowledge built up by others with that same perspective. Prolonged immersion in such an environment gradually colours their views of the world.

Programmers encounter this frequently. Coding is all about engaging in a dialogue with a rigid machines that don't tolerate ambiguity or understand intentions. They do exactly, and precisely, what we tell them to do. There is no greyness, no magic; the machines only follow their instructions as given and nothing else (if we ignore issues like faulty hardware and viruses).

This constant rigidity leads us to frame our instructions as being 'right' or 'wrong'. That is, if the code compiles and runs then it is right, else it is wrong. There is nothing in between.

Gradually this invades how we see things. We quickly go from the idea that the instructions are right, to the idea that we've built the right thing for the users, and then we talk about the right ways to design systems and run projects. But for many, it doesn't just stop there. It continues to propagate outwards, affecting the rest of their views of the world. And the way they interact with it.

A long, long time ago, on learning the basis of first learning object oriented, one of my friends declared "There can only be 'one' right object oriented design. Everything else is wrong". I giggled a bit when I heard this, but then I went on a rather prolonged ramble about the greyness of trade-offs.

I explained that known issues like space-time trade-offs meant that there were actually a great number of different, yet equally valid designs, all of which should be considered 'right'. That some of those trade-offs might be constrained by known user or operational requirements, but that there were many more that differed only because of 'free' parameters.

My friend was sceptical at first, but he came around when I began talking about different 'sorting' algorithms. Ones with very different computational complexities in the best, average and worst categories. They all sorted, quite efficiently, but the whole collection of behaviours was considerably varied. Optimizing for any one aspect of a complex system always offsets some other aspect. That is, one is traded off for the other.

The simplest example of this is 2D vectors on a Cartesian plot. If magnitude is the only constraint, then there are an infinite number of vectors that are all basically identical. It doesn't matter where they are on the plot, or at what angle. A single constraint in n variables maps lots of stuff onto each other.

Getting back to software, it's discrete, so there is actually a fixed set of 'good' object oriented designs, but it is a really huge set. Outside of that set there is a much larger one of all of the less than correct designs. But that still doesn't make any one specific design 'right', it just makes it one of the many 'appropriate' solutions. "Right" is definitely the wrong word to use in this context

When I started programming, the big decision we had to make was whether we were 'vi' or 'emacs' people. Both editors requires significant effort to learn to use correctly. Emacs was seen as the grander solution, but vi was pretty much available everywhere and consumed a lot less resources. On the early internet, wars raged between the proponents of both, each pounding on their keyboards about the rightness of their view. That of course is madness, editors are just a tool; the different varieties suite different people, but almost by definition there is no one perfect tool.

There is no 'right' editor. There are certainly some crappy editors out there, but they are invalid only because they are plagued with bugs, or are missing core functionality. They are deficient, rather than 'wrong'. If you can do the basics of editing text, then an editor is functional.

In that same sense, we can talk about lines of code being right, in that they actually compile, and we can talk about compiled code being right in the sense that it actually runs, and perhaps we could include 'proofs of correctness' to show that the code does what we think it does but beyond that it quickly becomes grey. There is no right program for the users. There is no right architecture. No right way to build systems. As we get farther away from the code, the shades of grey intensify. Right and wrong quickly become meaningless distinctions.

A quick perusal of the many discussions on the web about computers out there are predicated on this mistake. People argue in general about how one end of a trade-off is so much better than the other. But it doesn't make sense, and it siphons away energy from more meaningful discussions.

A black and white perspective comes with the early stages of learning to program. It is very easy to let those views cloud everything else. I fell into this trap for a long time. Gradually, I have been trying to correct my views. Right and wrong is far too rigid to use in being able to make sense of the world. It certainly doesn't help us understand how to interact with people or to come together and build sophisticated computer programs. It doesn't make life any easier, nor does it make anyone happy. It's sole purpose is to help down at the coding and debugging level, but once you've learned its place, you have to learn to leave it where it belongs.

The safest thing to do, is to not use the words 'right' or 'wrong' at all. They tend to lead towards missing the essential nature of the underlying trade-offs. It's far better to come at complex systems and environments with an objective viewpoint. In that way, any specifics might have better qualities, at the expense of known or unknown side-effects.

Another easy example is with computer languages. Some languages make doing a task faster and more understandable, but all of them are essentially equivalent in that they are Turing-complete. For a given problem, it might be more naturally expressed in Lisp, but it also might be way easier to find Java programmers that could continue to work on the project. The time saved in one regard, could be quite costly in another.

More to the point, the underlying paradigms available as primitives in Lisp could be reconstructed in Java, so that the upper-level semantics is similar. That is, you don't necessarily have to just do lispy things only in Lisp, they are transportable to any other programming language, but you have to take one step backwards from the language in order to see that. And you can't be objective if you are dead-set against the inherent wrongness of something.

Putting that all together, although at the lowest levels we need to view things in a black and white context, at the higher levels labelling everything as right and wrong is debilitating. If you're discussing some complex aspect of software development and you use the word 'right', you're probably doing it wrong.

Types of Data

The primary goal of any software system is to collect data.

Some of that data represents entities in the real world, it is raw information that is directly captured from the domain. Most systems store this raw data as simply as possible, although it usually gets re-keyed to align with the rest of the system, and sometimes the structure is rearranged. Usually it's preserved in its rawest format, so that it is easier to double check its correctness, and update it later if there are corrections.

Raw data accounts for the bulk of most data in the system, and it is usually at the core of allowing the functionality to really solve problems for the user. In most domains, it comes as a finite set of major entities whose structure is driven by the real world collection techniques involved in getting it.

Raw data has breadth or depth, and occasionally both. By breadth, I mean that there are a large number of underlying parts to it, such that if the data were stored in a normalized schema, there would be a huge number of different, interconnected tables. Healthcare is an example of a domain that has considerable breadth, given that there are a lot of different types of diseases, diagnostics and treatments.

By depth, I mean that the same entities are repeated over and over again, on masse. Intraday stock data is an example of depth. On an entity by entity basis, both breadth and depth usually don't exist together, although some domains are really combinations of the two. Bonds for example can have a large number of different flavors, and they also have lots of different intraday valuations. But the master data is always a different entity than the daily quotes.

Mostly depth tends to be static with regard to structure. Some domains have entities of static breadth, but more often than not the breadth is dynamic. It changes on a regular basis, which is often the reason it has so much substructure in the first place. Dynamic breadth can be contained via abstraction, but it becomes tricky for most people to understand and visualize the abstract relationships, so it isn't common practice. Instead there is continual scope creep.

Since computers can conveniently augment the original data, systems usually have a fair amount of derived data. Sometimes this data is calculated by extending the relationships, sometimes it is just basic summary information. Derived data is more often calculated on-the-fly, usually to avoid staleness and inconsistency. Some derived data is composed entirely of calculations using raw data. Some derived data is built on other derived values. Visualizing the dependencies as a directed graph, it is always true that all terminal nodes are raw data. That is, what goes into the calculations must come from somewhere.

The computations used are often called 'business logic', but some of the more esoteric ones really encompass the irrationality of historic conventions. Derived data is most often fast to calculate, but there are plenty of domains where it can take hours, or even perhaps days. Those types of big calculations are usually stored in the persistence technology, and as such are sometimes confused with raw data.

Raw data is often self-explanatory in terms of its structure as stored persistently. Derived data however usually requires deeper explanations, and in most systems is the predominate source of programming bugs. Not only do the programmers need to understand how it is constructed but the users do as well, and as a consequence of this most derived data calculations are either reasonably straight-forward or they are created from a component built and maintained by external domain experts who are authoritative references.

In well-organized designs, the derived data calculations are all collected together in some consistent subsystem. This makes it easier to find and correct problems. If the calculations are scattered throughout the code base, then inconsistencies are common and testing is very expensive.

The underlying calculations can and should be implemented in a stateless manner. That is all of the required data should be explicitly passed into the calculations. In some domains, since the underlying data triggers branches, it is not uncommon to see the code inverted. As the calculation progresses, the branches cause auxiliary data to be gathered. Thus if the data is type A, then table A is loaded and if it is B, then table B is loaded. This type of logic is easy to build up, but it complicates the control and error handling in the code, it also diminishes the reusability.

It takes a bit of thinking, but virtually all derived calculations can be refactored to have the necessary data passed in as the initial inputs. In that way, all valid rows from both table A and B are input, no matter what the underlying type. It is also true that the performance of coding it this way will be better. In a big system, the branching won’t gradually degenerate into too many sub-requests. Given this, derived data should always be implemented as a set of straight-forward functions that are stateless and entirely independent from the rest of the system. There should be no side-effects, no global access and no need for external interactions. Just a black-box engine that takes data and returns derived results. Getting derived data fully encapsulated makes it really easy to extend, and deploy in a wide variety of circumstances. In the rare circumstance where this is not possible for some tricky special cases, the rest of the derived data in the system should still be implemented correctly (one small special case does not excuse ignoring  best practices).

The third type of data in a system is presentation data, but given the above definitions of raw and derived, initially it might not seem like there is a lot of data in this category. However it is surprisingly large. Presentation is primarily about how we dress up data for the screen, but it also implicitly includes the navigational information about how the user got to the screen in the first place. Thus we have decorative data, like font choice, colors, etc. but we also have things like session ids, the current primary data key and any of the options that the user has set or customized. In most professional systems that actually includes a lot of user, group and system preferences, as well as authentication and authorization. Basically it’s anything that isn't raw, or derived from the raw data, which in modern systems is considerable.

Presentation data has often confused programmers, particularly with respect to patterns like model-view-controller (MVC). There is a tendency to only think of the model as the raw data, then implement the derived data as methods, and leave out the presentation data all together. If done this way, the construction generally degenerates into a mess. The model should be the model for all of the data in the whole system, and act as a 1:1 mapping between main entities and objects. Thus the model would contain sets of objects for all three types of data, which would include, for example, an object that encapsulates the current navigational information for a given user session. It would also include any fiddly little switches on the screen that the user has selected.

In most cases, the objects in the model would be nouns and their interactions would be equivalent to the way that the user describes what is happening at an interface level. The model, in this sense, is essentially object wrappers for any and all structural data within the system, at any time. In most cases, one can stop at the main entity level or one below, not all substructural data needs to be explicitly encapsulated with objects, but the depth should be consistent. Creating the full model for the system means that it is well-defined where to put code like validation and decoration. It also means that there is another 1:1 mapping between the widgets in the UI and the model. This again keeps the code organized and clearly defines only one place for any new logic.

An organized system is one where there is only one valid location for any of the code, and a well constructed system is where everything is exactly where it should be.

There are really only three types of data in a system: raw, derived and presentation. These percolate throughout the code. Most code in large systems is about moving the three types data around between the interface and the persistence technology. If the three types are well-understood, then very straight forward organizational schemes can be applied to keep them encapsulated and consistent. This schemes considerably reduce the amount of code needed, saving time and reducing testing and bugs. Most programs don't have to be as complicated as they have become, not only are they hard to extend, but it also decreases their performance. If we step back from the code, we are able to see these similarities and leverage them with simple abstractions and consistent rules. Minimal code is directly tied to strong organization, that is spaghetti code is always larger than necessary and usually disorganized on multiple different levels. If the data management is clean, the rest of the system cleanly falls into place.