Everybody loves a short summary, one that can easily compress a complicated idea into a simple concept. They are hard to find, but always very useful.
In software development there are many complex issues surrounding our computer language 'code' and the various conventions we use for programming. There are lots of different attributes that a program can have beyond just working, but ultimately there are only a few things that easily define a great program. With that in mind, I find the following summary quite reasonable:
The secret to great code is to get the smallest program possible without resorting to being clever, while generalizing to as large a problem space as possible given the time constraints. Make all of the broad strokes explicit, while making all of the details implicit. Get this right, keep it clean and consistent, and you've got elegance.
It is simple, as it should be, but still the underlying parts require deeper explanation.
THE SMALLEST CODE BASE
Software developers don't like to admit it, but the amount of the code in a system is a hugely significant issue for all sorts of reasons.
Programming is a massive amount of work. If you look at the various parts of development: analysis, implementation, testing and deployment, the second one -- implementation, which is the act of writing the code to implement the solution -- doesn't appear all that large in the overall process. It is only one of four basic steps. However, despite appearances, the size of the code drives issues with all of the other work, including the analysis. The code is the anchor for everything else; bigger code means more work and bigger problems.
The problem starts with having to convert your understanding of the user's need into some set of interfaces. Complicated problems produce complicated interfaces which feed on themselves. It is far more work to add a new function to Microsoft Word, for example, then it is to add it to some really simple interface. The analysis doesn't just cover the problem space, it also includes how to adapt the proposed solution to a specific tool. Adding a function to a small tool is way less work than adding it into some huge existing monolith. Because of this,the analysis changes depending on the target infrastructure. More code means more work integrating any new functionality.
The size of the code itself causes its own problems. If your intended solution requires 150,000 lines of code you'd be looking at a small team of programmers for over a year. If your intended solution requires 1,000,000 lines of code it will take a huge team many years to build it. Once it is built, if there is a problem with the 150,000 lines of code, refactoring chunks of it is significant, but not dangerous to the project. With 1,000,000 lines you are committed to your code base for good or bad. It, like the Titanic is slow to turn, so if there are any obstacles along the way such as icebergs you are in serious trouble. Why use 1,000,000 lines if 150,000 will do?
With each line of code, you commit to something you've got to maintain throughout history. It requires work to keep it updated, it even requires work to delete it. The more code there is, the more work that is required to just figure out the available options. Big code bases are unwieldy beasts that are expensive and difficult to handle. Developers often try to get away with just bolting a bit more code onto the side, but that rather obvious tactic is limited and ugly.
In all code there are always latent unwanted behaviors (bugs), that includes mature code that has been running in production for years. These 'problems' sit there like little land mines waiting for unsuspecting people to pass by and trigger some functionality. Testing is the act of mine-sweeping, where the testers spend as much time as they have, trying to detect as many of these problems as possible. Most current test processes are horribly inefficient, they end up spending way too much time on the easy issues and way too little time on the hard ones. Once the time is up, the code is released in whatever state. So often you'll find that cord works well for its simple common usages, but becomes increasingly unstable the more you try to push its envelop. Not surprisingly, that matches the testing.
Testing follows some type of inverse square law, e.g. it likely takes four times as much effort to test twice as much code. By committing to a bigger code base you are committing to a huge increase in the amount of testing, or as more often the case, you are actually diminishing your existing testing by a significant degree. So often, the code base doubles but the testing resources remain the same, only now they are 25% as effective.
With the increase in testing requirements being ignored, most big software packages get more operational problems and issues. For all software, there is a considerable support cost even if it is just an installer, an operator, a simple help desk and some beepers for the programmers. For large commercial projects support can include an entire division of the company.
Running code is hugely expensive, most programmers don't seem to understand this. They just hand off their work and don't think about the consequences. There are the fixed costs, but also those driven because of behavioral problems. A particularly bad bug in a commercial setting could cost millions of dollars and really hurt the reputation of a company. Even in an in-house setting, the code going down could delay other significant events costing money or bad publicity. The more useful the code, the more expense the failures.
Bigger code 'will' fail more often. From experience it is clear that with twice as much code, comes twice as many bugs. Programmers have a nearly consistent rate of adding bugs to their code, that is mostly independent of the actual programmer or testing. Better programmers often have less bugs per line of code, but they still have them, and because they tend to work on the more complicated sections of the system or write more code, it is not uncommon for their bug count to be higher. It stands to reason, if there is twice as much code, there are at least twice as many bugs, so the odds of getting a bug is twice as likely.
Finally, bigger code bases also mean bigger algorithms, and a lot more of them. The more complex code is harder to use and has more bugs, but it also means more documentation work. Usually the 'rules' for any of the functional behavior of the system start to become really 'sophisticated'. Well, sophisticated isn't the right word, overcomplicated is probably more appropriate. Once the functionality bloats, it takes 'essays' to even explain the simplest usage of the system, the support costs go through the roof. A very simple program that does one thing that is strictly following the conventions in an obvious way, probably doesn't need any help or tutorials. Once you grow to do fancier tasks with lots of customization, then online help becomes a necessity. Open up the system to allow the users to adapt it to themselves then you need lots pf tutorials, courses and books. A whole industry springs forth from really complex software. Photoshop is the classic example, with a phenomenal secondary industry devoted to helping people utilize the internals by translating between language of the programmers, and the language of the users (such as "how do I remove red-eye from this photo?").
NOT CLEVER
Given all of the above problems with big code bases, programmers shouldn't rush out to compact their code into the tiniest possible pieces. Size is an issue, but readable code is more important. If you get too clever and really tightly pack your code it can be impossible to understand. Clever is bad. Very bad.
The obfuscated C code contest is a great example of really clever ways to dramatically alter or reduce the size of the code. While it is entertaining, in practice it is extremely dangerous. Clever code packs way too much complexity into too small a package. If we wrote things once and only once then never touched it, that would be fine, but the lifespan of code is huge. In its life, there are always periods were people make fast judgment calls on the functioning of the code. Code needs a constant amount of never-ending fixing and updating. Stress is a part of software because the amount of work is always larger than the resources. Clever code just sets itself up to cause problems in the future. It is yet another land mine waiting to happen, as if it were just a bug of some sort.
You get away with some clever trick of the language or some other weird way of getting your results, it will be easily missed by someone else. That code is dangerous, and definitely not elegant.
Since code is just complicated by its very nature. All problem spaces have huge amounts of complexity, but we really want to lay out each and every line of code in the simplest, most straight-forwardly reusable manner possible. That also includes, not commenting too much, as well as too little. Taking away the readability of code is always asking for trouble. If you can give it to someone with a very light coding background and they get 'it' immediately then it is probably close enough to elegant. Programming students, for example should be able to easily understand the nature and purpose of the code. You shouldn't need an extensive background to read good code, although you definitely need one to write it.
The formatting, names of the variables, comments, and all of the syntactic attributes are very important in producing something that is easy to read. In big languages such Perl, showing a tremendous amount of discipline in 'not' using some of the more esoteric features of the language is the industrial strength way of coding. In Java, not following weak design pattern and bean conventions will produce things that are more readable and easily understood. Obscuring the underlying functioning of the code by focusing on its structure, just makes it harder to understand. The magic comes from generalizing it, not from being clever.
Some of the 'movements' in programming over the years have been counter-productive because they get too obsessed about the 'right way' to realize that they are doing extreme damage to the 'readability' of their code base; a factor that is far more important than being right. If it is readable and consistent it can't be too far away from elegant.
LARGEST PROBLEM SPACE
When you are solving a problem, the range of possible solutions extends from pounding out each and every instruction using a 'brute-force' approach, to writing some extremely generalized configurable program to solve a huge number of similar problems. At the one end of this spectrum, using brute-force, there is a tremendous amount of work in typing and maintaining a very fragile and stiff code base. The solution is static and brittle, needing lots of fixes. While it is probably huge, each sub-section of it is very straight-forward as it is just a long sequence of instructions to follow, e.g. get this file, open it, read the contents, put them in this structure, add these numbers, do these manipulations, save them in this format, etc. Most computer languages provide some higher level of abstraction, so at least each instruction is millions of lines of assembler, but it is still rigid and explicit.
Adding more degrees of freedom to the instructions, generalizing or making it dynamic, means that the resulting code can be used to solve more and more similar problems. The size of the problem space opens up and with additional configuration information; the usage of the code becomes huge.
As we shift the dynamic behavior away from the static lines of code, we have to provide some 'meta-data' in order for the generalized version of the code to work on a specific problem. In a very philosophical sense, the underlying details must always exist, no matter how general the solution. When we generalize however, we shift the details from being statically embedded into the code in a fragile manner, to being, either implicit, or explicitly held in the meta-data effecting the behavior.
Like energy, the primary problem domain details* can neither be created nor destroyed. You just shift them around. They just get shifted from being explicitly embedded into the code to existing somewhere else, either implicitly or in the configuration data.
*We can and do create massive amounts of artificial complexity, which creates artificial details, which can be refactored out of the code. If you can delete details without altering the functionality, then it was clearly artificial.
Way way off to the very end of the spectrum, one might imagine some very complicated all-purpose general piece of code the can do everything, but funny enough, that 'code' exists and is the computer itself. It is the ultimate general solution.
Building and maintaining a system is a huge amount of work that is often underestimated. Usually wherever a specific software based tool can be used to solve a problem, there is an abundance of similar problems that also need to be solved. Programmers love to just bite off a piece and chew on that, ignoring the whole space, but it is far more effective to solve a collection of problems all at once. All it takes is the ability to step back and look at the specific problems in their larger context.
Which of the business rules seem to bend, and how many other problems in the company have the same feel to them? The larger the problem space, the cheaper the solution. If you have ten departments in a company that all need customized phone books, solving each one by itself is considerably more work than solving them all together. If you have one group that needs approvals for their documents, that problem spans a huge number of different groups. They will all benefit by a common solution.
Generalizing makes the solution cheaper, and the reduces the overall work. Also, strangely enough, generalized code is always smaller than brute force. That means that the long term costs of maintaining a code base are cheaper as well. The size issue on its own is significant enough that in a long run perspective it is always worth generalizing to bring down the size of the code, even if there are no available similar problems. Generalizing can reduce the code base enough to get it to fit in the available development window. The technique can be applied as a measure to control the costs of the project and build more efficiently.
It also helps in maintaining consistency. If there is one routine that is responsible for rendering multiple screens in a system, but virtue of its usage it enforces consistency.
TIME CONSTRAINTS
Everybody underestimates the amount of time it takes to build software. They also underestimate the amount of effort it takes to keep it running. Like an iceberg, only a small portion of the code is actually visible to the users so they see it as a rather small malleable thing that can be easily changed. Any significant set of functionality gets locked in by its own size and complexity.
If you make frequent quick changes to your code base you'll quickly find out that you are destabilizing it. The more fast hacks you add, the worse the code becomes. There is always an expensive trade-off to be made between getting it done quickly and getting it done correctly. Many projects make the wrong choices, although the damage often takes years before it fully shows. Getting out one good release is easy, getting one out time and time again is difficult.
Without a doubt, time is the biggest problem encountered in software. There is never enough. That means that any 'technique' that helps reduce the time taken to do some work is probably good if it helps both in the short run and the long run. Any technique that adds extra work is probably bad. We should always keep in mind that sometimes you need to add a little extra effort in the short run to save time in the long run.
For instance, keeping the code clean and neat makes it easy to add stuff later. A well-maintained project takes more time, but is worth it. Sloppy is the friend of complexity.
Optimization is always about doing something extra now, that results in a gain later because the results get re-used over and over. Clean code saves time in understanding and modifying it. A little bit of extra work and discipline that pay off. Design saves time from being lost later. With a solid design you can fit right into the code you need, you don't have to waste a lot of time wandering around trying to guess at what might work.
Time is an all important quantity in software development. Wasting it is a bad idea. Some of the newer programming development techniques seek to make 'games' out of programming. It is an immature way of trying to remove some of the natural tediousness away from coding. We want to build things and get to the end as fast as possible. Playing around is avoiding the issue.
If you want to build great tools, at times it will be painful, it comes with the territory. No job is 100% entertaining, they all have their down-sides, that is why it is called work and we have to be paid to do it. Nothing wrong with hobbyist programmer's playing games and competing, but its is not appropriate for the work place.
Testing is one area that people waste massive amounts of time without getting any additional benefit. Again, some of the newer testing techniques add extra work, but in exchange they claim to reduce the amount of bugs. If that works it is great, but you have to be sceptical of most claims. Component testing thoroughly is good, but if you cannot assure the interaction of the components with each other, then there is always some minimal level of final testing that still needs to be done.
It is immutable, and as such is not possible to be optimized away. If you must test the final product as a whole coherent piece, than you cannot skip those tests no matter how much work you have done on the sub-components. If you are not skipping the tests, then you are not saving any time. If you down-grade the final tests to be less, then you upgrade the risks of allowing problems through. Of course this is true as a basic principle: in-stream testing in any process is there to reduce the amount of bouncing around between states, but it does significantly bump up the amount of testing work without increasing the quality. If the bouncing between states is not significant, then reducing it doesn't add much extra value. It doesn't negate any of the final testing, which still needs to be done.
In same way that one cannot solve the halting problem, the amount of testing required to achieve 100% certainty that there are no bugs is 'infinite'. It is an easy proof, once you accept the possibility that there is some sequence of input that causes the software to get into a state that can break it. Given that all non-trivial software maintains some form of internal-state, then to achieve absolute certainty that there are no bugs you have have to test every combination of possible input. Since there is no limit to the size of the input, the number of possible test scenarios is infinite, and it would take forever to create and apply them. Given our restrictions on time, unless we do our development on the edge of a black hole, there will always be some possibly of bugs with any release.
You may put in significant effort for a series of releases to really produce stellar quality, but you always need to remember that software development projects never really end. Sooner or later a bug is going to escape. In practice it is always sooner, and it always happens way more than most programmers anticipate. Although one person called it 'defeatist', it is a really good idea to plan for sending out patches to fix the code. Just assume there will be bugs and deal with it. If you build this into distribution and deployment, then when it is necessary it is just part of the system, otherwise the 'crisis' will eat up huge amounts of time.
Unexpected, 'expected' events cause delays, morale issues and scheduling conflicts. When we continually see the same problems happen over and over again, we need to accept them as part of process, rather than try to ignore them or rail against how unfair they are. If we anticipate problems with the deployment of the software we can build in the mechanisms to make this dealing with the problem easier. Taking a broad approach to development to include the development, testing and operations into the problem domain is the best way to build practical solutions that can withstand significant shifts in their environments.
The time issues never go away and ignoring them only makes it worse and more disruptive.
BROAD STROKES AND DETAILS
There are lots of arguments between strongly typed and loosely typed languages. Different camps of programmers feel that one or the other is the perfect approach to the problem. After lots of experience on both sides, I've come to realize that some problems are best handled with strongly typed approaches -- while others are best handled with very loose ones. The difference comes down to whether or not you need to be pedantic about the details.
For some types of solutions, the underlying correctness of the data is the key to making it work. In those cases you want to build up internal structures while programming that are very specific, and during the process you want to insure that the correct structure is actually built.
If your writing some code to write out a specific file format, for example, you'd like the internal structure to mirror the file format. In that way, depending on the format the structure and syntax are highly restricted. As the various calls go about in the system building up the structure, there is also code to make sure that the structure stays consistent and correct. With many types of errors, the closer the program stops after the first error, the easier it is to diagnose. When running in a development environment, a program that is strongly typed and checking its data can stop immediately the moment it deviates from the prescribed plan. That makes it really easy to find and fix the problems which also helps to minimize testing.
For some solutions, if you are performing a large number of steps and you want the program to be as tolerant as possible, then loosely typed is way better. It doesn't matter for example, how the data got into the program, instead it matters how it will be transformed. High-level programs, scripting, data filters, and any other program that needs to deal with a wide range of unexpected inputs fit well into this type of circumstance. The range of the data is massive, but the transformations are pretty trivial. If the program stopped each and every time the input was an unknown combination, the code would be so fragile that it would be useless. Loosely typing data under this circumstance means that the importance is on the set of instructions, they need to be completed, the data is only secondary. Scripting in particular requires this.
This dichotomy is true for all software. For some goals the data is the most important thing and it needs to be structured correctly. For some goals, it is the sequence of instructions and the final output that is significant. It doesn't matter how it got there.
So for example, we can use a typed language like Java to perform the algorithmic combinations, but a basically untyped tool like ant to insure that the code was built and deployed correctly. That the shell scripting languages in Unix are mostly loosely typed is no accident. The two approaches are needed for two very different problems.
It is also true that within all systems, the sequence of instructions at the higher level is more important, and the structure of data at the low level is important. The nice part is realizing that the depth of the code makes a big difference. If you are building a complex system, the broad strokes of the code should be loosely typed, they are more flexible and less rigid that way, while the detailed calculations should be strongly typed because the accuracy for the data is often the key. Loose typing at the higher level also helps in decoupling the architecture and splitting off the presentation from the underlying data. All these things work together at the different levels within any system.
Clearly any new language that covers the whole domain of building and deploying complex systems will cover the whole spectrum and be both strongly and loosely typed. Working both into the same syntax will be one of those key things that will bump us forward in complexity handling.
ELEGANCE
Clean and consistent code is a great secret in getting things launched, but it is exceptionally difficult to get a group of programmers to follow some underlying conventions. There are cultural issues that make it nearly impossible for big teams to synchronize their working practices. That could be why the initial version of most of commercial software is built initially by little teams and later handed off to big teams for maintenance. We still don't know how to coordinate our efforts correctly on a large scale. Our most common official methodologies are horrible.
Elegance, just for the sake of elegance is never a great idea. Elegance because it makes you job easier and it means that the software works better is a great idea. It is even better when it takes away a lot of the stresses associated with messy programming habits. It becomes a means to an end.
We build and maintain tools to help our users solve their problems, which are most generally in playing with their ever-increasing piles of data. The most important thing is that the tools we build actually solve the problems for the users. Everybody getting their input into the design, programmers having fun while coding, and processes being left wide open and 'casual' may amuse some people while they are working, but they do not get the basic tasks completed any faster. Spending time to understand the user's real needs, keeping the code clean and consistent, using real graphic design for the interface, and providing simple tools that are easy to understand and effective are some of the things that make the users come to appreciate (or not) the underlying software. Computers can make people's lives easier or they can make them harder, the difference is up to the abilities of the programmers involved with the code.
In the end, a short simple solution that works simply and consistently is as good as it gets. Bad, overly complicated bloated software with every function imaginable isn't good, and it isn't much of an accomplishment. All the fancy graphics, dancing baloney and crammed in information can't hide a badly written program. It's not the technology, it's what you do with it that matters.
Tuesday, February 19, 2008
Sunday, February 10, 2008
The Power of Expression
My writing archives are littered with half-completed, mostly dead posts on the nature of expression. More so than any other topic, this one has defeated any attempt to roll up my ideas into a coherent, finished piece of work.
Writing needs to come together in a way that leads the reader on a journey while leaving them satisfied at the end. Half-thoughts, while interesting, leave the reader longing for more. Sort of like an appetizer with no main course. You won't starve to death, but you're still very hungry afterward.
To get around that, this post is -- I guess -- a series of appetizers; hopefully enough to fulfill. If you don't like one, perhaps some of the following might be more satisfying. If you keep reading long enough, hopefully, you'll be satiated. If you're still hungry at the end, stay tuned, there will always be more.
THE NEVER CHANGING ELEMENTS OF DEVELOPMENT
Software development is all about building 'tools' for users to play with their piles of data. The most important attribute for any tool is that it is usable. The second most important attribute is that it is extensible, or at very least it can be kept updated.
A tool that only worked for a short time is a pain. Any investment in learning how to utilize it, is squandered when it is no longer available. Even the best tools take some effort to master, while most of the software ones take huge effort because of their poor designs.
There is an implicit covenant between the users and the programmers. The users commit to learning how to utilize the tools only if we commit to building and maintaining them over the long run. A 'release' is only a brief instance in the life of any software, and you're only as good as your last release.
Software itself is just a large set of instructions. These days, it is often a huge set of instructions, in fact, trillions and trillions of them if you are looking at assembler. Most modern computers are happily working their way through millions of lines of instructions every second. The sheer size of our existing code bases and amount of code that is being executed is daunting.
Collecting together these instructions is hard, messy and prone to failure. There is much we can do to improve our accuracy, but I'll leave those thoughts for another day.
Once we know what to build, if we didn't know better, we might try to manually type each and every one of the instructions into explicitly the computer. We did it that way initially, but we've learned a lot since then. Or have we?
Even though we are no longer manually flipping switches or pounding out endless lines of assembler, we still commonly employ higher level brute force approaches as our primary means in building software. Generally, most programmers build the system by belting out or copying each every line of code that needs to be executed. They'll use some 'theory' to reduce the redundant code, but it usually is not applied to great effect. Few code-bases are not endlessly-repeating chucks of nearly identical code, despite how we as an industry proclaim we are following principles like DRY (Don't Repeat Yourself -- The Pragmatic Programmers).
Even more disconcerting, on an industry level, is that we are madly adding in as much code as possible to handle all of our perceived problems. There is some type of implicit assumption that having more code will actually help. That if we just had 'enough' code, we could solve all of our user's problems.
The funny thing is that it is a waste of time. You can never win, the amount of brute force code you need is infinite and infinitely growing. E.g. the more crappy code you have, the more crappy code you need to monitor it. That's an exponentially escalating mess. It defines the building culture for 'many' of our current operating systems and major tools. Although we've found ways to add in instructions at a faster rate we have not fundamentally changed our approach to what we are doing. We're just pounding each and every instruction explicitly into the computer.
It might be OK if it wasn't for the fact that code rusts. If we have code we have to maintain it and if it is rapidly growing out of control, that means that sooner or later lots of that code is going to rust. We cannot support it all. Our approach is flawed.
At least there will always be work in programming. Unless there is a shift, companies will endlessly pound-out partial tools. And we will endlessly refactor those tools into arbitrarily inconsistent pieces. And people like virus writers will help, in producing counter-productive code that needs to be monitored and controlled. It really is quite endless with this approach. The more code you have, the more you need, the more work you have to do to keep it going. We are not evolving, we are just barely keeping up with the demand.
LANGUAGE EXPRESSION
There are huge debates over which programming language is best. This rather silly subjective argument has been going on with the entire life of software development. It is not that I don't think the choice of language is important. It can be a critical component in getting the tool successfully built. But inherently, all of the languages that we currently have -- suck. They suck to varying degrees, but they still suck.
We haven't found the right level of abstraction yet, that allows us to build our systems reliably and consistently. We're not even in the right corner of the solution space, so we are quibbling over an endless array of broken and incomplete languages. Do you care what the mnemonic is for incrementing a register in assembler? No, of course not, that discussion is long gone. Is a Pascal pointer better than a C one? That too is ancient history. So, too, will many of today's issues just disappear.
The language we want is the one that makes its representation the closest possible to the way we think about the problem. The 'farther' we have to translate the answers, the more likely there will be errors. The 4am test is critical. Will I be able to sort this out at 4am or do I have to do some huge amount of mental gymnastics? Clear, straight-forward syntax and semantics that match the problem space are inherently necessary in minimizing the undesirable translations from the real world into the computer.
None of our current language paradigms really match the problem space for which we are building. Not objected oriented, nor functional programming, nor any of the older models. Users don't come to us asking for 'objects', nor do they come to us asking for functional closures. There is little in the technical language that maps back to most problem domains.
They come to us to build very specific tools to solve their data pile problems. They talk to us about data and they talk to us about their problems. These other 'things' are abstract technology concepts. We spend a massive amount of time and effort mapping the user based problems onto abstract 'technology' issues. That is a critical amount of our effort.
Not that 'abstraction' itself is bad. The foundation on which we leverage our work across many problem domains is abstraction and it is here that we need to put more effort, not less. Abstractions are the answer to the brute force problem. Abstraction as a concept is great and truly important. It is just that most of our 'current' abstractions are not nearly as strong as we need; they could be more effective. So this leads to problems with 'expressing' our solutions in these underlying languages, technologies, and abstractions.
Instead of foolishly defending our own favorite languages, we should really try to come together and see what works and what doesn't. But honestly, not with a bias in trying to show one language is way better than any other. That really doesn't matter. These days the deciding factor isn't even the language itself, its the libraries and communities that matter most.
The problem we are trying to solve is relatively simple: we want to be able to build tools quickly and correctly. Never lose sight of that underlying problem. Being able to cut and paste a million lines of 'for' and 'if' statements into a barely-stable GUI that tortures its users is not the makings of a great programmer.
At some point, we will find better, stronger abstractions that are closer to the way our users express their problems. When we have reduced that impedance mismatch, building a system will become trivial. Deciding what to build, however, will never change. Understanding the structure of the 'data' will never change. It is only the way we instantiate our solutions that can be affected. That doesn't mean we won't necessarily find larger super-tools that will be leverage-able to provide the underlying mechanics for huge swatches of problems. Given the current redundancy of most of our systems it isn't hard to guess that yes, there are still more than a few elegant solutions just waiting to be found. We've only covered a tiny segment of the capabilities of our machines.
That growth that we still need to accomplish is the underlying essence behind my prediction for the future:
http://theprogrammersparadox.blogspot.com/2008/02/age-of-clarity.html
One day we understand the data we are collecting, and what it really means. Then we will be able to use this understanding to 'deterministically' improve ourselves and our societies. The problem that keeps us from getting to this point is not with our technologies, we just don't know how to use them properly yet. The key to solving this problem will absolutely be the computer; it was the single most significant invention of the 20th century, and it will drive huge social changes in the 21st. We are still vastly underrating the significant of these machines.
COMPLEXITY REVISITED
You can't get very far in software development without having to learn to deal with complexity. Software development management is complexity management. While there are lots of definitions for complexity, people seem to understand the essence of the concept, but they still have trouble with the mechanics.
If we pick a convenient way to break it down into underlying effects, it becomes easier to see where the problems arise. A simple clean definition is always the strongest starting point.
We can start with a few definitions: all 'business' domains have an inherent complexity. The business domain is the specific industry, problem, etc. for which you are writing the tool. Some generalized tools cover huge domains, but all tools must always cover some domain. Unless of course, the code is entirely random and pointless. Even a simple demo is aimed at a specific set of users.
Most developers understand this and go about performing or acquiring a significant amount of analysis of the business domain on which to build their solutions for their common user problems.
What often seems to get missed however, is that the development, testing, and deployment of the software itself is significant. The problem domain for any piece of software isn't just the business domain, it is all of the development domains as well. For example, if you write the perfect tool, but it is flaky, then it is not usable. Everything about the tool, including itself, is part of the tool's 'problem' domain.
In addition, for every technology used in providing the solution, there is an inherent underlying amount of complexity that comes from the technologies themselves. To write 'to' a specific operating system platform, for example, you have to understand the strengths and weakness of it, or your solution will be volatile. Depending on any specific aspect of any technology is a mandatory risk for a project, but a manageable one.
So, for our development project, we have a very huge problem domain with its inherent complexity and a significant amount of technical complexity for each and every piece in the system. If you were brilliant and your underlying technologies were clean, and this and only this was the sum total of the complexities in your solution, it would essentially be perfect. However, for all of these complexities, the culture of software development has an extreme tendency to 'add' in way more complexity on top of all of this.
Beyond the inherent complexity in a system, everything else is 'artificial'. It need not be there, but there is -- in practice -- generally a huge amount of it. It is entirely possible with enough effort to refactor any solution to completely remove, forever, all artificial complexity. That is true by definition. It serves no purpose other than to 'bulk' up the solution.
Frederick P. Brookes uses the term 'accidental' complexity, but I believe that includes both what I call artificial complexity and some of the technical complexity. This makes it a less than desirable term because there is nothing you can do about technical complexity, it is as much a part of the solution as the problem domain. Artificial complexity, on the other hand, is removable.
Also, accidental is a horrible word, although his intended definition is the centuries-old version favored by Aristotle. Oddly, I think that using archaic terms for modern things is in itself a form of artificial complexity, we like to make things sound special so we can be exclusive. Simple is better.
Artificial complexity for most development projects equals or exceeds the other inherent complexities. If not directly, the underlying technologies contribute huge amounts of fancy dancing that need not be necessary in an ideal world to properly complete the solution. The actual amount of artificial complexity in the software industry is astoundingly vast these days, and growing at an exponential rate. It is so large and so pervasive that most developers don't even realize how much of the underlying infrastructure could actually be simplified to make their lives easier. Getting it is a mind-blowing experience.
The funniest part about Computer Science is one's instinctive guess about software that might lead to the assumption that if a large group of people arwere working on the same code for year after year, it would be gradually approaching a system with a decreased amount of artificial complexity. As work progresses, the problem domain would grow larger and the solution, overall, should get simpler.
In reality, the longer these big teams work on their systems, the worse the artificial complex gets. In some cases, the entire terminology and development practices of some of these massive groups is so choked with artificial complexity that it probably represents up to 95% of their effort to discuss, push, rehash, extend or mess with their code on a regular basis. Artificial complexity breeds artificial complexity. Stay at it long enough, and most of what you have is just artificial complexity. Very little real stuff gets done underneath.
My favorite example of artificial complexity is a very visible one. There is but one operating system on the whole planet that distinguishes between binary and text files, and it is rumored that the cause of that distinction was a quick fix for a demo, many decades ago. The reason, so I was told, was that a specific hard drive needed to have the newline characters translated between the operating system and the disk. This then became the reason for differentiating between text and binary files. Translate in one case, ignore in the other.
So this was some simple little problem that briefly reared its head on some early DOS system. Of course, the ripples from this are visible all over. NTFS in Windows still differentiates for no apparent reason. Protocols like FTP require specification of this parameter. No doubt it has worked its way into a countless number of interfaces, particularly any that want portability is DOS or Windows. Million and millions of lines of code have had to deal with keeping track of this for one file or another. Millions and millions of hours of time have been spent debugging problems related to this.
This little 'artificial' distinction -- completely unnecessary -- has had a significant impact on the world. In fact, I'm willing to bet that if we took all of the effort involved in this silly little problem in one way or another and converted into some other form of constructive effort, we'd have quite the funky-cool skyscraper by now. Possibly the largest one on the planet. If you think of how many soon-to-be programmers will eventually trip up on this issue one day, it is very depressing.
Even more disconcerting, is when you consider that along the various decades there were multiple periods where this issue could have, and should have been put to bed. Removed, refactored or cleaned up. But still, it remains. And more importantly, its brothers and sisters and cousins -- wantonly little bits of artificial complexity -- all amount to more effort, than what we needed to solve the actual underlying problems for our users. I'll go out on a limb here, but it is a very thick one: the amount of artificial complexity in the software industry is larger than the amount of inherent complexity, but I'll leave that proposition for someone more knowledgeable and wiser to prove.
WHY EVEN SLICE AND DICE?
Modern day software developers are easily lost. While we may know what we want to accomplish, there is a dizzying array of technological and technique choices to be made before even sitting down to consider a design. With so many subjective arguments and contradictory opinions, it is easy to get lost amongst the voices screaming at each other about the right way to build software.
That is why it is so critical, time and again, to go back to the basics and reexamine them. When the noise gets too loud, you have to ground yourself in what you know to be universally true.
We chop programs up into little pieces to make them easier to build. That is the only reasons why we should be doing it. If it isn't easier, then it is just artificial complexity. That being said, there are so many 'theories' for programming, many of which are great, but all of which are dangerous if you take them too seriously. Again, "we chop programs up into bits to make them easier to build."
That means, they are easier to read, easier to fix, easier to understand, etc. The attributes 'gained' by chopping up the code are all good things that help in the long run. It is not about typing, nor about the 'right' way to do it, nor anything else. Elegance comes solely from the ease in which we can manipulate the system. Clever, convoluted 'tricky' code is never elegant.
In Java, for example, the whole idea of dumping things back into mindless sub-objects called 'beans' only to stitch them back to other fuller objects later, seems like an exercise in futility. For things to be readable, and elegant we want to bring all of the relevant code together, and we don't want to repeat it over and over. What are beans then, other than some semantic mess for non-object structures that aren't even convenient to use in the system.
Now given the above definition of elegance, you might be thinking that it is far easier to pound out a set of instructions, over and over again, then it is to apply some fancy abstraction to it, in an attempt to generalize the solution. That assertion might be true if the time spent pounding out the instructions wasn't significant. However, given that the more 'brutal', the brute force, the more work that is required to get the job done. And not just a little more work, we are talking about massive amounts of work. Pounding out the code is hugely time-consuming, and an infinitely losing proposition.
A good abstraction, that opens up the problem domain and really solves a series of related problems is absolutely more work. But, in comparison, it is only marginally more effort, relative to the alternative. If, for example, you find a way to create twenty GUI screens with one block of code and changing the incoming parameters, it may have taken you twice as much time as writing one of the screens, but 1/10 as much time as writing all twenty. The more powerful the abstraction, the more leverage. The more leverage, the more time 'saved'.
When we generalize programs, we still need to chop them up to be easier to build. All of the same reasons for slicing and dicing some explicit set of instructions also occurs for slicing and dicing some generalized set of instructions. We build with an abstraction in mind to make it easier to understand the code. We build with a pattern in mind for the same reason. The abstraction and the pattern are irrelevant, except in regards to making the underlying code easier to understand. A pattern helps to slice and dice, but unless it is also an abstraction, it should not leave remnants of artificial complexity in the code. Naming objects after design patterns, for example, is misleading. The data in a system is 'that' data, its structure is the pattern. It should be named for what it is, not how it is structured. It is the same as calling your intermediate counter variable in a 'for' loop 'integer'.
Again and again, movements grow from programmers that seek easier ways of building software. That is to be expected, but it is also to be expected that many of these movements are not improvements. With that in mind, we should always fall back to first principles when examining a new movement. If it does not jive with the base problem we are trying to solve, then it is a poor solution. Also, you should never buy the counter-argument that you have to try something to know if it is good or bad. Our ability to think through a problem is tremendous, and our ability to ignore the truth is also tremendous. Just because you find a technique fun, doesn't make it a good idea.
I'VE HAD TOO MUCH JAVA TODAY
Like all programmers, I often have a bias towards whatever technology I am currently working with. The more you dig in and understand something, the easier it becomes. Oddly even though I've been working heavily in Java recently, I don't find the language very appealing.
Languages range from being very flexible and accommodating, to being stiff and fragile. COBOL was the original stiff board. While it did an excellent job of making screens to capture data, it was just painfully boring to work with. The various incantations of Visual Basic were another example of stiff. The language forces its practitioners to pound out brute force code because of the weak semantics of the language. I've never seen elegant VB, and I would be surprised if I ever did.
Java has some of that stiffness. The original plan was to not provide too much rope to allow the programmers to hang themselves. Languages like C are incredibly flexible, but for most programmers that flexibility is dangerous. They don't use it wisely, so their code gets unstable. Language designers don't want to stifle expression, but they can stop their users from creating some types of programs. Stiffness is good and bad.
The biggest problem with Java isn't the language itself, it is the culture that grew out of the language after it matured. The underlying libraries in Java are just damned awkward, and that translated into the coding practices being dammed awkward. Things like beans and struts and just about every library I've ever seem is so over-engineered, inconsistent, and messy. Stuffing in fifty million unrelated Design Patterns became vogue, which fills the system with a tremendous amount of artificial complexity. So much, that working the whole environment on a messy operating system like Windows with one of the more modern icky IDEs, the collection of stuff is as arbitrary and convoluted as working on a mainframe or AS400s (iSeries). It is one giant arbitrary inconsistent mess. When I was younger, I remember not wanting to work on mainframes because the technology was arbitrary and ugly. Now it has found me on the PC. Java has become the new COBOL and Windows has become the new mainframe. The cycle of life perhaps?
There is much interest in adding new features like closures to the language. I'll dispense a little advice: Hey guys, the problem isn't that the language is missing stuff. The problem is that the libraries are damned awkward. Fix the problem. Re-release a Java2 with a decent set of clean and normalized libraries that don't suck. Cut back on the stupid functionality and focus on getting really really simple abstractions. You know you are close when the examples for simple things aren't hideously large. Make simple things simple. Look to the primitive libraries for other languages like Perl and C. Get a philosophy other than 'over-complicate-the-hell-out-of-it', and then clean up the implementation. And, ok make a couple of language changes, like making strings easy to use (remove StringBuffer), get rid of arrays and primitive types, and find a nicer syntax for callbacks. But please, whatever you do, don't adopt the C# strategy of just dumping more and more crap into the pot until you can't see the bottom, that's a guaranteed recipe for 'unstable'.
Despite my misgivings for Java, I'll probably be using it for a while still. At least until we can convince someone to bankroll a serious effort into finding new and better ways of really building systems (if you've got big money, I've got big ideas :-) But it doesn't seem as if the focus right now is on moving forward. We're just too busy drowning in our own artificial complexity to even consider that we shouldn't be. Besides, bugs are big business.
COMING BACK TO THE IMPORTANT OF LANGUAGES
The various sections in this post fit together like a meal of appetizers, because of how they relate to the way we express our solutions for our users. Our primary Computer language and its libraries may be the heart of our implementations but what we really do is translate the perceived needs of our users into long and complex sequences of instructions. By the time you include development, testing, packaging, and distribution a commercial software project will often involve the coordination of many full and partial computer languages. A web-based application may involve over a dozen. These different forms of expressing the solution fit some problems easily, but most require more effort. It is in this expression, that we so easily go astray.
I see each and every language as a series of good and bad attributes, for which I would really like to collect the good together and discard the bad. If you've read my earlier posting on primitives, you probably understand why that is not a great idea, but as an approach towards enhancing development, it is a good direction to start. We need a new language that more closely matches the way we express our problems with each other. We don't necessarily need a fancy 'natural' language system, but we should focus on reducing the amount of translation that is happening between analysis and implementation.
Our latest technologies are extremely complex. Much of it is accumulated artificial complexity, that could be removed if we have the nerve to refactor our solutions. When we find the right underlying abstractions, they will create a consistent layer on which we can easily express super-complicated problems. This combined with a more natural representation should give us a huge leap in technical sophistication. It is always worth noting that a computer is an incredibly powerful mind-machine and that our current level of software development is an extremely disappointingly crude attempt at utilizing it.
It should be easier to express our "understanding from the users" into specific tools. There is no real reason why I need to write, over and over again with the same basic solutions to the same basic technical problems. My real problem is the nature and structure of the data, not what type of list structure is returned from some bizarre internal call. We get so caught up in the fantasy of our brilliance in pounding out little solutions to the same common problems, that we forget about the big picture, the real problem we are trying to solve. Users want to manipulate a pile of data. We need to build specific tools to accomplish this. That underlying consistency threads all types of programming for all types of industries together into one giant related effort. We are building ever-increasing piles of data in the same way that ancient Egyptians were building ever increasingly large pyramids. It is just that it is hard to physically see our efforts, although the web does allow tourists to visit our piles.
A key source of our problems is our underlying technologies, particularly our languages. We need to fix or refactor these if we want to make building things easier. However, if things are too easy, programmers may actually refuse to use the technologies, because they take away too much of the fun. Oddly, one can easily suspect that Frederick P. Brookes is correct about there not being a silver bullet, not because it is impossible, but because people wouldn't use it, even if it existed. Humanity -- in that regard -- is a strange crowd.
Writing needs to come together in a way that leads the reader on a journey while leaving them satisfied at the end. Half-thoughts, while interesting, leave the reader longing for more. Sort of like an appetizer with no main course. You won't starve to death, but you're still very hungry afterward.
To get around that, this post is -- I guess -- a series of appetizers; hopefully enough to fulfill. If you don't like one, perhaps some of the following might be more satisfying. If you keep reading long enough, hopefully, you'll be satiated. If you're still hungry at the end, stay tuned, there will always be more.
THE NEVER CHANGING ELEMENTS OF DEVELOPMENT
Software development is all about building 'tools' for users to play with their piles of data. The most important attribute for any tool is that it is usable. The second most important attribute is that it is extensible, or at very least it can be kept updated.
A tool that only worked for a short time is a pain. Any investment in learning how to utilize it, is squandered when it is no longer available. Even the best tools take some effort to master, while most of the software ones take huge effort because of their poor designs.
There is an implicit covenant between the users and the programmers. The users commit to learning how to utilize the tools only if we commit to building and maintaining them over the long run. A 'release' is only a brief instance in the life of any software, and you're only as good as your last release.
Software itself is just a large set of instructions. These days, it is often a huge set of instructions, in fact, trillions and trillions of them if you are looking at assembler. Most modern computers are happily working their way through millions of lines of instructions every second. The sheer size of our existing code bases and amount of code that is being executed is daunting.
Collecting together these instructions is hard, messy and prone to failure. There is much we can do to improve our accuracy, but I'll leave those thoughts for another day.
Once we know what to build, if we didn't know better, we might try to manually type each and every one of the instructions into explicitly the computer. We did it that way initially, but we've learned a lot since then. Or have we?
Even though we are no longer manually flipping switches or pounding out endless lines of assembler, we still commonly employ higher level brute force approaches as our primary means in building software. Generally, most programmers build the system by belting out or copying each every line of code that needs to be executed. They'll use some 'theory' to reduce the redundant code, but it usually is not applied to great effect. Few code-bases are not endlessly-repeating chucks of nearly identical code, despite how we as an industry proclaim we are following principles like DRY (Don't Repeat Yourself -- The Pragmatic Programmers).
Even more disconcerting, on an industry level, is that we are madly adding in as much code as possible to handle all of our perceived problems. There is some type of implicit assumption that having more code will actually help. That if we just had 'enough' code, we could solve all of our user's problems.
The funny thing is that it is a waste of time. You can never win, the amount of brute force code you need is infinite and infinitely growing. E.g. the more crappy code you have, the more crappy code you need to monitor it. That's an exponentially escalating mess. It defines the building culture for 'many' of our current operating systems and major tools. Although we've found ways to add in instructions at a faster rate we have not fundamentally changed our approach to what we are doing. We're just pounding each and every instruction explicitly into the computer.
It might be OK if it wasn't for the fact that code rusts. If we have code we have to maintain it and if it is rapidly growing out of control, that means that sooner or later lots of that code is going to rust. We cannot support it all. Our approach is flawed.
At least there will always be work in programming. Unless there is a shift, companies will endlessly pound-out partial tools. And we will endlessly refactor those tools into arbitrarily inconsistent pieces. And people like virus writers will help, in producing counter-productive code that needs to be monitored and controlled. It really is quite endless with this approach. The more code you have, the more you need, the more work you have to do to keep it going. We are not evolving, we are just barely keeping up with the demand.
LANGUAGE EXPRESSION
There are huge debates over which programming language is best. This rather silly subjective argument has been going on with the entire life of software development. It is not that I don't think the choice of language is important. It can be a critical component in getting the tool successfully built. But inherently, all of the languages that we currently have -- suck. They suck to varying degrees, but they still suck.
We haven't found the right level of abstraction yet, that allows us to build our systems reliably and consistently. We're not even in the right corner of the solution space, so we are quibbling over an endless array of broken and incomplete languages. Do you care what the mnemonic is for incrementing a register in assembler? No, of course not, that discussion is long gone. Is a Pascal pointer better than a C one? That too is ancient history. So, too, will many of today's issues just disappear.
The language we want is the one that makes its representation the closest possible to the way we think about the problem. The 'farther' we have to translate the answers, the more likely there will be errors. The 4am test is critical. Will I be able to sort this out at 4am or do I have to do some huge amount of mental gymnastics? Clear, straight-forward syntax and semantics that match the problem space are inherently necessary in minimizing the undesirable translations from the real world into the computer.
None of our current language paradigms really match the problem space for which we are building. Not objected oriented, nor functional programming, nor any of the older models. Users don't come to us asking for 'objects', nor do they come to us asking for functional closures. There is little in the technical language that maps back to most problem domains.
They come to us to build very specific tools to solve their data pile problems. They talk to us about data and they talk to us about their problems. These other 'things' are abstract technology concepts. We spend a massive amount of time and effort mapping the user based problems onto abstract 'technology' issues. That is a critical amount of our effort.
Not that 'abstraction' itself is bad. The foundation on which we leverage our work across many problem domains is abstraction and it is here that we need to put more effort, not less. Abstractions are the answer to the brute force problem. Abstraction as a concept is great and truly important. It is just that most of our 'current' abstractions are not nearly as strong as we need; they could be more effective. So this leads to problems with 'expressing' our solutions in these underlying languages, technologies, and abstractions.
Instead of foolishly defending our own favorite languages, we should really try to come together and see what works and what doesn't. But honestly, not with a bias in trying to show one language is way better than any other. That really doesn't matter. These days the deciding factor isn't even the language itself, its the libraries and communities that matter most.
The problem we are trying to solve is relatively simple: we want to be able to build tools quickly and correctly. Never lose sight of that underlying problem. Being able to cut and paste a million lines of 'for' and 'if' statements into a barely-stable GUI that tortures its users is not the makings of a great programmer.
At some point, we will find better, stronger abstractions that are closer to the way our users express their problems. When we have reduced that impedance mismatch, building a system will become trivial. Deciding what to build, however, will never change. Understanding the structure of the 'data' will never change. It is only the way we instantiate our solutions that can be affected. That doesn't mean we won't necessarily find larger super-tools that will be leverage-able to provide the underlying mechanics for huge swatches of problems. Given the current redundancy of most of our systems it isn't hard to guess that yes, there are still more than a few elegant solutions just waiting to be found. We've only covered a tiny segment of the capabilities of our machines.
That growth that we still need to accomplish is the underlying essence behind my prediction for the future:
http://theprogrammersparadox.blogspot.com/2008/02/age-of-clarity.html
One day we understand the data we are collecting, and what it really means. Then we will be able to use this understanding to 'deterministically' improve ourselves and our societies. The problem that keeps us from getting to this point is not with our technologies, we just don't know how to use them properly yet. The key to solving this problem will absolutely be the computer; it was the single most significant invention of the 20th century, and it will drive huge social changes in the 21st. We are still vastly underrating the significant of these machines.
COMPLEXITY REVISITED
You can't get very far in software development without having to learn to deal with complexity. Software development management is complexity management. While there are lots of definitions for complexity, people seem to understand the essence of the concept, but they still have trouble with the mechanics.
If we pick a convenient way to break it down into underlying effects, it becomes easier to see where the problems arise. A simple clean definition is always the strongest starting point.
We can start with a few definitions: all 'business' domains have an inherent complexity. The business domain is the specific industry, problem, etc. for which you are writing the tool. Some generalized tools cover huge domains, but all tools must always cover some domain. Unless of course, the code is entirely random and pointless. Even a simple demo is aimed at a specific set of users.
Most developers understand this and go about performing or acquiring a significant amount of analysis of the business domain on which to build their solutions for their common user problems.
What often seems to get missed however, is that the development, testing, and deployment of the software itself is significant. The problem domain for any piece of software isn't just the business domain, it is all of the development domains as well. For example, if you write the perfect tool, but it is flaky, then it is not usable. Everything about the tool, including itself, is part of the tool's 'problem' domain.
In addition, for every technology used in providing the solution, there is an inherent underlying amount of complexity that comes from the technologies themselves. To write 'to' a specific operating system platform, for example, you have to understand the strengths and weakness of it, or your solution will be volatile. Depending on any specific aspect of any technology is a mandatory risk for a project, but a manageable one.
So, for our development project, we have a very huge problem domain with its inherent complexity and a significant amount of technical complexity for each and every piece in the system. If you were brilliant and your underlying technologies were clean, and this and only this was the sum total of the complexities in your solution, it would essentially be perfect. However, for all of these complexities, the culture of software development has an extreme tendency to 'add' in way more complexity on top of all of this.
Beyond the inherent complexity in a system, everything else is 'artificial'. It need not be there, but there is -- in practice -- generally a huge amount of it. It is entirely possible with enough effort to refactor any solution to completely remove, forever, all artificial complexity. That is true by definition. It serves no purpose other than to 'bulk' up the solution.
Frederick P. Brookes uses the term 'accidental' complexity, but I believe that includes both what I call artificial complexity and some of the technical complexity. This makes it a less than desirable term because there is nothing you can do about technical complexity, it is as much a part of the solution as the problem domain. Artificial complexity, on the other hand, is removable.
Also, accidental is a horrible word, although his intended definition is the centuries-old version favored by Aristotle. Oddly, I think that using archaic terms for modern things is in itself a form of artificial complexity, we like to make things sound special so we can be exclusive. Simple is better.
Artificial complexity for most development projects equals or exceeds the other inherent complexities. If not directly, the underlying technologies contribute huge amounts of fancy dancing that need not be necessary in an ideal world to properly complete the solution. The actual amount of artificial complexity in the software industry is astoundingly vast these days, and growing at an exponential rate. It is so large and so pervasive that most developers don't even realize how much of the underlying infrastructure could actually be simplified to make their lives easier. Getting it is a mind-blowing experience.
The funniest part about Computer Science is one's instinctive guess about software that might lead to the assumption that if a large group of people arwere working on the same code for year after year, it would be gradually approaching a system with a decreased amount of artificial complexity. As work progresses, the problem domain would grow larger and the solution, overall, should get simpler.
In reality, the longer these big teams work on their systems, the worse the artificial complex gets. In some cases, the entire terminology and development practices of some of these massive groups is so choked with artificial complexity that it probably represents up to 95% of their effort to discuss, push, rehash, extend or mess with their code on a regular basis. Artificial complexity breeds artificial complexity. Stay at it long enough, and most of what you have is just artificial complexity. Very little real stuff gets done underneath.
My favorite example of artificial complexity is a very visible one. There is but one operating system on the whole planet that distinguishes between binary and text files, and it is rumored that the cause of that distinction was a quick fix for a demo, many decades ago. The reason, so I was told, was that a specific hard drive needed to have the newline characters translated between the operating system and the disk. This then became the reason for differentiating between text and binary files. Translate in one case, ignore in the other.
So this was some simple little problem that briefly reared its head on some early DOS system. Of course, the ripples from this are visible all over. NTFS in Windows still differentiates for no apparent reason. Protocols like FTP require specification of this parameter. No doubt it has worked its way into a countless number of interfaces, particularly any that want portability is DOS or Windows. Million and millions of lines of code have had to deal with keeping track of this for one file or another. Millions and millions of hours of time have been spent debugging problems related to this.
This little 'artificial' distinction -- completely unnecessary -- has had a significant impact on the world. In fact, I'm willing to bet that if we took all of the effort involved in this silly little problem in one way or another and converted into some other form of constructive effort, we'd have quite the funky-cool skyscraper by now. Possibly the largest one on the planet. If you think of how many soon-to-be programmers will eventually trip up on this issue one day, it is very depressing.
Even more disconcerting, is when you consider that along the various decades there were multiple periods where this issue could have, and should have been put to bed. Removed, refactored or cleaned up. But still, it remains. And more importantly, its brothers and sisters and cousins -- wantonly little bits of artificial complexity -- all amount to more effort, than what we needed to solve the actual underlying problems for our users. I'll go out on a limb here, but it is a very thick one: the amount of artificial complexity in the software industry is larger than the amount of inherent complexity, but I'll leave that proposition for someone more knowledgeable and wiser to prove.
WHY EVEN SLICE AND DICE?
Modern day software developers are easily lost. While we may know what we want to accomplish, there is a dizzying array of technological and technique choices to be made before even sitting down to consider a design. With so many subjective arguments and contradictory opinions, it is easy to get lost amongst the voices screaming at each other about the right way to build software.
That is why it is so critical, time and again, to go back to the basics and reexamine them. When the noise gets too loud, you have to ground yourself in what you know to be universally true.
We chop programs up into little pieces to make them easier to build. That is the only reasons why we should be doing it. If it isn't easier, then it is just artificial complexity. That being said, there are so many 'theories' for programming, many of which are great, but all of which are dangerous if you take them too seriously. Again, "we chop programs up into bits to make them easier to build."
That means, they are easier to read, easier to fix, easier to understand, etc. The attributes 'gained' by chopping up the code are all good things that help in the long run. It is not about typing, nor about the 'right' way to do it, nor anything else. Elegance comes solely from the ease in which we can manipulate the system. Clever, convoluted 'tricky' code is never elegant.
In Java, for example, the whole idea of dumping things back into mindless sub-objects called 'beans' only to stitch them back to other fuller objects later, seems like an exercise in futility. For things to be readable, and elegant we want to bring all of the relevant code together, and we don't want to repeat it over and over. What are beans then, other than some semantic mess for non-object structures that aren't even convenient to use in the system.
Now given the above definition of elegance, you might be thinking that it is far easier to pound out a set of instructions, over and over again, then it is to apply some fancy abstraction to it, in an attempt to generalize the solution. That assertion might be true if the time spent pounding out the instructions wasn't significant. However, given that the more 'brutal', the brute force, the more work that is required to get the job done. And not just a little more work, we are talking about massive amounts of work. Pounding out the code is hugely time-consuming, and an infinitely losing proposition.
A good abstraction, that opens up the problem domain and really solves a series of related problems is absolutely more work. But, in comparison, it is only marginally more effort, relative to the alternative. If, for example, you find a way to create twenty GUI screens with one block of code and changing the incoming parameters, it may have taken you twice as much time as writing one of the screens, but 1/10 as much time as writing all twenty. The more powerful the abstraction, the more leverage. The more leverage, the more time 'saved'.
When we generalize programs, we still need to chop them up to be easier to build. All of the same reasons for slicing and dicing some explicit set of instructions also occurs for slicing and dicing some generalized set of instructions. We build with an abstraction in mind to make it easier to understand the code. We build with a pattern in mind for the same reason. The abstraction and the pattern are irrelevant, except in regards to making the underlying code easier to understand. A pattern helps to slice and dice, but unless it is also an abstraction, it should not leave remnants of artificial complexity in the code. Naming objects after design patterns, for example, is misleading. The data in a system is 'that' data, its structure is the pattern. It should be named for what it is, not how it is structured. It is the same as calling your intermediate counter variable in a 'for' loop 'integer'.
Again and again, movements grow from programmers that seek easier ways of building software. That is to be expected, but it is also to be expected that many of these movements are not improvements. With that in mind, we should always fall back to first principles when examining a new movement. If it does not jive with the base problem we are trying to solve, then it is a poor solution. Also, you should never buy the counter-argument that you have to try something to know if it is good or bad. Our ability to think through a problem is tremendous, and our ability to ignore the truth is also tremendous. Just because you find a technique fun, doesn't make it a good idea.
I'VE HAD TOO MUCH JAVA TODAY
Like all programmers, I often have a bias towards whatever technology I am currently working with. The more you dig in and understand something, the easier it becomes. Oddly even though I've been working heavily in Java recently, I don't find the language very appealing.
Languages range from being very flexible and accommodating, to being stiff and fragile. COBOL was the original stiff board. While it did an excellent job of making screens to capture data, it was just painfully boring to work with. The various incantations of Visual Basic were another example of stiff. The language forces its practitioners to pound out brute force code because of the weak semantics of the language. I've never seen elegant VB, and I would be surprised if I ever did.
Java has some of that stiffness. The original plan was to not provide too much rope to allow the programmers to hang themselves. Languages like C are incredibly flexible, but for most programmers that flexibility is dangerous. They don't use it wisely, so their code gets unstable. Language designers don't want to stifle expression, but they can stop their users from creating some types of programs. Stiffness is good and bad.
The biggest problem with Java isn't the language itself, it is the culture that grew out of the language after it matured. The underlying libraries in Java are just damned awkward, and that translated into the coding practices being dammed awkward. Things like beans and struts and just about every library I've ever seem is so over-engineered, inconsistent, and messy. Stuffing in fifty million unrelated Design Patterns became vogue, which fills the system with a tremendous amount of artificial complexity. So much, that working the whole environment on a messy operating system like Windows with one of the more modern icky IDEs, the collection of stuff is as arbitrary and convoluted as working on a mainframe or AS400s (iSeries). It is one giant arbitrary inconsistent mess. When I was younger, I remember not wanting to work on mainframes because the technology was arbitrary and ugly. Now it has found me on the PC. Java has become the new COBOL and Windows has become the new mainframe. The cycle of life perhaps?
There is much interest in adding new features like closures to the language. I'll dispense a little advice: Hey guys, the problem isn't that the language is missing stuff. The problem is that the libraries are damned awkward. Fix the problem. Re-release a Java2 with a decent set of clean and normalized libraries that don't suck. Cut back on the stupid functionality and focus on getting really really simple abstractions. You know you are close when the examples for simple things aren't hideously large. Make simple things simple. Look to the primitive libraries for other languages like Perl and C. Get a philosophy other than 'over-complicate-the-hell-out-of-it', and then clean up the implementation. And, ok make a couple of language changes, like making strings easy to use (remove StringBuffer), get rid of arrays and primitive types, and find a nicer syntax for callbacks. But please, whatever you do, don't adopt the C# strategy of just dumping more and more crap into the pot until you can't see the bottom, that's a guaranteed recipe for 'unstable'.
Despite my misgivings for Java, I'll probably be using it for a while still. At least until we can convince someone to bankroll a serious effort into finding new and better ways of really building systems (if you've got big money, I've got big ideas :-) But it doesn't seem as if the focus right now is on moving forward. We're just too busy drowning in our own artificial complexity to even consider that we shouldn't be. Besides, bugs are big business.
COMING BACK TO THE IMPORTANT OF LANGUAGES
The various sections in this post fit together like a meal of appetizers, because of how they relate to the way we express our solutions for our users. Our primary Computer language and its libraries may be the heart of our implementations but what we really do is translate the perceived needs of our users into long and complex sequences of instructions. By the time you include development, testing, packaging, and distribution a commercial software project will often involve the coordination of many full and partial computer languages. A web-based application may involve over a dozen. These different forms of expressing the solution fit some problems easily, but most require more effort. It is in this expression, that we so easily go astray.
I see each and every language as a series of good and bad attributes, for which I would really like to collect the good together and discard the bad. If you've read my earlier posting on primitives, you probably understand why that is not a great idea, but as an approach towards enhancing development, it is a good direction to start. We need a new language that more closely matches the way we express our problems with each other. We don't necessarily need a fancy 'natural' language system, but we should focus on reducing the amount of translation that is happening between analysis and implementation.
Our latest technologies are extremely complex. Much of it is accumulated artificial complexity, that could be removed if we have the nerve to refactor our solutions. When we find the right underlying abstractions, they will create a consistent layer on which we can easily express super-complicated problems. This combined with a more natural representation should give us a huge leap in technical sophistication. It is always worth noting that a computer is an incredibly powerful mind-machine and that our current level of software development is an extremely disappointingly crude attempt at utilizing it.
It should be easier to express our "understanding from the users" into specific tools. There is no real reason why I need to write, over and over again with the same basic solutions to the same basic technical problems. My real problem is the nature and structure of the data, not what type of list structure is returned from some bizarre internal call. We get so caught up in the fantasy of our brilliance in pounding out little solutions to the same common problems, that we forget about the big picture, the real problem we are trying to solve. Users want to manipulate a pile of data. We need to build specific tools to accomplish this. That underlying consistency threads all types of programming for all types of industries together into one giant related effort. We are building ever-increasing piles of data in the same way that ancient Egyptians were building ever increasingly large pyramids. It is just that it is hard to physically see our efforts, although the web does allow tourists to visit our piles.
A key source of our problems is our underlying technologies, particularly our languages. We need to fix or refactor these if we want to make building things easier. However, if things are too easy, programmers may actually refuse to use the technologies, because they take away too much of the fun. Oddly, one can easily suspect that Frederick P. Brookes is correct about there not being a silver bullet, not because it is impossible, but because people wouldn't use it, even if it existed. Humanity -- in that regard -- is a strange crowd.
Saturday, February 2, 2008
The Age of Clarity
"What do we really know? Hmmm." I pondered as we walked.
I was out with the dog the other night. The quiet tranquil nature of empty suburban streets is a great place for deep thinking. The cold chill of winter keeps one from wandering too far off topic while wandering aimlessly in the streets. Dogs make wonderful intellectual companions for these types of journeys because they don't interrupt with too many questions. They are very good listeners.
I was pondering information quality, and I foolishly started to wonder about how much inaccurate information was choking up my memory. Certainly, there are lots of spin, lies, half-truths, deceptions and other stuff built up over the years from less than quality sources like politics, news and TV. Somethings in my memory are just easy simplifications. Somethings are out right fabrications. There is also the changing nature of science, and our non-stop quest for learning. Some of my knowledge is just 'relative', it wouldn't stand up to a universal judge. It is considered true here and now, but won't be in the future. In an overall sense, how much of this is really accurate?
If you factor in all of the different reasons for low quality, and take a big sweeping guess, the amount of truth in our brains could be lower than 30%. Just a wild guess, but I could easily believe that 1 in every 3 three facts in my brain are true, while the other 2 are questionable for all sorts of reasons. I am just speculating of course, but in this misinformation age, we are full of a tremendous amount of low quality knowledge. And it feels like it is growing at an ever increasing rate, although that might just be our ability to confirm that it is suspect.
THOSE WHO FORGET HISTORY ARE DOOMED TO REPEAT IT
In the past, mankind was mostly ignorant of the accuracy of their information. They could take pride in their depth of knowledge without ever knowing how dubious it really was. Now all we have to do is check wikipedia and we can instantly find out that truth, well, err at least a pointer towards the truth.
How often have I pulled forth some ancient fact from the depths of my brain, only to discover that it was fundamentally untrue? Worse still is how those facts actually make it into my head in the first place. Some were obviously from disreputable sources, but others had come from well-known authorities, and were still incorrect. My problem is not loss or corruption of memory, it is the opposite, these 'facts' stay for far too long. If I just dumped them faster, I might find they were more accurate overall.
It is oddly telling. It allows us to guess that this huge degree of inaccuracy in our current knowledge is actually some type of pointer towards the future. The Renaissance was an awaking about the world that we live in. A moment when we first opened our eyes and saw it for what it actually was. This in turn drove the foundations for the industrial age, where we learned to create and use an unlimited number of machines. One of those machines, the computer, has driven us into an information age, where we collect huge piles of information, about virtually everything in this world. There is a trend here. The next age will follow along in this sequence.
Even though we have built up a tremendous collection of fantastic machines, they do not serve us well. We can build things, but we have trouble maintaining them. Our massive and complex cities crumble around us. We are forever fighting a losing battle against entropy; like a runner that has leaned too far forward we are continually off balance. We must continue to build to move forward, we don't know how to preserve what we have and we don't know how to live within our environmental means. We grow at a severe cost to the world around us.
With all of our equipment and learning, collecting data is still a hit or miss proposition. We just guess at what we want to collect and how it is structured. It is not orderly and we don't have any underlying theories that drive our understanding. Computer Science is still so young that is frequently wrong. Often it is just random guessing. We are currently only utilizing a small fraction of the capabilities of our computers because we keep bumping into complexity thresholds each time we try to build truly sophisticated systems. We are trapped with crude software.
Even thought we can collect the data, we continuously fail to be able to mine or interpret it. We gather the stuff, format it and then save it to backup tapes. But we get little actual use from all of our work in collecting it. Some decisions are made from the data, but given the real amount of underlying information contained in our efforts we could actually use what we have to make really sophisticated decisions. To actually know, for fact that we are changing things for the better. If we understood what we have.
THE YELLOW BRICK ROAD
Given those trends, it is not hard at all to predicate the future. The next step in the sequence. The path we must take is the only one available: machines to labour for us physically and mentally lead to vast infrastructures and vast piles of information. We built up these things, but we don't understanding them, and we have trouble keeping them going.
It is not like we will wake up one day and the light will get turned on, but I imagine that over time like a dull and steady wind blowing away the haze, much of what we know will become clear and finally fit into place. It will be a modern day Renaissance reoccurring not with our perspective of the world around us, but with our perspective of the information and knowledge that we have collected. It will take time. Many years, decades or even a century or two, but one day there will be an 'age of clarity', where mankind can finally see the information around them for what it actually is. That is, if we survive the turmoil of our current societies; we have so many dangers that await us, because of what we know, but don't yet understand.
And what could we expect in such an age? I imagine that we will have a real understanding of information, probably based on a currently unknown science. Maybe several. We will know how to quickly, conveniently monitor and collect information for any questions. Inherently, we will understand the truthfulness of what we collect, and we'll be able to immediately use this information to ascertain whether things are improving or getting worse. The term 'immediately' being one of the very key points.
Unlike now, this won't be a big effort, but rather something simple that people do as a matter of due diligence. Government effectiveness for example, will be based on simple true numbers that show that things are improving or getting worse. Unlike the statistics of our day, these numbers and their interpretation, based on science will be irrefutable. We will be able to show cause and effect relationships between policies and real life. We will be able to measure the effectiveness, not guess at it. If we say things are getting better, it won't just be 'spin'.
Underneath, if we capture enough data, we will get a vibrant picture of all of the relationships, how they fit with each other and what they really mean. When we choose to make changes, they will not be partially-informed guesses, they will be tangible deterministic improvements to our societies that will work as expected. In the same way that the industrial age leapt from wildly building things to the reproducible industrialization of products with a tremendous amount of consistent quality, we will shift our understanding of the data around us. Like the difference between B&W photography and color we will learn how to start really capturing the information that is of real value, and we will learn how to really interpret it.
AT THE EDGE OF REASON
Does it sound too overly deterministic or crazy? Whatever comes in the future, it must be something that isn't here now. So, if it isn't pushing the envelop of convention, then its not really much of a prediction, is it? Jules Verne wrote about ships that travelled underwater, a famously crazy concept if ever there was one, except that it now has become common knowledge. He wrote about air ships, defying gravity and hanging in the sky with birds, clearly another bit of wackiness. Yet, this too is common, and rather boring now.
Does it sound very similar to what we have now? For all we know, we know so very little. We have many approaches and methods to really prove things, to get the the real underlying truth, but because we can't do that easily on a grand scale we are awash in misinformation. All of this low quality knowledge chokes our pathways and keeps us from progressing. It becomes food for subjective arguments, and endless discussions. And while some of us may suspect falsehoods, proving it is costly and often distracting. We can't fight all of the battles all of the time, so the low quality stuff washes over us like a tsunami.
Sometimes when I am out walking the dog, my mind drifts around to us being so sophisticated that there is not much left in this world that we don't know. That 'proposition' is comforting in many ways, but patently false. Like the pre-Renaissance societies, we think we have reached some level of sophistication, but we barely even realize how to keep our own existence from spiraling out of control. And what we don't know, is the question: "what do we really know?" We feel pride in having built up knowledge bases like the World Wide Web, but realistically the things are a mess. What good is a massive unorganized pile of data, if we can't use it to answer the serious questions in our lives? We live in an age where subjective arguments are possible for most of what we commonly deal with in our lives. Everything is up for grabs; everything is based on opinion. We can barely distinguish the quality of our facts, let alone position them into some coherent and universally correct structure of the world around us. For all that we know, we are still incredibly ignorant.
The next big thing then is obvious. If we are too survive, then we have to pass through the Clarity Age. We have no choice. If this understanding hasn't already popped into someone else's brain, it was bound to sooner or later. You can't get very far down the path, if you don't know where the path lies. It is murky now, and for us to progress it must be clear.
Woof, woof, woof! My thinking and wanderings were interrupted because the dog spotted a raccoon. I was riped from the depths by the pulling, jumping and barking. In the here and now, I am reminded that it is best if I move on quickly to keep the dog from making too much racket. I don't want to wake up my whole neighborhood with the commotion. However much I long to spend time in the future, I must live with the world around me as it is now. These dark ages are apt to last a while, possibly my entire life. I ought not to waste it, pining for enlightenment.
I was out with the dog the other night. The quiet tranquil nature of empty suburban streets is a great place for deep thinking. The cold chill of winter keeps one from wandering too far off topic while wandering aimlessly in the streets. Dogs make wonderful intellectual companions for these types of journeys because they don't interrupt with too many questions. They are very good listeners.
I was pondering information quality, and I foolishly started to wonder about how much inaccurate information was choking up my memory. Certainly, there are lots of spin, lies, half-truths, deceptions and other stuff built up over the years from less than quality sources like politics, news and TV. Somethings in my memory are just easy simplifications. Somethings are out right fabrications. There is also the changing nature of science, and our non-stop quest for learning. Some of my knowledge is just 'relative', it wouldn't stand up to a universal judge. It is considered true here and now, but won't be in the future. In an overall sense, how much of this is really accurate?
If you factor in all of the different reasons for low quality, and take a big sweeping guess, the amount of truth in our brains could be lower than 30%. Just a wild guess, but I could easily believe that 1 in every 3 three facts in my brain are true, while the other 2 are questionable for all sorts of reasons. I am just speculating of course, but in this misinformation age, we are full of a tremendous amount of low quality knowledge. And it feels like it is growing at an ever increasing rate, although that might just be our ability to confirm that it is suspect.
THOSE WHO FORGET HISTORY ARE DOOMED TO REPEAT IT
In the past, mankind was mostly ignorant of the accuracy of their information. They could take pride in their depth of knowledge without ever knowing how dubious it really was. Now all we have to do is check wikipedia and we can instantly find out that truth, well, err at least a pointer towards the truth.
How often have I pulled forth some ancient fact from the depths of my brain, only to discover that it was fundamentally untrue? Worse still is how those facts actually make it into my head in the first place. Some were obviously from disreputable sources, but others had come from well-known authorities, and were still incorrect. My problem is not loss or corruption of memory, it is the opposite, these 'facts' stay for far too long. If I just dumped them faster, I might find they were more accurate overall.
It is oddly telling. It allows us to guess that this huge degree of inaccuracy in our current knowledge is actually some type of pointer towards the future. The Renaissance was an awaking about the world that we live in. A moment when we first opened our eyes and saw it for what it actually was. This in turn drove the foundations for the industrial age, where we learned to create and use an unlimited number of machines. One of those machines, the computer, has driven us into an information age, where we collect huge piles of information, about virtually everything in this world. There is a trend here. The next age will follow along in this sequence.
Even though we have built up a tremendous collection of fantastic machines, they do not serve us well. We can build things, but we have trouble maintaining them. Our massive and complex cities crumble around us. We are forever fighting a losing battle against entropy; like a runner that has leaned too far forward we are continually off balance. We must continue to build to move forward, we don't know how to preserve what we have and we don't know how to live within our environmental means. We grow at a severe cost to the world around us.
With all of our equipment and learning, collecting data is still a hit or miss proposition. We just guess at what we want to collect and how it is structured. It is not orderly and we don't have any underlying theories that drive our understanding. Computer Science is still so young that is frequently wrong. Often it is just random guessing. We are currently only utilizing a small fraction of the capabilities of our computers because we keep bumping into complexity thresholds each time we try to build truly sophisticated systems. We are trapped with crude software.
Even thought we can collect the data, we continuously fail to be able to mine or interpret it. We gather the stuff, format it and then save it to backup tapes. But we get little actual use from all of our work in collecting it. Some decisions are made from the data, but given the real amount of underlying information contained in our efforts we could actually use what we have to make really sophisticated decisions. To actually know, for fact that we are changing things for the better. If we understood what we have.
THE YELLOW BRICK ROAD
Given those trends, it is not hard at all to predicate the future. The next step in the sequence. The path we must take is the only one available: machines to labour for us physically and mentally lead to vast infrastructures and vast piles of information. We built up these things, but we don't understanding them, and we have trouble keeping them going.
It is not like we will wake up one day and the light will get turned on, but I imagine that over time like a dull and steady wind blowing away the haze, much of what we know will become clear and finally fit into place. It will be a modern day Renaissance reoccurring not with our perspective of the world around us, but with our perspective of the information and knowledge that we have collected. It will take time. Many years, decades or even a century or two, but one day there will be an 'age of clarity', where mankind can finally see the information around them for what it actually is. That is, if we survive the turmoil of our current societies; we have so many dangers that await us, because of what we know, but don't yet understand.
And what could we expect in such an age? I imagine that we will have a real understanding of information, probably based on a currently unknown science. Maybe several. We will know how to quickly, conveniently monitor and collect information for any questions. Inherently, we will understand the truthfulness of what we collect, and we'll be able to immediately use this information to ascertain whether things are improving or getting worse. The term 'immediately' being one of the very key points.
Unlike now, this won't be a big effort, but rather something simple that people do as a matter of due diligence. Government effectiveness for example, will be based on simple true numbers that show that things are improving or getting worse. Unlike the statistics of our day, these numbers and their interpretation, based on science will be irrefutable. We will be able to show cause and effect relationships between policies and real life. We will be able to measure the effectiveness, not guess at it. If we say things are getting better, it won't just be 'spin'.
Underneath, if we capture enough data, we will get a vibrant picture of all of the relationships, how they fit with each other and what they really mean. When we choose to make changes, they will not be partially-informed guesses, they will be tangible deterministic improvements to our societies that will work as expected. In the same way that the industrial age leapt from wildly building things to the reproducible industrialization of products with a tremendous amount of consistent quality, we will shift our understanding of the data around us. Like the difference between B&W photography and color we will learn how to start really capturing the information that is of real value, and we will learn how to really interpret it.
AT THE EDGE OF REASON
Does it sound too overly deterministic or crazy? Whatever comes in the future, it must be something that isn't here now. So, if it isn't pushing the envelop of convention, then its not really much of a prediction, is it? Jules Verne wrote about ships that travelled underwater, a famously crazy concept if ever there was one, except that it now has become common knowledge. He wrote about air ships, defying gravity and hanging in the sky with birds, clearly another bit of wackiness. Yet, this too is common, and rather boring now.
Does it sound very similar to what we have now? For all we know, we know so very little. We have many approaches and methods to really prove things, to get the the real underlying truth, but because we can't do that easily on a grand scale we are awash in misinformation. All of this low quality knowledge chokes our pathways and keeps us from progressing. It becomes food for subjective arguments, and endless discussions. And while some of us may suspect falsehoods, proving it is costly and often distracting. We can't fight all of the battles all of the time, so the low quality stuff washes over us like a tsunami.
Sometimes when I am out walking the dog, my mind drifts around to us being so sophisticated that there is not much left in this world that we don't know. That 'proposition' is comforting in many ways, but patently false. Like the pre-Renaissance societies, we think we have reached some level of sophistication, but we barely even realize how to keep our own existence from spiraling out of control. And what we don't know, is the question: "what do we really know?" We feel pride in having built up knowledge bases like the World Wide Web, but realistically the things are a mess. What good is a massive unorganized pile of data, if we can't use it to answer the serious questions in our lives? We live in an age where subjective arguments are possible for most of what we commonly deal with in our lives. Everything is up for grabs; everything is based on opinion. We can barely distinguish the quality of our facts, let alone position them into some coherent and universally correct structure of the world around us. For all that we know, we are still incredibly ignorant.
The next big thing then is obvious. If we are too survive, then we have to pass through the Clarity Age. We have no choice. If this understanding hasn't already popped into someone else's brain, it was bound to sooner or later. You can't get very far down the path, if you don't know where the path lies. It is murky now, and for us to progress it must be clear.
Woof, woof, woof! My thinking and wanderings were interrupted because the dog spotted a raccoon. I was riped from the depths by the pulling, jumping and barking. In the here and now, I am reminded that it is best if I move on quickly to keep the dog from making too much racket. I don't want to wake up my whole neighborhood with the commotion. However much I long to spend time in the future, I must live with the world around me as it is now. These dark ages are apt to last a while, possibly my entire life. I ought not to waste it, pining for enlightenment.
Tuesday, January 22, 2008
Essential Development Problems
Over the years I've worked in many development sites, read tonnes of code and heard way too many horror stories about development projects gone bad. While software development is inherently risky, many of the problems I have witnessed were self-inflicted, thus fixable. It is an odd historically-driven aspect of programming, our need to make the work more difficult than necessary.
I wasn't focusing on writing yet another list of things to simplify development, but in my other writings the same issues kept bubbling up over and over again. Sometimes even when you explicitly see something, it is not easy to put a name or description to it. In this case, as I explored more elementary topics I kept getting these 'pieces' floating to the top, but not fitting into the other works. Once I collected these four together, they fit quite naturally.
For this blog entry, I'd like to address this commonality with basic development problems, but try as much as possible to avoid falling into simple platitudes (no guarantees). These problems are elemental, mostly obvious, but surprisingly persistent. Software developers, I am strongly aware, are their own worst enemy. Sure, there are good rants to be had at the expense of management, users or technology, but the present of other problems doesn't validate the desire to overlook the fixable ones.
PROBLEM: PERCEPTION
One significant problem that rears its ugly head again and again, is our actual perception of what we are doing. We all know that if you think you will fail, you definitely will. There must also be some well-known universal truth that states if you go in with the wrong viewpoint, it will significantly reduce your likelihood of success. Even in the case where you are positive, if you really don't understand what you are doing, then you are implicitly relying on luck. And some days, you're just unlucky. If you want to be consistently good at something, you really need to understand it.
When I first started programming I subscribed to the 'magic broom' theory of programming. The best illustration comes from the episode called the Sorcerer's Apprentice in Disney's animated film Fantasia. There is at least one youTube link to it:
http://www.youtube.com/watch?v=LD8HDta7Z_4
In the episode, Mickey Mouse as a Wizard's apprentice, gets a hold of the 'magic hat', then starts issuing special commands to an inanimate broom to deliver water to the Wizard's workshop. Mostly because he falls asleep, the circumstances start to go rapidly out of control. Awaking in a panic, he is unable to stop the growing problem. A hatchet job only intensifies the issue. In the end, he even tries consulting the manual, a desperate step to be sure, but it is far too little, far too late. Is this not an excellent allegory of a programming project gone awry?
My early view of programming was similar. You issued a series of magic commands to the various 'brooms', and presto, blamo, your work is done for you. It is all magic underneath. In this animation, the fact that any apprentice with access to the hat can easily issue commands, even when they do not fully understand the consequences of their actions, relates this back to computers quite effectively. As this common quote likes to describe:
"To err is human, but to really foul things up requires a computer." -- Farmers' Almanac, 1978
Computers amplify their user's abilities; they are the mental equivalent of a bulldozer.
Needless to say, I was disappointed when I discovered that computers were just simple deterministic machines that never deviated from what they are told, and that the commands themselves where only abstractions thought up by other programmers. A significant amount of the 'magic' is just artificial complexity added in over years and years of development.
This is nothing magical about computers. They are machines that unless there is a hardware failure, consistently do the same things day after day. Of course, that's no longer so obvious to most users of our modern 'personal computers'. They have becomes so complex and often so erratic, that many people feel they have personality. Some even think that their machines are actively plotting against them.
THE ART OF COMPUTER PROGRAMMING
For a quite a while I subscribed to the 'writing software is an art form' school of thinking. This view sees programming as a very specific art form, similar to painting or writing poetry. It is an inherently elitist viewpoint based around the idea that some people are naturally born to 'it', while others shouldn't even try. Not everyone is cut out to be an artist, or in this case a programmer.
While not quite as fantastical as the magic broom theory, there is an implication that software is not repeatable, in the same way that any large great work of art is not repeatable. If you accept this, the consequences are quite scary. Given that we are dependent on software, this view directly implies that there is no way to consistently make software stable. The 'fate' of any version of the system rests with the ability to get a hold of the right 'artists'. And these are limited in quantity. So limited, that many programmers would insist that there is only a small handful. Or, at very least a few thousand. But certainly not the masses of people slaving away in IT right now.
Software as an art form also implies that there is some 'essence' that we can't teach. Great art for me goes beyond technique and materials, it happens when an 'artist' manages to capture 'emotion' in the work along with all of the other details. A grand painting inspires some emotional response, while a work of 'graphic art' just looks pretty. It's the same with musicians and pop music. Rarely does a pop song paint a vivid picture, instead people just tap along to the beat. When it 'gets' to you it is art, otherwise it is just 'easy listening'.
Given that definition, it's hard to picture a software application that directly inspires the user on an emotional level. That would be a word processor that makes you depressed just by starting it, long before you've been disappointed by its inherent bugs. Or a browser that just makes you happy, or angry, or something. The idea, then in this context is clearly silly. A tool, such as a hammer, may look pretty but it does not carry with it any emotional baggage. For that you need art, and the 'thing' that a true artist has, that is not 'common' in the rest of the population (and often requires a bit of mental instability), is the capability to embed 'emotion' directly in their work for all to see. A very rare skill.
What is driving a lot of programmers towards the 'art form' theory is their need to see what they do as 'creative'. As it involves complex analysis, and building tools to solve specific problems, there is a huge amount of creativity required to find a working solution to a set of problems. But, and this needs to be said, once that design is complete you've got to get down to work and actually build the thing. Building is just raw and ugly work, it is never pretty. If you are doing it well, it shouldn't be creative.
I don't want for example, the electrician in a new house randomly locating the plugs based on his own emotional creativity at the moment. That would suck, particularly if I have to plug things into the ceiling in some rooms because it was a bad hair day. Artists can be temperamental and have periods were they cannot work. Software developers are professionals, and come to the office every day, ready to work (lack of available coffee can change this).
As for the elitist perspective, there are certainly skills in software development, such as generalization and analysis that are extremely difficult to master. And in the case of generalization, some people naturally have more abstract minds than others. Beyond that, the 'encoding' of a design into chunks of code with specific structures is not difficult. Not unless it is just not specified. Elegance is also hard to master, and certainly some people see that right away, but most people should be able to write a basic software program that works at least. Of course, 'just working' is only a small part of the overall solution.
HARD-WORK
Software development is often 5% creativity to find the solution and 95% work to get it implemented. Sometimes, particularly at the beginning, when you are still looking for the 'shape' of the tool, it may be more like 10% or 15% creativity, but ultimately the work done to implement the code is just work and nothing more. If we want it to be more reliable, then we need to accept it as less creative.
We like to make programming hard, because it satisfies our egos. We feel good if we solve problems that we believe other people cannot solve. Our own personal self-esteem is ridding on this. But our desire to inject 'creativity' into the implementation process, is really quite insane. In no other field would it be acceptable to make a plan, then while implementing the plan, go off and do something completely different. Is our high rate of failure, not directly linked to this self-destructive behavior? Even more oddly, the often stated evolution from foolishly ignoring the 'plan', is to just not have one in the first place. That way you don't have to ignore it. That can't end well.
For me, I have grown as a programmer only when I accepted that there is less creativity in what I do, than I would like. That's OK, I look to my 'life' to fulfil some of my creative juices, with things like drawing or photography. Really good hobbies that allow unadulterated expression to bloom. With that concession, I can see software development through an objective eye. From that perspective I found I could build faster, better and more accurately.
There are very complicated parts of programming. In particular, analysis, because of the inherent messiness of people is hard and often because it is the initial cause that knocks over the other dominoes in the project. But, once in having decided on what it is that we need to build, the 'hardness' of the problem is significantly reduced. The creative part comes and goes. We shouldn't hold onto it because we want to appear smart, or because we are trying to be an artist, or even because it makes the days more entertaining. Adding 'artistic' inconsistencies to a program is fun, but people hate it, it is messy, and it ends up being more work. You get a lot of misery in exchange for very little creativity.
PROBLEM: DISCIPLINE
Good programmers write elegant bits of code. Great programmers write consistent code. More so than any other 'skill', programmers need self-discipline to keep their code clean and consistent, after all they will be working on it for years and years. Forget all of the techniques, approaches or styles, it all comes down to using one, and only one consistent way to do things. Even if that way is arbitrary, that is irrelevant (well, its not, but it is close enough that it doesn't matter).
Honestly, it doesn't actually matter what style of code you write so long as you are consistent. If you implement some programming construct into the code repeatably, but identically, it becomes a simple matter of removing it and refactoring it with something more advanced. If you implemented it into the system in twenty different ways, the problem of just finding what to change and what not to change is complicated, never mind replacing the actual code. A stupid idea implemented consistently is worth far more than a grand idea implemented in multiple different ways. At least with a consistent stupid idea, you can fix it easily.
It isn't hard to stop every once in a while and refactor the code to cleanup the multiple ways of handling the same problem or consolidate several different versions of the essentially same block of code. It is just self-discipline. I've always like to start all major development with some warm-up refactoring work. In that way, as the project progresses, the code base gets cleaner. Gradually over time the big problems are handled, and removed.
For this, programmers love to blame management for not giving them the time, but if it is a standard at the start of any development cycle, the time is usually available. Just don't give management the option to remove the work, make it a mandatory part of the development. Because it is first, it will get done, even if the current cycle ends in a rush. Cleaning up the code isn't hard, it usually isn't time consuming, and it is definitely professional.
For both individuals and groups, if you look at their code base, you get a picture of how well they are operating. Big ugly messy code bases, show an inherent sloppiness. Sloppiness is always tied heavily to bugs and frequent interface inconsistencies. It is also commonly tied to schedule slippages and bad work estimates. Sloppy code is hard to use and nearly impossible to leverage. Sloppy code is common. Even most of the cleanest commercial products barely score a 6 out of 10 on any reasonable cleanliness scale, while most are far less than that. Other than procrastination, there often isn't even an excuse for messy code.
We don't like discipline because it is repetitive. We don't like repetition because it isn't creative. These two problems are clearly linked. The most 'creative' programmers often sit atop of huge messes. Sure, they get it, now. But should they ever come back in eight years and have to alter their own code, as I did, they too would get to thinking "boy, did I make a mess of this".
An important corollary of maintaining self-discipline is to accept that the code isn't just written for yourself, successful code will have lots of authors over the years.
PROBLEM: GENERALIZATION
Computers need to follow long sequences of instructions in order to implement their functionality. Using 'brute' force as a technique to write a program is an approach that comes from pounding out, in as specific a manner as possible, all of the possible instructions in a computer language that implements that piece of functionality. Aside from being incredibly long, the results tend towards being fragile. Mostly the repetitive nature of the instructions, as they get changed over time tend towards the various blocks of code falling out of sync with each other. These differences create the instabilities in behavior so often called bugs. They also make the software fugly. While brute force is slow to build, fragile and hard to maintain, it is by far the most common way for programmers to write systems.
If you generalize the code, a smaller amount of it can solve much larger problems. It is this 'batching' of code that is often needed to keep up with the demand and schedules for development. If you need to write 50 screens, each one takes a week, so you've got a year's worth of effort. But if you take two weeks to write a generalize screen that would act in place of 10 other screens, then you could compact your work down to 10 weeks. Each new screen is twice as hard as the original, but leveraging them cuts the work by 1/5. Of course you'll never be that lucky, you might only get 4 out of 5 to be generalized, but your now at 18 weeks which is still half the time. That type of 'math' is common in coding, but most people don't see it, or are afraid to go the generalization route out of fear that a screen might take 4 to 8 weeks instead of 2. Note that: 8*4 = 32 + 10, is still less than 50, and your 4X off your estimate. Doesn't that still allow for 8 weeks of vacation?
Generalization is hard for many people, and I think it is this ability that programmer's so often confuse with creativity. Generalizing is probably a form of creativity, but it is the ability to alter ones perspective that is key. To generalize a solution, you need to take a step back from the problem and look at it abstractly:
http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html
There have been many attempts to take generalizations, as specific abstractions and embed them directly in the programming languages or process. Object-oriented design and programming, for example appear to follow in the Abstract Data Type (ADT) footsteps, by taking a style based abstraction and extending it to various programming languages. Design Patterns are essentially style based abstractions that are commonly, and incorrectly used as base implementation primitives. While a language or a style may provide a level of abstraction, the essence of abstracting is to compact the code into something that is generalized, reusable and tight. Any of the inherent abstractions provided assistance, but used on their own or abused they will not meet the requirements of really reducing the size of the problem. Often, when improperly used they actually convolute the problems and make the complexity worse.
Even as we generalize, we still need the solution to be clean, consistent and understandable by the many people that will come in contact with the code over its years of life. Abstractions that convolute are not the same as ones that simplify. Focusing on the data always helps, and researching existing algorithms can be a huge savings in experimentation. Often the best approach is to start with some simplified algorithm, and extend it to meet the overall problem space.
This higher-level view simplifies the details, allowing less code, reuse, optimizations and many other benefits. Of course it is harder to write, but with experience and practice it comes smoothly for most programmers. It should be considered a core software developer skill.
Generalization, when it is implemented properly is an abstraction that is encapsulated at specific layers within the system. An elegant implementation of an abstraction is the closest thing we have to a silver bullet. Most programming problems resemble fractals with lots of little similar, but slightly different problems all over the place. If you implement something that handles the base and can be stretched in many ways to meet all of the variations, then one simple piece of code could solve a huge number of problems. That type of leverage is necessary in all programming projects to keep up or meet the deadlines. The reduction of code is necessary to allow for easy expansions of the functionality.
PROBLEM: ANALYSIS
If you have master the first three things in this post, you have the ability to build virtually anything. However, that doesn't help you, unless you actually know 'what' to build.
The weakest link and clearly the hardest task of most programming projects is the analysis. It is usually weak for several key reasons (a) the problem domain is not well understood, (b) the other influencing domains, like development and operations are not factored in and (c) the empathy for the user is missing, or was misdirected.
Software is just a tool for the users to work with. The only thing it can actually do is be used to build up a pile of data. Given that simplicity you'd think it would be easy to relate the tool back to the problem domain. But, as with all things, two common problems exist, (1) the designers never leave their shop to take a look at the actual usage of their tools, and (2) even if they do, they don't do it enough, so that they have the wrong impression about what is actually 'important'.
To be able to build up a workable software tool, you first need to understand the vocabulary of the industry. Then, you need to understand the common processes. Then you need to tie it together to find places where specific digital tools would actually make the lives of the users easier, not harder. Nothing is worse than a tool that just makes misery in a poor attempt to 'control' or 'restrict'. If you want the users to appreciate the tool, it needs to solve a problem for them. If you really want them to love the tool, it needs to solve lots of problems.
The key to understanding a problem domain is listening to the potential users. They have the various 'fragments' that are needed to piece together the solution. Although the users are the experts in the problem domain, the developers should be the experts in building tools. That means that the nature and understanding of the problem comes from the users, while the nature and understanding of the solution comes from the developers.
Analysis is an extremely difficult skill to master, it is not just taking notes, writing down 'requirements' or any other simple documentation process. It is listening to what the users are saying, and reading between the lines to put together a comprehensive understanding of what is really driving their behavior and how that relates back to their need to amass a big pile of data about something. In the end, thought, it is always worth remembering that software is just a tool to manipulate data. Knowing a lot about the users is important, but knowing everything is not necessary. It is usually best to stick close to the data, the processes and the driving forces behind their actions.
Understanding the problem domain is important, but the whole space is not just the problem domain, but also the development and operations domains. How the program is created and maintained are often key parts of the overall problem. Building software in a place that cannot maintain it is asking for trouble. Building something that is hard to install is not good either. The development, testing and deployment domains have an effect if you consider the 'whole' product as it goes out over time and really gets used.
Removing big programming problems produces more stable releases, requiring less testing. That in turn means the code gets into usage faster. If you utilize this, the releases can be smaller, and the benefits get to the user faster. This shorten cycle of development, especially in the early development days helps to validate direction more closely. It doesn't save you from going down the wrong path, but it does save you from going down too far to turn back.
An important point for analysis is to always err on the side of being too simple. The primary reason for this is that it is easy to add complexity when you determine that it is needed, but very difficult to remove it. Once you've zoomed yourself in believing a model or rules are correct, deleting parts of it becomes impossible. Working up from simple, often meets with the development goals and the product goals as well. Another reason is that a tool that is too complex is not usable, while one that is too simple is just annoying. It is better to keep the users working, even if they are frustrated, then it is to grind it all to a halt. They have goals to accomplish too.
Many programmer's are quick to put down the complains of their users as misdirected, thinking they just noise, or ignorance. Where there is smoke, there is usually fire. A lesson that all programmers should consider. The biggest problem in most systems today is the total failure of the programmers to have any empathy for their users. And it shows. "Use this because I think it works, and I can do it", produces an arrogant array of badly behaving, but pragmatically convenient code. That of course is backwards, if the code needs to be 'hard' to make the user experience 'simple' then that is the actual work that needs to be done. Making the code 'simple', and dismissing the complaints from the users, is a cop-out to put it mildly.
People are messy, inconvenient, lazy and they almost always take the easiest path. Sometimes this means that you need to control their behavior. However, overly strict tools diminish their own usefulness. Sometimes this means giving the users as much freedom as possible. However this is more work. Balancing this contradiction is a key part of building great tools. If the essence of software development rests on deterministic behavior, the essence of understanding users rest on their irrational and inconsistent actions. This is nothing wrong with this, in so long as you build any 'uncertainty' into the architecture. If the users really can't decide between two options, then both should be present. If they choose easily, then so can you. Match the fuzziness in the responses of the users with the fuzziness in the analysis with the fuzziness in the design.
Feedback is the all important final point. If your tool simplifies the issues, then there will be problems with its deployment. This should feedback into the upcoming development cycles, in a way that allows for the overall understanding to grow properly. Ultimately, some new code should be added, and some code should be deleted. It is worth noting, again, for self-discipline reasons, that any code that should be deleted, is actually deleted. It should go.
In this way, over time the tool expands to fill in the solution and more and more of the corner-cases get handled. The results of a good development project will get better over time, that may seem obvious, but with so many newer commercial versions of popular software getting significantly 'worse' than their predecessors, we obviously need to explicitly say this.
IT SHOULD, AND CAN BE EASIER
Hang around enough projects and you'll find that most 'fixable' implementation problems generally come from one the above issues. There are, of course, other issues and possibly serious management problems circling around like vultures, but those are essentially external. You just have to accept some aspects of every industry as they are forever beyond your sphere of control; a very wise senior developer once told me when I was young: "pick your battles". There is no point fighting a losing battle, it is a waste of your energy.
Those things that come from within, or nearby are things that can be controlled, fixed or repaired. Flailing away at millions of lines of sloppy code is painful, but enhancing millions of lines of elegant code is exhilarating. The difference is just refactoring. It could be a lot of time, for sure, but it will never get done if you never start, and to benefit, you needn't do it all right away. Cleaning up code is an ongoing chore that never ends.
You may have noticed that the these development problems are mostly personal things. Things that each developer can do on their own. For individuals and small teams this is fairly easy. Big teams are a whole other problem. Often the real problems with the huge teams are at their organizational level. Inconsistencies happen because the teams are structured to make them happen. At the lower-levels in a big team the best you can do is assure that your own work is reasonable and that you try to enlighten as many of your colleagues as possible to follow suit. Getting all the programmers to 'orient' themselves in the same direction is a management issue. Mapping the architecture in a way to avoid or minimize overlaps and inconsistencies onto the various teams is a design problem; part of the project's development problem domain.
Finally, you know you've stumbled onto something good when it summarizes well:
If you change your perspective and keep the discipline, mostly your code will work. If you add in good analysis it will satisfy. If you generalize, then you can save masses of work, and get it done in a reasonable amount of time. These four things more than any others are critical. Dealing with them makes the rest of software development easy and reliable.
I wasn't focusing on writing yet another list of things to simplify development, but in my other writings the same issues kept bubbling up over and over again. Sometimes even when you explicitly see something, it is not easy to put a name or description to it. In this case, as I explored more elementary topics I kept getting these 'pieces' floating to the top, but not fitting into the other works. Once I collected these four together, they fit quite naturally.
For this blog entry, I'd like to address this commonality with basic development problems, but try as much as possible to avoid falling into simple platitudes (no guarantees). These problems are elemental, mostly obvious, but surprisingly persistent. Software developers, I am strongly aware, are their own worst enemy. Sure, there are good rants to be had at the expense of management, users or technology, but the present of other problems doesn't validate the desire to overlook the fixable ones.
PROBLEM: PERCEPTION
One significant problem that rears its ugly head again and again, is our actual perception of what we are doing. We all know that if you think you will fail, you definitely will. There must also be some well-known universal truth that states if you go in with the wrong viewpoint, it will significantly reduce your likelihood of success. Even in the case where you are positive, if you really don't understand what you are doing, then you are implicitly relying on luck. And some days, you're just unlucky. If you want to be consistently good at something, you really need to understand it.
When I first started programming I subscribed to the 'magic broom' theory of programming. The best illustration comes from the episode called the Sorcerer's Apprentice in Disney's animated film Fantasia. There is at least one youTube link to it:
http://www.youtube.com/watch?v=LD8HDta7Z_4
In the episode, Mickey Mouse as a Wizard's apprentice, gets a hold of the 'magic hat', then starts issuing special commands to an inanimate broom to deliver water to the Wizard's workshop. Mostly because he falls asleep, the circumstances start to go rapidly out of control. Awaking in a panic, he is unable to stop the growing problem. A hatchet job only intensifies the issue. In the end, he even tries consulting the manual, a desperate step to be sure, but it is far too little, far too late. Is this not an excellent allegory of a programming project gone awry?
My early view of programming was similar. You issued a series of magic commands to the various 'brooms', and presto, blamo, your work is done for you. It is all magic underneath. In this animation, the fact that any apprentice with access to the hat can easily issue commands, even when they do not fully understand the consequences of their actions, relates this back to computers quite effectively. As this common quote likes to describe:
"To err is human, but to really foul things up requires a computer." -- Farmers' Almanac, 1978
Computers amplify their user's abilities; they are the mental equivalent of a bulldozer.
Needless to say, I was disappointed when I discovered that computers were just simple deterministic machines that never deviated from what they are told, and that the commands themselves where only abstractions thought up by other programmers. A significant amount of the 'magic' is just artificial complexity added in over years and years of development.
This is nothing magical about computers. They are machines that unless there is a hardware failure, consistently do the same things day after day. Of course, that's no longer so obvious to most users of our modern 'personal computers'. They have becomes so complex and often so erratic, that many people feel they have personality. Some even think that their machines are actively plotting against them.
THE ART OF COMPUTER PROGRAMMING
For a quite a while I subscribed to the 'writing software is an art form' school of thinking. This view sees programming as a very specific art form, similar to painting or writing poetry. It is an inherently elitist viewpoint based around the idea that some people are naturally born to 'it', while others shouldn't even try. Not everyone is cut out to be an artist, or in this case a programmer.
While not quite as fantastical as the magic broom theory, there is an implication that software is not repeatable, in the same way that any large great work of art is not repeatable. If you accept this, the consequences are quite scary. Given that we are dependent on software, this view directly implies that there is no way to consistently make software stable. The 'fate' of any version of the system rests with the ability to get a hold of the right 'artists'. And these are limited in quantity. So limited, that many programmers would insist that there is only a small handful. Or, at very least a few thousand. But certainly not the masses of people slaving away in IT right now.
Software as an art form also implies that there is some 'essence' that we can't teach. Great art for me goes beyond technique and materials, it happens when an 'artist' manages to capture 'emotion' in the work along with all of the other details. A grand painting inspires some emotional response, while a work of 'graphic art' just looks pretty. It's the same with musicians and pop music. Rarely does a pop song paint a vivid picture, instead people just tap along to the beat. When it 'gets' to you it is art, otherwise it is just 'easy listening'.
Given that definition, it's hard to picture a software application that directly inspires the user on an emotional level. That would be a word processor that makes you depressed just by starting it, long before you've been disappointed by its inherent bugs. Or a browser that just makes you happy, or angry, or something. The idea, then in this context is clearly silly. A tool, such as a hammer, may look pretty but it does not carry with it any emotional baggage. For that you need art, and the 'thing' that a true artist has, that is not 'common' in the rest of the population (and often requires a bit of mental instability), is the capability to embed 'emotion' directly in their work for all to see. A very rare skill.
What is driving a lot of programmers towards the 'art form' theory is their need to see what they do as 'creative'. As it involves complex analysis, and building tools to solve specific problems, there is a huge amount of creativity required to find a working solution to a set of problems. But, and this needs to be said, once that design is complete you've got to get down to work and actually build the thing. Building is just raw and ugly work, it is never pretty. If you are doing it well, it shouldn't be creative.
I don't want for example, the electrician in a new house randomly locating the plugs based on his own emotional creativity at the moment. That would suck, particularly if I have to plug things into the ceiling in some rooms because it was a bad hair day. Artists can be temperamental and have periods were they cannot work. Software developers are professionals, and come to the office every day, ready to work (lack of available coffee can change this).
As for the elitist perspective, there are certainly skills in software development, such as generalization and analysis that are extremely difficult to master. And in the case of generalization, some people naturally have more abstract minds than others. Beyond that, the 'encoding' of a design into chunks of code with specific structures is not difficult. Not unless it is just not specified. Elegance is also hard to master, and certainly some people see that right away, but most people should be able to write a basic software program that works at least. Of course, 'just working' is only a small part of the overall solution.
HARD-WORK
Software development is often 5% creativity to find the solution and 95% work to get it implemented. Sometimes, particularly at the beginning, when you are still looking for the 'shape' of the tool, it may be more like 10% or 15% creativity, but ultimately the work done to implement the code is just work and nothing more. If we want it to be more reliable, then we need to accept it as less creative.
We like to make programming hard, because it satisfies our egos. We feel good if we solve problems that we believe other people cannot solve. Our own personal self-esteem is ridding on this. But our desire to inject 'creativity' into the implementation process, is really quite insane. In no other field would it be acceptable to make a plan, then while implementing the plan, go off and do something completely different. Is our high rate of failure, not directly linked to this self-destructive behavior? Even more oddly, the often stated evolution from foolishly ignoring the 'plan', is to just not have one in the first place. That way you don't have to ignore it. That can't end well.
For me, I have grown as a programmer only when I accepted that there is less creativity in what I do, than I would like. That's OK, I look to my 'life' to fulfil some of my creative juices, with things like drawing or photography. Really good hobbies that allow unadulterated expression to bloom. With that concession, I can see software development through an objective eye. From that perspective I found I could build faster, better and more accurately.
There are very complicated parts of programming. In particular, analysis, because of the inherent messiness of people is hard and often because it is the initial cause that knocks over the other dominoes in the project. But, once in having decided on what it is that we need to build, the 'hardness' of the problem is significantly reduced. The creative part comes and goes. We shouldn't hold onto it because we want to appear smart, or because we are trying to be an artist, or even because it makes the days more entertaining. Adding 'artistic' inconsistencies to a program is fun, but people hate it, it is messy, and it ends up being more work. You get a lot of misery in exchange for very little creativity.
PROBLEM: DISCIPLINE
Good programmers write elegant bits of code. Great programmers write consistent code. More so than any other 'skill', programmers need self-discipline to keep their code clean and consistent, after all they will be working on it for years and years. Forget all of the techniques, approaches or styles, it all comes down to using one, and only one consistent way to do things. Even if that way is arbitrary, that is irrelevant (well, its not, but it is close enough that it doesn't matter).
Honestly, it doesn't actually matter what style of code you write so long as you are consistent. If you implement some programming construct into the code repeatably, but identically, it becomes a simple matter of removing it and refactoring it with something more advanced. If you implemented it into the system in twenty different ways, the problem of just finding what to change and what not to change is complicated, never mind replacing the actual code. A stupid idea implemented consistently is worth far more than a grand idea implemented in multiple different ways. At least with a consistent stupid idea, you can fix it easily.
It isn't hard to stop every once in a while and refactor the code to cleanup the multiple ways of handling the same problem or consolidate several different versions of the essentially same block of code. It is just self-discipline. I've always like to start all major development with some warm-up refactoring work. In that way, as the project progresses, the code base gets cleaner. Gradually over time the big problems are handled, and removed.
For this, programmers love to blame management for not giving them the time, but if it is a standard at the start of any development cycle, the time is usually available. Just don't give management the option to remove the work, make it a mandatory part of the development. Because it is first, it will get done, even if the current cycle ends in a rush. Cleaning up the code isn't hard, it usually isn't time consuming, and it is definitely professional.
For both individuals and groups, if you look at their code base, you get a picture of how well they are operating. Big ugly messy code bases, show an inherent sloppiness. Sloppiness is always tied heavily to bugs and frequent interface inconsistencies. It is also commonly tied to schedule slippages and bad work estimates. Sloppy code is hard to use and nearly impossible to leverage. Sloppy code is common. Even most of the cleanest commercial products barely score a 6 out of 10 on any reasonable cleanliness scale, while most are far less than that. Other than procrastination, there often isn't even an excuse for messy code.
We don't like discipline because it is repetitive. We don't like repetition because it isn't creative. These two problems are clearly linked. The most 'creative' programmers often sit atop of huge messes. Sure, they get it, now. But should they ever come back in eight years and have to alter their own code, as I did, they too would get to thinking "boy, did I make a mess of this".
An important corollary of maintaining self-discipline is to accept that the code isn't just written for yourself, successful code will have lots of authors over the years.
PROBLEM: GENERALIZATION
Computers need to follow long sequences of instructions in order to implement their functionality. Using 'brute' force as a technique to write a program is an approach that comes from pounding out, in as specific a manner as possible, all of the possible instructions in a computer language that implements that piece of functionality. Aside from being incredibly long, the results tend towards being fragile. Mostly the repetitive nature of the instructions, as they get changed over time tend towards the various blocks of code falling out of sync with each other. These differences create the instabilities in behavior so often called bugs. They also make the software fugly. While brute force is slow to build, fragile and hard to maintain, it is by far the most common way for programmers to write systems.
If you generalize the code, a smaller amount of it can solve much larger problems. It is this 'batching' of code that is often needed to keep up with the demand and schedules for development. If you need to write 50 screens, each one takes a week, so you've got a year's worth of effort. But if you take two weeks to write a generalize screen that would act in place of 10 other screens, then you could compact your work down to 10 weeks. Each new screen is twice as hard as the original, but leveraging them cuts the work by 1/5. Of course you'll never be that lucky, you might only get 4 out of 5 to be generalized, but your now at 18 weeks which is still half the time. That type of 'math' is common in coding, but most people don't see it, or are afraid to go the generalization route out of fear that a screen might take 4 to 8 weeks instead of 2. Note that: 8*4 = 32 + 10, is still less than 50, and your 4X off your estimate. Doesn't that still allow for 8 weeks of vacation?
Generalization is hard for many people, and I think it is this ability that programmer's so often confuse with creativity. Generalizing is probably a form of creativity, but it is the ability to alter ones perspective that is key. To generalize a solution, you need to take a step back from the problem and look at it abstractly:
http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html
There have been many attempts to take generalizations, as specific abstractions and embed them directly in the programming languages or process. Object-oriented design and programming, for example appear to follow in the Abstract Data Type (ADT) footsteps, by taking a style based abstraction and extending it to various programming languages. Design Patterns are essentially style based abstractions that are commonly, and incorrectly used as base implementation primitives. While a language or a style may provide a level of abstraction, the essence of abstracting is to compact the code into something that is generalized, reusable and tight. Any of the inherent abstractions provided assistance, but used on their own or abused they will not meet the requirements of really reducing the size of the problem. Often, when improperly used they actually convolute the problems and make the complexity worse.
Even as we generalize, we still need the solution to be clean, consistent and understandable by the many people that will come in contact with the code over its years of life. Abstractions that convolute are not the same as ones that simplify. Focusing on the data always helps, and researching existing algorithms can be a huge savings in experimentation. Often the best approach is to start with some simplified algorithm, and extend it to meet the overall problem space.
This higher-level view simplifies the details, allowing less code, reuse, optimizations and many other benefits. Of course it is harder to write, but with experience and practice it comes smoothly for most programmers. It should be considered a core software developer skill.
Generalization, when it is implemented properly is an abstraction that is encapsulated at specific layers within the system. An elegant implementation of an abstraction is the closest thing we have to a silver bullet. Most programming problems resemble fractals with lots of little similar, but slightly different problems all over the place. If you implement something that handles the base and can be stretched in many ways to meet all of the variations, then one simple piece of code could solve a huge number of problems. That type of leverage is necessary in all programming projects to keep up or meet the deadlines. The reduction of code is necessary to allow for easy expansions of the functionality.
PROBLEM: ANALYSIS
If you have master the first three things in this post, you have the ability to build virtually anything. However, that doesn't help you, unless you actually know 'what' to build.
The weakest link and clearly the hardest task of most programming projects is the analysis. It is usually weak for several key reasons (a) the problem domain is not well understood, (b) the other influencing domains, like development and operations are not factored in and (c) the empathy for the user is missing, or was misdirected.
Software is just a tool for the users to work with. The only thing it can actually do is be used to build up a pile of data. Given that simplicity you'd think it would be easy to relate the tool back to the problem domain. But, as with all things, two common problems exist, (1) the designers never leave their shop to take a look at the actual usage of their tools, and (2) even if they do, they don't do it enough, so that they have the wrong impression about what is actually 'important'.
To be able to build up a workable software tool, you first need to understand the vocabulary of the industry. Then, you need to understand the common processes. Then you need to tie it together to find places where specific digital tools would actually make the lives of the users easier, not harder. Nothing is worse than a tool that just makes misery in a poor attempt to 'control' or 'restrict'. If you want the users to appreciate the tool, it needs to solve a problem for them. If you really want them to love the tool, it needs to solve lots of problems.
The key to understanding a problem domain is listening to the potential users. They have the various 'fragments' that are needed to piece together the solution. Although the users are the experts in the problem domain, the developers should be the experts in building tools. That means that the nature and understanding of the problem comes from the users, while the nature and understanding of the solution comes from the developers.
Analysis is an extremely difficult skill to master, it is not just taking notes, writing down 'requirements' or any other simple documentation process. It is listening to what the users are saying, and reading between the lines to put together a comprehensive understanding of what is really driving their behavior and how that relates back to their need to amass a big pile of data about something. In the end, thought, it is always worth remembering that software is just a tool to manipulate data. Knowing a lot about the users is important, but knowing everything is not necessary. It is usually best to stick close to the data, the processes and the driving forces behind their actions.
Understanding the problem domain is important, but the whole space is not just the problem domain, but also the development and operations domains. How the program is created and maintained are often key parts of the overall problem. Building software in a place that cannot maintain it is asking for trouble. Building something that is hard to install is not good either. The development, testing and deployment domains have an effect if you consider the 'whole' product as it goes out over time and really gets used.
Removing big programming problems produces more stable releases, requiring less testing. That in turn means the code gets into usage faster. If you utilize this, the releases can be smaller, and the benefits get to the user faster. This shorten cycle of development, especially in the early development days helps to validate direction more closely. It doesn't save you from going down the wrong path, but it does save you from going down too far to turn back.
An important point for analysis is to always err on the side of being too simple. The primary reason for this is that it is easy to add complexity when you determine that it is needed, but very difficult to remove it. Once you've zoomed yourself in believing a model or rules are correct, deleting parts of it becomes impossible. Working up from simple, often meets with the development goals and the product goals as well. Another reason is that a tool that is too complex is not usable, while one that is too simple is just annoying. It is better to keep the users working, even if they are frustrated, then it is to grind it all to a halt. They have goals to accomplish too.
Many programmer's are quick to put down the complains of their users as misdirected, thinking they just noise, or ignorance. Where there is smoke, there is usually fire. A lesson that all programmers should consider. The biggest problem in most systems today is the total failure of the programmers to have any empathy for their users. And it shows. "Use this because I think it works, and I can do it", produces an arrogant array of badly behaving, but pragmatically convenient code. That of course is backwards, if the code needs to be 'hard' to make the user experience 'simple' then that is the actual work that needs to be done. Making the code 'simple', and dismissing the complaints from the users, is a cop-out to put it mildly.
People are messy, inconvenient, lazy and they almost always take the easiest path. Sometimes this means that you need to control their behavior. However, overly strict tools diminish their own usefulness. Sometimes this means giving the users as much freedom as possible. However this is more work. Balancing this contradiction is a key part of building great tools. If the essence of software development rests on deterministic behavior, the essence of understanding users rest on their irrational and inconsistent actions. This is nothing wrong with this, in so long as you build any 'uncertainty' into the architecture. If the users really can't decide between two options, then both should be present. If they choose easily, then so can you. Match the fuzziness in the responses of the users with the fuzziness in the analysis with the fuzziness in the design.
Feedback is the all important final point. If your tool simplifies the issues, then there will be problems with its deployment. This should feedback into the upcoming development cycles, in a way that allows for the overall understanding to grow properly. Ultimately, some new code should be added, and some code should be deleted. It is worth noting, again, for self-discipline reasons, that any code that should be deleted, is actually deleted. It should go.
In this way, over time the tool expands to fill in the solution and more and more of the corner-cases get handled. The results of a good development project will get better over time, that may seem obvious, but with so many newer commercial versions of popular software getting significantly 'worse' than their predecessors, we obviously need to explicitly say this.
IT SHOULD, AND CAN BE EASIER
Hang around enough projects and you'll find that most 'fixable' implementation problems generally come from one the above issues. There are, of course, other issues and possibly serious management problems circling around like vultures, but those are essentially external. You just have to accept some aspects of every industry as they are forever beyond your sphere of control; a very wise senior developer once told me when I was young: "pick your battles". There is no point fighting a losing battle, it is a waste of your energy.
Those things that come from within, or nearby are things that can be controlled, fixed or repaired. Flailing away at millions of lines of sloppy code is painful, but enhancing millions of lines of elegant code is exhilarating. The difference is just refactoring. It could be a lot of time, for sure, but it will never get done if you never start, and to benefit, you needn't do it all right away. Cleaning up code is an ongoing chore that never ends.
You may have noticed that the these development problems are mostly personal things. Things that each developer can do on their own. For individuals and small teams this is fairly easy. Big teams are a whole other problem. Often the real problems with the huge teams are at their organizational level. Inconsistencies happen because the teams are structured to make them happen. At the lower-levels in a big team the best you can do is assure that your own work is reasonable and that you try to enlighten as many of your colleagues as possible to follow suit. Getting all the programmers to 'orient' themselves in the same direction is a management issue. Mapping the architecture in a way to avoid or minimize overlaps and inconsistencies onto the various teams is a design problem; part of the project's development problem domain.
Finally, you know you've stumbled onto something good when it summarizes well:
If you change your perspective and keep the discipline, mostly your code will work. If you add in good analysis it will satisfy. If you generalize, then you can save masses of work, and get it done in a reasonable amount of time. These four things more than any others are critical. Dealing with them makes the rest of software development easy and reliable.
Monday, January 14, 2008
The Construction of Primitives
There is still plenty of 'meat' left in exploring fundamental development issues. We could spend time on the more interesting higher-order topics, but when the discussions get gummed up in problems with basic terminology, understanding or process we don't make significant headway. As a young industry, Computer Science has established some level of competency, but many of our key assumptions are still open for discussion. We have built on our knowledge, but our results are clearly not reliable.
Software developers, it seems, have all gone off to their own little worlds, a situation which is quite possibly analogous to mathematics prior to Sir. Issac Newton managing to enforce notational standards. With everyone running around using their own unique terms and definitions, we are getting a lot of impedance mismatches in our conversations. As far as I know, there isn't even an 'official' body on which we all could agree as being the respected authority on software development. Our industry has fragmented into a million pieces.
For this post I figured I would continue on digging deeply with a look into the way we choose to decompose code into primitive functions. Mostly I'll use the rather older term 'function' when I talk about a block of code, but this discussion is applicable to any functional paradigm including object-oriented, for which the term 'method' is more appropriate. But I'll stick to using function because it is the term with the least connotations.
Again, as with any of these abstract discussions this will start fairly conventionally and then get plenty weird as it winds it way into the depths.
PRIMITIVE FUNCTIONS
We are all familiar with arithmetic; it is a simple branch of mathematics that consists of the four basic operators +, -, * and /. True number theorists would also define some information with respect to the underlying set of numbers, e.g. integers, real, complex, etc. but for this discussion I'd rather stay away from group and ring theory; they are two rather complex branches of mathematics that are interesting, but not immediately relevant.
What is important is that we can also see the above operators as the functions 'add', 'subtract', 'multiple' and 'divide'. Each one of these functions takes two arguments and returns a resulting value. We get something vaguely like:
add(x,y) := return (x + y);
sub(x,y) := return (x - y);
mult(x,y) := return (x * y);
div(x,y) := return (x / y);
There are other 'wordings' that are possible to describe these functions. I think in some cases you might call them axioms, in some contexts they could represent a geometry, or we could talk about them as language tokens that have a specific expressibility. In x86 assembler the following functions for handling unsigned types are similar:
ADD -- Add
SUB -- Subtraction
MUL -- Unsigned multiply
DIV -- Unsigned divide
There are many more 'isomorphic' ways of addressing the same four functions, but in the end, they are all essentially a description of the same four 'primitive' functions that comes from the collection of operators available in arithmetic.
Whatever terminology or space you want to use to look at these four functions, what I want to draw towards you attention is not the verbiage used to describe them, nor their underlying behaviors, instead it is the relationship between the four things themselves. Note that a) they are all of the possible operators* b) they do not overlap** and c) they form a complete set***.
* all of the operators for 'arithmetic' only, on the ring for real numbers for + and * or something like that (it has been decades since I took ring theory).
** multiply is really X added Y times to itself, but you need the Y for the number of adds, so even though it is rooted in addition, it is still something unique.
*** add is the inverse of sub, div is the 'sortof' inverse of mul (see group theory). They have all of the consistent and complete properties: associative, commutative, inverse, entity, etc.
We can see that these four primitive functions form a 'language' for which we can express all possible arithmetic problems. If you need to do any arithmetic, all you need to know and understand are these four functions. They are simple, consistent and complete.
On a lexicographical note, I prefer to the use the term 'primitive' to describe any set of functions which do not overlap with each other. I think 'atomic' was another term we used in school, because each operation was an atom at the data structure level -- which might have possibly been a pre-quantum-physics usage -- but we have added so many connotations to that term that it has a very definitive meaning to most people. Also, 'atomic' sometimes means a function that will execute in one all-or-nothing shot, such as 'test and set'. Other terms like elementary, fundamental and primal, etc. are OK, but 'primitive' works well because we use it in regard to type information as well, e.g. 'int' is a primitive type. So, the functions are primitive, but not in the sense that they are crude. It is the sense that they are at the very bottom level and are not decomposable into smaller 'primitives'.
A DIFFERENT SET OF FUNCTIONS
So far, so easy. Just a set of the four base functions that completely cover and define arithmetic. Now suppose we created eight more functions, each taking two values, and working them as follows:
f1 = (2*x*x - y) / 2*x
f2 = x + 2*y
f3 = 4*x
f4 = 3*x
f5 = x*(-2*x - 5*y)
f6 = 2*x*y
f7 = x - y
f8 = y*( -2*x*(x + y) + x)
What is interesting about these eight functions is that none of them is identical to any of the original four functions. Also, they are obviously quite complex. Interestingly, combined they cover the exact same space as the original four functions:
add = f1 - f2 / f3
sub = f2 * f4 + f5
mul = f4 + f5 / f6
div = f6 * f7 + f8
However, as functions they have a significant amount of overlap with each other. f3 and f4 for example are close but not identical. Most of the functions use the * operator. All of them have at least one one underlying arithmetic operator, while one has as many as 6. Together they form another 'complete' interface for arithmetic, but they are clearly not primitive functions. They are composites.
So, we have two complete sets of functions, each forming an API that can be used to express any and all arithmetic problems. They are similar, equally expressive, but clearly the second one with eight functions is considerably more complex than the first. It would be possible to memorize all eight functions and use them entirely in place of +, -, * and /, but given their rather useless added complexity who would bother?*
*You can speak Klingon? You're kidding :-O
Now this might seem to be a contrived example, but it happens frequently on a larger scale. Consider a library for a graphic user interface (GUI) that consists of 300 basic functions. Now consider another one that has over 4000 functions. It is obvious that if we could find 300 functions to express most of the functionality needed for handing a GUI application, then few of the 4000 functions would likely be 'primitive'. While the overlap between the two libraries might not be exact, without even getting into the specifics we can start seeing when things are composed of primitives and when they are composed of many composite functions.
But we do have to be a little careful. There can of course be multiple different independent sets of primitives that share the same 'space'. The classic example is with Boolean logic. AND, OR and NOT form a complete set of operators. NAND by itself does as well. The two sets of primitives are completely interchangeable, consistent and complete, although one is three times the size of the other.
Primitive functions can exist for anything, although we generally tend to see them as elementary operations on things like abstract data types. For example, linked-lists generally have the functions: new, find, delete and insert defined. Although data-structures are a more popular usage, every sequence of instructions can be broken down into a reasonable set of primitives. And every set of primitives can be consistent and complete.
WHY IS THIS IMPORTANT?
We break our programs into smaller pieces so that we can manage the complexity.
At each different layer we provide some type of interface so that ourselves or more often, other programmers can access the underlying layer. We always want the simplest possible 'interface' that can do the job, given whatever level of abstraction and encapsulation are necessary. Any interface that is even marginally more complex is 'artificially' complex, because it is possible to remove that complexity.
There may be circumstances where artificial complexity is acceptable because of the abstraction or the need for an optimized non-primitive version of the function, but that is probably a weakness in the interface and most likely represents a bad design. Non-primitives are repetitive, which increases the risk of bugs.
Our best interfaces are the ones that presents a consistent set of primitives to the user without overlap. The functions themselves form the basis for the grammar needed to spell out the problem at a higher level. We simply 'say' the higher-level problem in terms of the lower-level primitives. Arithmetic, for example is spoken in terms of addition, subtraction, multiplication and division. It forms the basis of any book-keeping system. With these basic functions we could start to lay the foundations for a double entry accounting program.
A good interface, with its consistency allows the user to remember only a small number of 'axioms' that they can then use to spell out a larger number of higher-order problems. Completeness and consistency with primitives makes it easier, and thus significantly faster to express the solution with the overall set of available functions. All of the extra artificial complexity massively gums up the works. It is easy to remember the four operators for arithmetic, but although it is only another four operators, try actually writing out any type of significant arithmetic calculation using the arbitrarily complex eight functions in the first example. Although the API is only slightly larger, the complexity is through the roof.
GETTIN' JIGGY WIT IT
In an earlier entry on abstraction I suggested that data was only an abstract projection of something in the real world:
http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html
That abstract view of data is extremely useful in accepting some of the behavioral characteristics of data within a computer. Extending this a bit, there are essentially only two things in all software: data and code. There is nothing else; everything is either one or the other. Going farther, all code is broken down into functions, and on a very esoteric level there is no real difference at any give point in time, between the data itself and the functions as they exist in the computer.
The key point here is 'in time'. Functions execute, but frozen in time they are just data as it exists in the system. A great visualisation example of this is a fractal generation program. Before it has been run, the fractal program will only have high-level information regarding a Mandelbrot fractal. But that information, in the form of data, is stored as a set of symbolic finite discrete parameters that explain to the computer the underlying 'details' of a fractal. When you run the program, you pick a range for the fractal to be 'visualized' within. The program generates from its internal information a large set of data that it can display as an image. This image shows what the fractal looks like at a specific range.
The information within a fractal is infinite, you can endlessly explore the depths forever, there is no limit*. But at any given point in 'time' all of the data on a computer is discrete and finite. That you appear to be browsing through this infinitely deep Mandelbrot fractal is somewhat of a trick. The computational power of your computer is stretching out the fractal definition infinitely over the time dimension while inside of the computer at point there exists just the finite set of information describing the fractal, and a few images of it at various ranges. Nothing more.
* there are algorithms, such as Karatsuba that can deal with an arbitrarily large precision, given an arbitrarily large amount of time. At some depth, physical resources like disk space may become an issues as well.
A function as a piece of data at a point in time is a hard concept to grasp. The converse, making functions out of all of the data in the system is much easier and a standard for object-oriented programming. Just about everything you access in most modern object-oriented programs is a 'method' for some underlying object. The data exists, but is almost always wrapped in a functional layer. Interestingly enough, most of the Design Patterns center around data, but some are effectively data'izing function calls, such as the Command pattern. Commands which are 'tied' to functions, once issued become data, allowing them to be 'undone'.
Although it is rather an abstract notion, data and functions are essentially the same thing. Well, at least at any discrete slice in time. There are many languages and techniques where data and functions are mixed to great effect. Being able to abstractly see one in terms of the other allows for a broader level of generalization.
PRIMITIVE DATA
Given that data and functions are mostly interchangeable, we can generalize some of our understanding about primitive functions.
Data can be primitive as well, but the internal meaning is a little different than functions. Primitive data is information that has been parsed down into its smallest usable units 'within' a system. If for example, the system stored and used Universal Resource Locators, better known as URLs, to maintain the location of dependencies on the Web, these are the base level and there is no need to decompose them any further. However some systems may need to break up the data to access smaller pieces such as the embedded host name or protocol information. In that case, they need to parse the URLs to access the sub-data. URLs may be primitive in one system, but not in another.
Dealing with non-primitive data brings up two common problems: a) redundant data and b) data overloading. If for example, the code requires both the URL and the host name, some programmers would parse one from the other and store both.
Redundant 'anything' in a computer is always dangerous because as the code is changed frequently by the programmers, there is a significant risk that changes might not occur equally to both copies. At all levels in the code, the data should be manipulated only in its most primitive form. That might change depending on the level, but it needs to be consistent with each segment of the code. Composite data should be deconstructed into primitives immediately. Keeping that type of 'discipline' in development ensures that there are far less bugs to be found.
Data overloading is a related problem. The programmer essentially combines two separate and distinct variables into a single one. Most often we find this happening with calculated conditions that are lying around in the code, such as user-defined options or internal flags. The original definition means something specific, but over time more and more code gets attached to a conditional variable, but with very different meanings. The code breaks when a fix is applied that fixes one usage for the variable, but breaks it for the other one.
The fix for this type of problem is really easy, just break the variable into two distinct variables, updating the correct data type for each. While it is a hard problem to explain, it is a common one in coding and an easy one to fix. Programmers tend towards saving a bit of space from time-to-time, so they frequently overload the meaning of existing variables, instead of creating new ones.
Primitive data is easier to see in a system because all you need do is follow the data around and see if it is decomposed any further. If it is, the 'parsing' should be pushed back as early as possible in the code, and the 're-assembly' as late as possible. If all of the data is stored at the same depth within the code, copies of it are not necessary. In many systems, there can be dozens of copies of the same data as it winds it way through the logic, buffers and caching. This redundant data is dangerous, costly and mostly avoidable.
SUMMARY
There are a couple of very important points spawned by this discussion.
The first of which is that as software developers we should endeavour to pick primitives for all of our API function calls and all of our internal data. We do this because it make the code a) easier, b) more memorable and c) less complex.
For functions, we need to realize that everything we write is an interface and the 'users' are our fellow programmers. In building a system, every function call belongs to some interface. Beyond just good design and simpler code we should also have empathy for those poor fellows stuck in the same position as ourselves. Nobody wants to have to memorize 200 convoluted function calls when 20 would do.
The same is true for data. Systems that are constantly breaking up data, then reassembling it, waste huge amounts of resources and are inherently convoluted. It is not a complicated amount of refactoring to push 'parsing' to be earlier and 'reconstruction' to be later.
If we stick to primitives, it simplifies the code and keeps it closer to optimal; we don't end up with a lot of convoluted data manipulations that aren't necessary. We really want to decompose things in the smallest consistent bits because that is the most expressive and least complex approach towards building.
For functions we also want to 'implement' the 'complete' API at any layer in the system, whether or not we think we are going to use it right way. Ignoring the 'Speculative Generality' code smell issue, whenever we implement a layer for ourselves or other programmers to use, that layer is an interface. The Speculative Generality smell is about not spending effort building something because you are just speculating that someone might use it. That is different than only doing a 'half' job on an interface. At some point, always, if half the code exists, someone will need the other 'half'. That is not speculating, that is fact. It is about finishing off what you started. Where you can add something, you should be able to delete it. If you can edit it, you can save it or create a new one. Half implemented tools are horrible.
The final point is that as consumers of other's code, we should demand that they provide to us the simplest possible code to get the job done. It is not uncommon to see very large and overly complex APIs full of overlapping, inconsistent poorly thought-out functions. These types of interfaces require considerable extra effort in order to utilize them. Not only is effort wasted in development, but it is massively wasted in utilizing this type of code. We need to be more vocal in not accepting this type of mess. If you build on a dependency that is convoluted now, how much worse will it get in the future? Thanks to 'backward compatibility' bad things no longer get removed from code, they just fester and grow worse over time. If we accept a mess now, it will only get worse. Programmers need to stop being fooled by shiny new technology; it is often only shiny because of excessive marketing.
The core thing to remember is that there is really no technical reason for us to have to frame our solutions in non-primitive constructs. And each and every time we do, we have just added artificial complexity to the system. It is always possible to get to an underlying set of primitives. It may take some work to 'find' or 'fix' the code, but the 'value' of that work is beyond doubt.
Software developers, it seems, have all gone off to their own little worlds, a situation which is quite possibly analogous to mathematics prior to Sir. Issac Newton managing to enforce notational standards. With everyone running around using their own unique terms and definitions, we are getting a lot of impedance mismatches in our conversations. As far as I know, there isn't even an 'official' body on which we all could agree as being the respected authority on software development. Our industry has fragmented into a million pieces.
For this post I figured I would continue on digging deeply with a look into the way we choose to decompose code into primitive functions. Mostly I'll use the rather older term 'function' when I talk about a block of code, but this discussion is applicable to any functional paradigm including object-oriented, for which the term 'method' is more appropriate. But I'll stick to using function because it is the term with the least connotations.
Again, as with any of these abstract discussions this will start fairly conventionally and then get plenty weird as it winds it way into the depths.
PRIMITIVE FUNCTIONS
We are all familiar with arithmetic; it is a simple branch of mathematics that consists of the four basic operators +, -, * and /. True number theorists would also define some information with respect to the underlying set of numbers, e.g. integers, real, complex, etc. but for this discussion I'd rather stay away from group and ring theory; they are two rather complex branches of mathematics that are interesting, but not immediately relevant.
What is important is that we can also see the above operators as the functions 'add', 'subtract', 'multiple' and 'divide'. Each one of these functions takes two arguments and returns a resulting value. We get something vaguely like:
add(x,y) := return (x + y);
sub(x,y) := return (x - y);
mult(x,y) := return (x * y);
div(x,y) := return (x / y);
There are other 'wordings' that are possible to describe these functions. I think in some cases you might call them axioms, in some contexts they could represent a geometry, or we could talk about them as language tokens that have a specific expressibility. In x86 assembler the following functions for handling unsigned types are similar:
ADD -- Add
SUB -- Subtraction
MUL -- Unsigned multiply
DIV -- Unsigned divide
There are many more 'isomorphic' ways of addressing the same four functions, but in the end, they are all essentially a description of the same four 'primitive' functions that comes from the collection of operators available in arithmetic.
Whatever terminology or space you want to use to look at these four functions, what I want to draw towards you attention is not the verbiage used to describe them, nor their underlying behaviors, instead it is the relationship between the four things themselves. Note that a) they are all of the possible operators* b) they do not overlap** and c) they form a complete set***.
* all of the operators for 'arithmetic' only, on the ring for real numbers for + and * or something like that (it has been decades since I took ring theory).
** multiply is really X added Y times to itself, but you need the Y for the number of adds, so even though it is rooted in addition, it is still something unique.
*** add is the inverse of sub, div is the 'sortof' inverse of mul (see group theory). They have all of the consistent and complete properties: associative, commutative, inverse, entity, etc.
We can see that these four primitive functions form a 'language' for which we can express all possible arithmetic problems. If you need to do any arithmetic, all you need to know and understand are these four functions. They are simple, consistent and complete.
On a lexicographical note, I prefer to the use the term 'primitive' to describe any set of functions which do not overlap with each other. I think 'atomic' was another term we used in school, because each operation was an atom at the data structure level -- which might have possibly been a pre-quantum-physics usage -- but we have added so many connotations to that term that it has a very definitive meaning to most people. Also, 'atomic' sometimes means a function that will execute in one all-or-nothing shot, such as 'test and set'. Other terms like elementary, fundamental and primal, etc. are OK, but 'primitive' works well because we use it in regard to type information as well, e.g. 'int' is a primitive type. So, the functions are primitive, but not in the sense that they are crude. It is the sense that they are at the very bottom level and are not decomposable into smaller 'primitives'.
A DIFFERENT SET OF FUNCTIONS
So far, so easy. Just a set of the four base functions that completely cover and define arithmetic. Now suppose we created eight more functions, each taking two values, and working them as follows:
f1 = (2*x*x - y) / 2*x
f2 = x + 2*y
f3 = 4*x
f4 = 3*x
f5 = x*(-2*x - 5*y)
f6 = 2*x*y
f7 = x - y
f8 = y*( -2*x*(x + y) + x)
What is interesting about these eight functions is that none of them is identical to any of the original four functions. Also, they are obviously quite complex. Interestingly, combined they cover the exact same space as the original four functions:
add = f1 - f2 / f3
sub = f2 * f4 + f5
mul = f4 + f5 / f6
div = f6 * f7 + f8
However, as functions they have a significant amount of overlap with each other. f3 and f4 for example are close but not identical. Most of the functions use the * operator. All of them have at least one one underlying arithmetic operator, while one has as many as 6. Together they form another 'complete' interface for arithmetic, but they are clearly not primitive functions. They are composites.
So, we have two complete sets of functions, each forming an API that can be used to express any and all arithmetic problems. They are similar, equally expressive, but clearly the second one with eight functions is considerably more complex than the first. It would be possible to memorize all eight functions and use them entirely in place of +, -, * and /, but given their rather useless added complexity who would bother?*
*You can speak Klingon? You're kidding :-O
Now this might seem to be a contrived example, but it happens frequently on a larger scale. Consider a library for a graphic user interface (GUI) that consists of 300 basic functions. Now consider another one that has over 4000 functions. It is obvious that if we could find 300 functions to express most of the functionality needed for handing a GUI application, then few of the 4000 functions would likely be 'primitive'. While the overlap between the two libraries might not be exact, without even getting into the specifics we can start seeing when things are composed of primitives and when they are composed of many composite functions.
But we do have to be a little careful. There can of course be multiple different independent sets of primitives that share the same 'space'. The classic example is with Boolean logic. AND, OR and NOT form a complete set of operators. NAND by itself does as well. The two sets of primitives are completely interchangeable, consistent and complete, although one is three times the size of the other.
Primitive functions can exist for anything, although we generally tend to see them as elementary operations on things like abstract data types. For example, linked-lists generally have the functions: new, find, delete and insert defined. Although data-structures are a more popular usage, every sequence of instructions can be broken down into a reasonable set of primitives. And every set of primitives can be consistent and complete.
WHY IS THIS IMPORTANT?
We break our programs into smaller pieces so that we can manage the complexity.
At each different layer we provide some type of interface so that ourselves or more often, other programmers can access the underlying layer. We always want the simplest possible 'interface' that can do the job, given whatever level of abstraction and encapsulation are necessary. Any interface that is even marginally more complex is 'artificially' complex, because it is possible to remove that complexity.
There may be circumstances where artificial complexity is acceptable because of the abstraction or the need for an optimized non-primitive version of the function, but that is probably a weakness in the interface and most likely represents a bad design. Non-primitives are repetitive, which increases the risk of bugs.
Our best interfaces are the ones that presents a consistent set of primitives to the user without overlap. The functions themselves form the basis for the grammar needed to spell out the problem at a higher level. We simply 'say' the higher-level problem in terms of the lower-level primitives. Arithmetic, for example is spoken in terms of addition, subtraction, multiplication and division. It forms the basis of any book-keeping system. With these basic functions we could start to lay the foundations for a double entry accounting program.
A good interface, with its consistency allows the user to remember only a small number of 'axioms' that they can then use to spell out a larger number of higher-order problems. Completeness and consistency with primitives makes it easier, and thus significantly faster to express the solution with the overall set of available functions. All of the extra artificial complexity massively gums up the works. It is easy to remember the four operators for arithmetic, but although it is only another four operators, try actually writing out any type of significant arithmetic calculation using the arbitrarily complex eight functions in the first example. Although the API is only slightly larger, the complexity is through the roof.
GETTIN' JIGGY WIT IT
In an earlier entry on abstraction I suggested that data was only an abstract projection of something in the real world:
http://theprogrammersparadox.blogspot.com/2008/01/abstraction-and-encapsulation.html
That abstract view of data is extremely useful in accepting some of the behavioral characteristics of data within a computer. Extending this a bit, there are essentially only two things in all software: data and code. There is nothing else; everything is either one or the other. Going farther, all code is broken down into functions, and on a very esoteric level there is no real difference at any give point in time, between the data itself and the functions as they exist in the computer.
The key point here is 'in time'. Functions execute, but frozen in time they are just data as it exists in the system. A great visualisation example of this is a fractal generation program. Before it has been run, the fractal program will only have high-level information regarding a Mandelbrot fractal. But that information, in the form of data, is stored as a set of symbolic finite discrete parameters that explain to the computer the underlying 'details' of a fractal. When you run the program, you pick a range for the fractal to be 'visualized' within. The program generates from its internal information a large set of data that it can display as an image. This image shows what the fractal looks like at a specific range.
The information within a fractal is infinite, you can endlessly explore the depths forever, there is no limit*. But at any given point in 'time' all of the data on a computer is discrete and finite. That you appear to be browsing through this infinitely deep Mandelbrot fractal is somewhat of a trick. The computational power of your computer is stretching out the fractal definition infinitely over the time dimension while inside of the computer at point there exists just the finite set of information describing the fractal, and a few images of it at various ranges. Nothing more.
* there are algorithms, such as Karatsuba that can deal with an arbitrarily large precision, given an arbitrarily large amount of time. At some depth, physical resources like disk space may become an issues as well.
A function as a piece of data at a point in time is a hard concept to grasp. The converse, making functions out of all of the data in the system is much easier and a standard for object-oriented programming. Just about everything you access in most modern object-oriented programs is a 'method' for some underlying object. The data exists, but is almost always wrapped in a functional layer. Interestingly enough, most of the Design Patterns center around data, but some are effectively data'izing function calls, such as the Command pattern. Commands which are 'tied' to functions, once issued become data, allowing them to be 'undone'.
Although it is rather an abstract notion, data and functions are essentially the same thing. Well, at least at any discrete slice in time. There are many languages and techniques where data and functions are mixed to great effect. Being able to abstractly see one in terms of the other allows for a broader level of generalization.
PRIMITIVE DATA
Given that data and functions are mostly interchangeable, we can generalize some of our understanding about primitive functions.
Data can be primitive as well, but the internal meaning is a little different than functions. Primitive data is information that has been parsed down into its smallest usable units 'within' a system. If for example, the system stored and used Universal Resource Locators, better known as URLs, to maintain the location of dependencies on the Web, these are the base level and there is no need to decompose them any further. However some systems may need to break up the data to access smaller pieces such as the embedded host name or protocol information. In that case, they need to parse the URLs to access the sub-data. URLs may be primitive in one system, but not in another.
Dealing with non-primitive data brings up two common problems: a) redundant data and b) data overloading. If for example, the code requires both the URL and the host name, some programmers would parse one from the other and store both.
Redundant 'anything' in a computer is always dangerous because as the code is changed frequently by the programmers, there is a significant risk that changes might not occur equally to both copies. At all levels in the code, the data should be manipulated only in its most primitive form. That might change depending on the level, but it needs to be consistent with each segment of the code. Composite data should be deconstructed into primitives immediately. Keeping that type of 'discipline' in development ensures that there are far less bugs to be found.
Data overloading is a related problem. The programmer essentially combines two separate and distinct variables into a single one. Most often we find this happening with calculated conditions that are lying around in the code, such as user-defined options or internal flags. The original definition means something specific, but over time more and more code gets attached to a conditional variable, but with very different meanings. The code breaks when a fix is applied that fixes one usage for the variable, but breaks it for the other one.
The fix for this type of problem is really easy, just break the variable into two distinct variables, updating the correct data type for each. While it is a hard problem to explain, it is a common one in coding and an easy one to fix. Programmers tend towards saving a bit of space from time-to-time, so they frequently overload the meaning of existing variables, instead of creating new ones.
Primitive data is easier to see in a system because all you need do is follow the data around and see if it is decomposed any further. If it is, the 'parsing' should be pushed back as early as possible in the code, and the 're-assembly' as late as possible. If all of the data is stored at the same depth within the code, copies of it are not necessary. In many systems, there can be dozens of copies of the same data as it winds it way through the logic, buffers and caching. This redundant data is dangerous, costly and mostly avoidable.
SUMMARY
There are a couple of very important points spawned by this discussion.
The first of which is that as software developers we should endeavour to pick primitives for all of our API function calls and all of our internal data. We do this because it make the code a) easier, b) more memorable and c) less complex.
For functions, we need to realize that everything we write is an interface and the 'users' are our fellow programmers. In building a system, every function call belongs to some interface. Beyond just good design and simpler code we should also have empathy for those poor fellows stuck in the same position as ourselves. Nobody wants to have to memorize 200 convoluted function calls when 20 would do.
The same is true for data. Systems that are constantly breaking up data, then reassembling it, waste huge amounts of resources and are inherently convoluted. It is not a complicated amount of refactoring to push 'parsing' to be earlier and 'reconstruction' to be later.
If we stick to primitives, it simplifies the code and keeps it closer to optimal; we don't end up with a lot of convoluted data manipulations that aren't necessary. We really want to decompose things in the smallest consistent bits because that is the most expressive and least complex approach towards building.
For functions we also want to 'implement' the 'complete' API at any layer in the system, whether or not we think we are going to use it right way. Ignoring the 'Speculative Generality' code smell issue, whenever we implement a layer for ourselves or other programmers to use, that layer is an interface. The Speculative Generality smell is about not spending effort building something because you are just speculating that someone might use it. That is different than only doing a 'half' job on an interface. At some point, always, if half the code exists, someone will need the other 'half'. That is not speculating, that is fact. It is about finishing off what you started. Where you can add something, you should be able to delete it. If you can edit it, you can save it or create a new one. Half implemented tools are horrible.
The final point is that as consumers of other's code, we should demand that they provide to us the simplest possible code to get the job done. It is not uncommon to see very large and overly complex APIs full of overlapping, inconsistent poorly thought-out functions. These types of interfaces require considerable extra effort in order to utilize them. Not only is effort wasted in development, but it is massively wasted in utilizing this type of code. We need to be more vocal in not accepting this type of mess. If you build on a dependency that is convoluted now, how much worse will it get in the future? Thanks to 'backward compatibility' bad things no longer get removed from code, they just fester and grow worse over time. If we accept a mess now, it will only get worse. Programmers need to stop being fooled by shiny new technology; it is often only shiny because of excessive marketing.
The core thing to remember is that there is really no technical reason for us to have to frame our solutions in non-primitive constructs. And each and every time we do, we have just added artificial complexity to the system. It is always possible to get to an underlying set of primitives. It may take some work to 'find' or 'fix' the code, but the 'value' of that work is beyond doubt.
Subscribe to:
Posts (Atom)