Computer programming, for many coders is essentially creating a series of instructions that need to be executed. The order of execution is unimportant, so long as all of the instructions have been completed. This instruction-oriented perspective generally implies that the breakup of the actual instructions themselves into methods or functions is more or less arbitrary. So long as enough instructions are executed, these programmers don't seem to mind the structure or order. It is a common way of seeing code, but it has its own inherent problems.
Over the years, I've found that the more randomized the code base, the harder it is to make it work properly. Random and brute force code both have the annoying attribute that you cannot easily tell visually whether or not the code is correct, or even close to correct. On the other hand, a well-balanced, well-structured program not only looks cleaner, but if all of the pieces are in the right place, the imperfections are obvious. Bad code stands out.
Yes, you can see the bugs caused by the inconsistencies in construction. They become obvious blights on an otherwise clean canvas. You should be able to read the code and have some idea of what it is doing, and whether or not it will work correctly.
Clearly, being able to visually detect inconsistencies in code is a highly critical aspect in achieving high quality. Testing is hit or miss, with never enough resources, getting it right at a lower lever is far more effective.
Since we're frequently digging into the code, if it is obvious that there are problems that need to be corrected, it is easier to correct the problems as they are found, rather than allow them to build up into bigger issues.
If code is just some mysterious mess until it's running in a debugger, then new code is tossed in haphazardly, causing a toxic buildup. Relying on a debugger is a dangerous practice because you're only walking through one specific path at a time, this makes it easy and likely that the corner cases will have a significant number of bugs.
While paradigms such as object-oriented were intended to discourage programmers from creating spaghetti code, they can't actually stop it from happening. Logic spread randomly through a messy series of objects, even if they have plausible real-world sounding names, is no better than a random series of functions and procedures. There must be structure to the code, or else the code is a mess.
Nothing in code should ever be arbitrary or random. Ever.
The way to avoid these types of problems is by effectively normalizing the code. Relational database theory has a similar concept, whereby a set of rules is applied to a schema in increasing order to make it more and more orderly. The most orderly version, known as 4th normal form (or possibly 5th, I always get that confused), is considered to be the correct one to used in most general circumstances. Certainly, even if the schema has been denormalized for performance, most good data architectures are still well aware of the equivalent fourth normal form version of their database. They know what it is, before they choose to violate it.
The process of normalization applies an increasingly strict rule-base on an existing structure, to force it into some generalized simplifications. You can't arbitrary simplify everything:
http://theprogrammersparadox.blogspot.com/2007/12/nature-of-simple.html
but these rules of schema normalization take into account the necessary variables to bring down the schema's redundancy and overall complexity. There are no doubt trade-offs made, but they are fairer than just leaving it up to chance.
Code too, can be modified with a simple set of rules until it is in a cleaner more normalized version. Simplifications, particularly when there are subjective elements are never straight-forward, but within reason the purpose of applying rules to a code base is to amplify the readability without incurring a considerable expense to the performance. This brings down the major variables into something more tangible.
This isn't a particularly new idea, refactoring has been around for ages, but I often think that many people aren't applying it effectively because they have no idea what it should become. It's a series of small transformations based on localized issues, but that still leaves it all rather arbitrary. When, where and why at a global level do these things help the code, and when are they actually making it worse?
FROM EXPERIENCE
For this posting, which is going to be very long, I want to go through my own perspective on code. In particular, on how I see it's internal structure and why I think it can be normalized. It's a long, and often painful argument, but without people understanding the foundations I have no really good way of just boiling this down into nice little bits of advice. Sorry.
In one of my very early development experiences I was lucky enough to work with a medium sized code base that had been heavily edited by a lot of very determined programmers. The results were as close to elegant as I've ever been able to see in any real production code.
Everything had its place, and there was a place for everything; it all fit nicely, it was obvious and everything was right were you would expect it. If you closed your eyes and guessed where a specific set of instructions would have been placed, you'd find that they were almost always exactly were they should be. It was in its own technical way incredibly beautiful.
Fixing and extending a good normalized code base is a pleasant task, hacking away at some pile of muck is not. There are less bugs, changes are obvious and extending the original code is actually fun. Because I had a good experience early on, I was always really sensitive to the difference between a disorganized mess and something more elegant. And, more importantly I was always aware of just how little can often differentiate the two.
The biggest problem has always been trying to explain this to other programmers. I can go on forever about attributes and properties of code, but to most people that just doesn't stick. My "obsession" with arrangement to some, seems counter-productive, but only until they've learned for themselves that in getting it right, we don't have to needless keep pounding on the same code hoping that it will work better each time. Good code is easy to make work, poor code is not. Good code is always less work.
To some, the concept of elegance may not make sense, but to someone who's seen it, it is crazy not to build things this way. I'm not into putting in any extra work into my projects, I've just learned that keeping the code clean, simple, consistent and elegant is actually the least amount of effort. If we keep up with discipline, the workload decreases. It's that real understanding of how much time gets wasted with sloppy code and quick stupid changes that drives me forward, nothing else. The shortest path to a good long term product is via elegance. I know this to be true from multiple different working experiences.
To get the full sense of normalization, you need to understand the context, so I'll start off this discussion with a few abstract perspectives on code. Weird, yes, but entirely necessary later when understanding my (poor) attempts at normalization rules.
DECOMPOSITION
Software is a long sequence of instructions assembled for computer hardware to execute. It has a beginning, and at some point* it has an end. Over any given instance of the lifespan of a piece of software, the instructions executed are a finite list. The list may change from run to run, but it still is fixed.
* everything ends, but more specifically best practices for computer operations should involve rebooting the machine on a fixed schedule. Theoretical models of computing like finite state or Turing machines have infinite paper, but that just complicates matters unnecessarily in this case.
You can see the instructions from the hardware perspective, say as micro-code or assembler that is is functioning, but it is just as easy and convenient to see them in terms of a higher-level language, one that supports the modern notions of functions, scope, conditionals and looping constructs. Mostly for this discussion any the of the functional, or procedural languages will do (they do embed specific paradigms into their mechanics, but not enough to change their underlying nature).
In the crudest sense, if we wanted to create a new software program, we could just create a program with each and every instruction included, in its proper order. Yes, there would be a huge amount of redundancy, in fact most of the program would be redundant repetitive tasks happening over and over again.
For a program running with over an hour's worth of CPU time, there be would be a massively large number of instructions. It would be insane to attempt to sit down and type all of those instructions into a computer. Even with a totally impossible 100% accuracy, it would takes years and years to complete the work. Clearly that's impossible.
While it's one big set of instructions, most software interacts with some other control mechanism, be it a user, some hardware, or some other software. In that way we can partition the whole as just smaller sets of instructions triggered by individual entry-points into the list. Each subset performs some discrete piece of functionality and then returns back to one or more controlling loops.
In these many subsets of the instructions there are a huge number of repeating patterns of various sizes. Patterns that repeat quickly, over and over again, and longer running patterns that pay out through similar instructions for huge sections of the code. Patterns within patterns.
So we can really see software as a smaller set of lists mapped back to specific functional actions. Lists that are driven by functionality. This perspective helps to break down the big problem into smaller ones, but it's still not really that useful.
DIRECTED WITH NO CYCLES
The idea of having lots of these smaller lists does not make it easy to picture or build complex software. We need a better viewpoint for assembling the functionality.
The list-of-instructions view of software may be interesting from an conceptual point of view, but it really does not match how we build the code. To save time, energy and to make it less likely to have problems we have to take these huge lists and mentally break them down into a large number of sublists that we call names like functions, procedures or methods. The difference between the three is not important for this particular essay, so I'll refer to each smaller block of instructions as a just a function.
We continuously deconstruct the bigger lists into many many smaller ones, primarily to make the problem easier to handle. Once the functions are small enough, they become readily implementable.
A typical program consists of thousands and thousands of functions, broken down into collections based on underlying functionality and/or data. We group these functions together with various concepts likes libraries, packages, modules, etc.
Often at an even higher level, referred to as the architecture, we collect the libraries, packages, modules, etc. into larger parts, called things likes engines, subsystems, components, etc. In this way we start building up more complicated structural pieces from the little pieces that we have just torn off the main list. At each level it's just a specific term attached to a sub-list of some size.
Mostly we start by looking down on the problem, then decompose it into little pieces, and start building it up again. These layers of abstraction help us to encapsulate the massive complexity of the system, into a small finite number of discrete components that should all work together nicely. Software is total is too complex, so we must continually break it into pieces.
FUNCTIONAL PATHWAYS
Functions are a common visual representation for us. We work with them, but we've also become completely used to seeing them in other circumstances like stack traces. When an un-handled error occurs, most modern languages dump out a stack trace, a list of the the currently executing functions, at the time of the error.
This is a useful debugging device, but there is also more happening here. A stack trace is a specific pathway in the software. A collection of functions executed at a specific time, in a specific order. While we may see this as a time slice leading to an erroneous condition, the truth is that you could create a stack-trace for each and every instruction in the language. Why would you do this? If you took all of the possible stack traces, treating them as paths, you could assemble a much larger data-structure that shows the complete run-time linkages within your program. You'll get one big massive execution graph.
Nice, but it's still not leading anywhere. A graph is a rather hard structure to deal with. In it's purest definition it is just an unordered collection of vertices and edges. There's lots of theory and algorithms to deal with them, but life would easier if we continue to simplify.
We can flatten the expressibility somewhat by the realization that any cycles in the graph are caused by recursion. Function A calls function B which calls function A again. It is interesting to know where and when the design is recursive, but not a necessarily bit of knowledge for handling normalizations. Thus we can drop the recursions, by simply truncating any path at the first sign of a repetitive element.
This leaves us with a simpler structure, generally known as a dag, which stands for directed acyclic graph. What's nice about this structure, is that it pretty much looks like a tree where some of the children have been repeated in different locations. A tree where many different parents can point to the same underlying children. Thousands of functions point to utility functions like string append, for example. There is a lot of overlap.
Just to keep life mildly simpler, for the rest of this post I'll talk about the execution graph as a tree. When you see the word "tree", think dag, although I prefer the earlier term because it fits in a bit more with my concerns.
In this discussion I'm not readily concerned with recursion, or that fact that the same function pops up in multiple different places in the same tree. They may have some impact on a higher perspective, but that really shouldn't make a difference here. Because of that, we can just choose to see the whole thing as one big execution tree of functions.
THE IMPORTANCE OF TREES
Sometimes, if you get things framed with the right perspective, understanding comes more naturally. In this case, if we can see all software programs as just big trees of functions, we can make some every interesting statements about their arrangement and structure.
In most large programs in the first few levels of each tree, there is often some control looping construct such that the programmer has no influence over. Beyond that, at specific entry points in the system, a programmer can start attaching code in specific sub-trees. Simple programs might have a small number of entry-points, while complex ones might have hundreds.
Seeing a big complex program as a massive tree of functions is probably more detail than most people can handle, so we need to focus in on the details instead. We're not particularly interested in the whole of the program, as much as we are interested in specific sub-trees of the program, and often just within limited ranges (depths) for those trees. What we are most interested in is two things: the relative level of similar functions, and the sub-tree scope of any accessed data.
But we'll have to digress for the moment.
TYPES OF CODE
There are, as it were, only a small number of things that you are actually doing with your code.
Some code is basically a single long running algorithm that follows a particular set of logic to achieve a result, basically a specific connected series of instructions. In some cases, a large collection of algorithms has been stitched together conceptually in something like an engine, all co-operating with each other. More complex, but basically the same as a single algorithm.
Some code is just glue. We are tying together disassociated parts of the system, at either a very high level like a GUI interface, or a low level like an asynchronous callback. Another huge amount of glue in most systems is just taking an internal data model and allowing it to be persistent. Glue is really just a mapping between two orthogonal interfaces.
The final common type of code are those sets of reusable primitives intended to work over and over again. Common routines are here, but so are all of the explicit data handling that forms some internal model of the data that is accessed by other parts of the system. Not always, but the bulk of many large complex systems is the composition/parsing/traversal code that wraps the main data-types. We spend a lot of resources converting the persistent form into something more flexible, apply some simple type of operators and then repackaging it for long-term storage again.
Thus we have: algorithms, glue and primitives forming our most basic types of code.
Algorithms are easy to deal with, in that you really want to get the entire algorithm all into one big function. Splitting it over a lot of little functions, even if it matches some paradigm like object-oriented generally makes it significantly harder to debug. The biggest most important attribute of an algorithm is that it works. Usually it forms some anchor for the functionality, and it's often subject to permutations on input, making testing all the more critical.
A big function that handles the algorithm simplifies any of the issues, so its worth violating paradigms like object-oriented in order to maintain the oneness of the algorithm. Of course, the design of a full engine, particularly if it has lots of co-operating algorithms is considerably more difficult, as the programmer is forced to balance distributing the logic for cleanliness with making it more complex. Realistically, it often takes several attempts to find good trade-offs for complex engines, experience pushes the developers to accept having to do way more refactoring on that type of code, then is normal.
Glue code is just ugly by nature, and usually uglier in languages that don't make static initial declarations an easy process. Code that sits between any two arbitrary interfaces is inherently ugly by definition and there is little, other than comments to help. Glue is glue, and it is increasingly common in our code bases, the side-effect of having lots of underlying libraries to call. The best results are that the glue itself is encapsulated and not allowed to leak out across of the rest of the design. More about that later.
So mostly, the heart and soul of our systems are the models and primitive functions we build based around the fair amount of data that needs to be manipulated. We spend a great deal of effort in modern systems copying the data back and forth between a persistence representation and the runtime one. We generally build systems by implementing some internal model of the data we want to manipulate and then map it forwards and backwards to the other parts of the system. Forwards to the interfaces, and GUI. Backwards to the database and persistence.
For all of the complexities of modern software, there really isn't all that much happening under the hood. Sure there is a lot of copying the data around, combining it together and then parsing it again. Moving it from this block of functions, over to another one, and then back. Often there are tangles of if/else statements blocking out endless features, strange sub-loops, and scary error handling. And of course the GUI is inherently ugly, but so is the persistence handling, both parts of the system that quickly degenerate into silliness.
Early spaghetti code was such because it had no inherent structure. Concepts like abstract data-types (ADT) came along, giving us ways to create structure out of modeling the data in the system. We moved more of our code base into being nice well structured primitives. Object-orientation is just a language based implementation of that philosophy. In each case, the structure of the code is actually driven by the structure of the data. Sometimes it gets confused, and often it is not implemented that way, but that's the core of the ideas behind these paradigms.
This, I think is important to understand because it means that inherently the way we have been pushing ourselves to structure our code has always been indirectly driven by the actual structure of the data that we are choosing to manipulate. Granted, this often gets lost in modern dogma, but once we get back to understanding our execution trees and the scope of data within them, this data-oriented approach makes far more sense. Basing the system around the way data is transformed is a simpler perspective than basing it on the millions of steps needed to complete those transformations.
BALANCED TREES
Returning to our overall perspective, we can see every program as a series of entry-points into various sub-trees of functions. If we want the cleanest most simplified system then we can apply various rules at this level to move the instructions and/or functions around to achieve the cleanest, most balanced version of the sub-trees possible. The benefit of all this effort should be to reduce the system to a simple enough state that a larger degree of deficiencies become obvious visually-detectable coding problems.
Two of the key properties in the tree are balance and symmetry. Balance not only refers to width/height of the tree being optimized, it also implies that any two given sub-trees that are similar are in balance with each other, roughly the same height, width and depth and that the arguments to the different functions at the head of the sub-trees are nearly or exactly the same.
The first big property, balance, means that co-aligned primitives should sit at the same level together. All of the similar sub-trees always start together on the same level. All of the primitives are balanced if they form sub-trees of approx the same level and size. The level and depth of all similar functions should be in balance with each other.
For any instruction in a sub-tree, if there is a symmetrical instruction, it too should be in balance. For instance, an 'open' at a specific level should also have a 'close' at that level. The open/close pair should bound a block of code, visually, even if that means they exist by themselves.
This property of symmetry is important because it's absence is easily noticeable. It is a great way to spot code that is out of place. If all the functions have a starting instruction, and an ending one, then any function missing one or both is a problem. When we cannot use the computer to enforce this type of consistency, such as in aspect-oriented programming, we must do so visibly.
If the same underlying code is being used in multiple places at various different levels than that is an indicator of a problem. The underlying code and data should fit neatly into the puzzle. The more the structure is graph-like, the messier the architecture is. If all of the calls of a specific function are on the same tree level, then the use of that function is well-balanced.
MIXED PRIMITIVES AND OTHER FAUX PAS
One very common structural problem is to create a set of primitives from one interface paradigm and mix them with another set from other paradigm providing multiple redundant interfaces to the same underlying code. This common problem, that you'll often see in popular Java libraries for example, is caused by some assumption that more is better or that the library would be more beneficial if it was more flexible. Bad idea. Two overlapping primitives sets just expands out the complexity for no real benefit.
A complete primitive set forms a close loop, with just one non-overlapping operation per primitive. Simple examples are add/delete/modify or insert/update/delete, or even add/subtract/multiple/divide. What is crucial here is that all other operations can be expressed as a set of primitives, and that that total set spans all of the possible functionality. There is one and only way way to do everything with a balanced set of primitives, if there are two ways to accomplish the same goal, then one or more of the operators are overlapping.
It is far better to create two separate, clean implementations, one for each primitive set, then to mix the two together. It's just opening the door to potentially dangerous corner-case problems caused by badly mixing the calls. Why waste time working out all of the weird interactions, especially if they aren't necessary or shouldn't be used in that way. Why give programmers the means to write increasingly convoluted steps, just because they mis-understood how to work with each individual primitive set. It's the type of wasted effort that we should have learned to avoid by now.
Null handling is another common problem, although not necessarily that structural. Programmers overuse the nulls, but their purpose and point are very explicit. For instance there is no difference between an empty container and a null. Why distinguish with containers? Having to test if a container is null, and then again if it is empty is useless code. Just never allow null containers, and use the one and only condition as the indicator. Structurally empty containers overlap with nulls in virtually all usages. Nulls as an out-of-band signal for a condition are sometimes necessary, but not if their meaning is fake or artificial. To many programs are poor tangled webs of over-extended null handling.
Exception handling, another overused language feature, was intended to clean up specific low level handling code, and build a better highway for systems to pass up significant errors. Often, thought, programmers go beyond that low level, and high level usage, and start indiscriminately using it everywhere. Syntax paradigms like try/catch form secondary execution paths through the system. One nice execution path is visually verifiable, but overlap a lot of little, radically places and one quickly swamps the usefulness of the syntax.
Programming is often about restraint, self-discipline and reductionism. Exception handling is one case where it is wise to get rid of as many handler blocks as possible. For low-level external error handling, and high level handing, try/catch blocks are extremely helpful, but used anywhere else they should be eyed with suspicion.
All three of these issues are really just instances of programmers added an extra level of complexity in their instructions to over-compensate for the overall lack of structure. Wasted nulls and excessive try/catch blocks are very noticeable blights in elegant code, but just fit into the background noise in messy code.
Once the code has been balanced to some degree, it is far easier to see what can be easily deleted, because it is serving no real functional purpose.
DATA AND CODE
Looking at programs as sub-trees of functions allows us to give great consideration to the program's overall structure without getting too lost in the details. But the code by itself will not fully normalize a program into something elegant.
Programs are always composed of two distinct, and often conflicting things: code and data. The sub-trees lay a structural framework, but we also need to understand how the data access is distributed through-out the overall structure.
If we look at all of the data in the system, we can see that it is a relatively small discrete collection of data-structures, which are essentially containing all of the data-types in the system. That is, for any given system, the amount of data used in it is both limited and finite*. You could create a small fixed list of the major entities.
* even when most programmers support dynamic data representations they often do so in very static ways, defeating the full power of their dynamic code. It's a safe bet to ignore dynamic code, or at least to contain it all into a set of fixed 'dynamic' data-type (thus making it limited and finite).
This notion is extremely helpful because we can start looking at the scope of all of the data, in terms of the trees in the system. A well-balanced, normalized bit of code will encapsulate specific data structures within specific sub-trees. The data is hidden from any code outside of that tree. This is information hiding, and encapsulation (if the code is buried there too).
We want, at each functional level, for the concepts, information and ideas below that sub-tree to be a small consistent set. As we descend further down into the tree, we want a more enhanced scoping of the data. The data and code in a given fixed set of primitive string utilities for instance would, underneath it all, refer to just strings and specific manipulations. As the data sinks lower into the tree, the understanding of the data should be more and more general. Explicit parameters at a high level, are a hash table below that, and then just strings and keys below that. Thus the language we use in the code to describe variables, function names, parameters, etc all match the level and scope of the data. As the level gets deeper, the terminology gets more general.
For a normalized model, the collection of sub-trees that make up the interface all encapsulate the scope of the underlying data. Any data that gets beyond that scope "leaks" into other parts of the system, is global or is effectively global. And these problems with the data are more common than expected.
STATE AND ITS EFFECTS
A global variable is one that is accessible from any location in the entire tree. We've known for a long time that globals are considered dangerous, because they allow multiple access points in different parts of the system to quickly fall out of sync even with simple changes. A reckless change can be followed by a long and painful hunt for the culprit. Because of this, we actively try to avoid globals.
What we know is true and a big problem in the whole tree is also true and a big problem for any given sub-tree within the system. For any sub-tree, any common similar location of data is the "state" of that sub-tree. Sometimes we don't see it as such, but if multiple locations within the tree access the same variable, then it is essentially a global. Scoped, a bit, but still global.
We've known for a long time that state is bad. State hugely increases the likelihood of errors, and makes it very hard to test to see if the software is working or not. State problems may require weird compound testing methods to re-create, so they are very expensive both in terms of development and testing. Most non-obvious bugs* are a result of state problems of some type.
* OK, threading bugs in Java, and hanging pointers in C and C++ are probably way more popular, but these were "features" added to the languages to keep programmers employed.
Implicit state is still state-based, and is far more dangerous because the programmers are generally unaware of the problem. Any sort of data that is not explicitly passed in and/or out of a function is some type of state. That means that any and every side-effect is an implicit state of some kind. Any state that is changed in many different places in the program, even if the change method is a covering function call, is a defacto global variable, with all of its inherent problems and weaknesses. Any and all of the locations are dangerous.
Stateless code was a great idea and a best practice goal for a while, but that went horribly wrong with paradigms like object-oriented. Objects are inherently state-happy, and in that way they often hide other more hideous state problems from unsuspecting programmers. An instance of an object can be scattered across an execution graph like a splatter paint artwork. One object instance can easily be acting as a bad global variable for any other. it can get very ugly, very quickly.
Without careful consideration of these structural relationships, old problems that we banished for good reasons in the past can easily crop back into our code bases. Worse, they can be effectively hidden to most developers. Toss in a fine helping of threads, and it is easy to understand why so many popular applications occasionally, and often quietly cease to behave correctly. And why they do so only in a tiny fraction of their runs. Seemly random, non-deterministic problems, lurking in the background, wasting lots of and lots of time and effort.
These realizations, are one of the primary reasons why it is important to sometimes change perspective on a problem. Hidden, yet inherent flaws in one viewpoint become far more obvious and understandable from another.
WASTED RESOURCES
If you trace out the data in many systems you will find that it progresses through the code, jump-by-jump in a series of copies. Sometimes buffer copies, sometimes it is being parsed, somethings is is being reassembled. The path of any given piece of data through a system always involved lots of copies. Modern languages and paradigm have made this problem worse, causing this type of bloat to increase rapidly.
From a tree perspective, what is happing is that the data is scoped within a large series of different sub-trees. As it leaves one sub-tree, it is copied into the next one. In this sense, we can see each copy as a implicit violation of the sub-tree encapsulation. More to the point, if a large chunk of data is copied into a sub-tree only to facilitate some small set of manipulations, then that specific code could easily be moved to a more appropriate tree.
In that sense, the smaller the tree and the fewer of them that hold the data, the more encapsulated it is. Watching how the data flows through out the system gives a good indication as to a better working structure.
ARCHITECTURAL LINES
At an even higher level, we can see the architecture as how the big major sub-trees in the system are laid out with respect to each other. Balance and symmetry apply here as well as anywhere else.
To get a real architectural line between two pieces of code, they both need to have entirely separate sub-trees and data. Overlap in either crosses the line.
Encapsulation is burying all of the messy details, code and data, of something behind a small subset of sub-trees. They act as the interface, that hides all of the other detail. Decomposing a problem properly makes it easier to build a real workable solution, not just one that is close to workable. We need to encapsulate the details in order to manage the complexity of the project and actually get it done.
More importantly, libraries, modules, etc. should be organized about their underlying data not on their algorithmic code. That principle makes it really easy to just see the library as data containment functionality for a specific data type in the program. The algorithm handling and the data handling should be separated.
As a related note, often many user libraries and packages combine mashes of algorithms and data-handling that are inconsistent and unbalanced. Clearly defining the structure around well-balanced decompositions would make using most libraries considerably easier to use. We need a movement that wraps simple components based on very specific and complete access to specific data structures or algorithms, in a fully complete, access type of way. That would make the choice of using a specific library, really a decision about supporting a new type of data, and it would also cut down on new versions and upgrades.
The spasmodic and arbitrary blend of data and functionality whipped up into most modern libraries forces a constant cycle of updating, if for no other reasons than to try to get some of the contained functionality into a more complete state. For many libraries, this dynamic upgrade path is not necessary, but simply a by-product of disorganization and bad partitioning. This a clear example of why better normalized libraries would significantly cut-down on development effort.
REFACTORING
Knowing what a good structure is, doesn't help unless there is some easy and simple way to get any program there. Refactoring acts as the micro-normalization rules that can allow a programmer to start with anything and make it more orderly. Of course, simple consistency is also critical in making it all hold together.
You can see all of the refactoring algorithms as just ways of pushing and pulling the code up and down between the different levels of the tree. In this sense we can balance the functions, and then balance their usage, then balance the data, etc. Think of it like the "roll" operations in a weighted-balanced binary tree.
It is possible to take any working program, and after apply a very long series of non-volatile refactorings, return to another working program. Refactoring doesn't have to interfere with the functioning of the code, in fact it is far better to pass through with a large series of non-altering changes first, before moving onwards to expanding the code base to add in new functionality.
Not all refactorings in this way will be non-destruction, because by definition some of them will actually be removing bugs from the system. The changes in behavior are often ultimately good, but then there can be unexpected dependencies tied to buggy code. Under these types of circumstances, it's based to temporarily duplicate the code, with a new clean version and an existing broken one. That makes it possible to reassembles all of the pieces first and do some comparison testing to insure that the none of the behavior has changed, before moving on to deleting the dependencies on the broken code. Getting the code quickly back to working order quickly finds obvious problems and keep the development work moving forward in a series of small independent discrete steps.
Normalized code that has been refactored, then retested (lightly) sets a strong base for extending the system to encompass the next level of functionality. Without this type of behavior, the code simply degenerates into some hideous onion-nightmare, a sad and embarrassing state of affairs that is entirely unnecessary. Each time the code degenerates, it becomes more of a work magnet, drawing in masses of wasted time debugging stupid problems and working around fixable issues. Anybody working on that type of code base knows that pretty quickly more effort goes into badly patching sloppy problems, then goes into new development. A sad, and absolutely avoidable state.
FINAL SUMMARY
I wasn't really specific in producing a finite set of "forms" for normalizing code. But if you see it as a structural problem, then the rules themselves are less important, they are simply the easier way to transform one structure into a better one. The final structure is what's key.
Someday, I'm sure someone will come along with a clearer set of rules. Something that can easily fit onto the back of refactoring, that makes it easily understood at the higher level.
We know the code is normalized by the fact that the final structure we create, is an easy to read one. We've simplified the execution graph. The code maps to the structure, which maps back to the code again. A messy graph is usually messy code.
Be careful in applying this knowledge, for as I said in "The Nature of Simple", human based simplifications are not the same as machines ones. We are somewhat flawed, and as such our normalizations will be too. We don't want things that are truly universally simplified, just ones that are 'simpler' to us.
If that's true, then why bother? Like the database, a good developer knows what is normal form for their code, even if they don't strictly follow it. There are exceptions, but you cannot understand when they are OK, if you do not grasp the complete picture. Breaking the rules without understanding them just pushes back the success onto luck. Relying on luck fails often enough.
Don't forget, that this work, extra as it may be, isn't to be done for fun, or because it's right. It is to be done to make it easier to move the code base forward to the next version. It is to be done to clean up the old messes, and to make way for a better version. It is to be done to save time, and allow us to leverage our coding abilities better, instead of our ability to continuously hit "next" in a debugger. It's neither arbitrary nor extra, simply work that needs to be completed to insure that the overall health of the project gets better from month to month, not worse.
Sunday, October 12, 2008
Sunday, September 21, 2008
The Necessity of Determinism
Perspective is relative. You might, for instance, see a desktop computer as an item you purchase, and then update with various software to complete the package. A stand-alone machine acting as a tool to help you manage your data. This has been our traditional view of computers, and until recently it has been mostly correct.
If you take an abstract view, any given computer has exactly F ways to fail. By failure, I don't mean small annoying bugs, but rather total and complete melt down of the platform to the point where it requires a significant fix: reboot, new hardware, or new software. It is unusable without intervention.
F varies, based on the hardware, operating system and any installed software. A couple of decades ago, with a simple IBM box, DOS 6 and a normal upscale vendor for peripherals, there might have been hundreds of ways for the machine to fail. Any of the core hardware could burn out, software could go wild, or the OS could find a reason to crash. The actual number doesn't matter, we far more interested in the way its changing, and why its changing.
In a sense, on using the machine to complete a task there was one way to success, and F ways to fail. The successful branch is overwhelmingly likely, but the size of F is not insignificant.
DETERMINISM AND ITS NON
There is a dictionary definition for the word determinism that describes it as a philosophical doctrine. A belief in cause and effect. In Computer Science however, we tend to use the term more as an objective property of a system, such as a deterministic finite automata (DFA). The transitions between states in the automata are driven deterministically by the input. There is an observable causal relationship. If the system does exactly what you would expect it to do, no more, and no less, then it is deterministic.
Computers, by their very nature are entirely deterministic. The CPU starts executing a fix series of instructions, such that given the same initial pre-conditions the results will always be the same. That deterministic quality, as we'll get into more detail, is very important for making the computers a useful tool.
Interestingly enough, although their behavior is predicable, computers can be used to simulate non-deterministic behavior. In at least a limited sense, the regular expression operator * eats up an arbitrary number of characters in a string until it is finished or the expression is proven to not match. a*b behaves differently depending on the input string, matching a string like annnnb, but not one like baaaaaa.
A common way of implementing this type of functionality in software is by using a non-deterministic finite automata (NDFA), which is a rather long way of describing an abstract machine with a set of internal states where the transition from one state to another is caused by a non-deterministic reaction to the input. You just don't know when the machine will change state.
You'd think that writing something on a deterministic computer to have non-deterministic behavior would be a complex problem, but they solved that rather early in our history by spawning off heaps of DFAs for each possible state that could exist. These possible transition automata may or may not exist, one input of which may collapse all of them into a single defined state. The NDFA may be any number of possible DFAs, but similar to Quantum probabilities, they all collaps down to one (or none) at the very end.
With this knowledge, it became easy to simulate a NDFA using a dynamic list of DFAs. Non-determinism, it seems, can be easily simulated. Or at least aspects of it.
A neat trick, but it also holds a broader understanding: non-deterministic behavior is easily emulated by determinism machines. Just because the essence of something is predictable, doesn't mean that everything you do with it will be predicable as well.
INCREASING WOES
Over the years, as the software has gotten more complex, the number of possible failures, F, has risen significantly. I believe it is rising exponentially, but proving that is beyond my horizon.
These days, there are many more reasons why modern desktop computers fail. Hardware, while getting cheaper is also less in quality. Software, while more abundant has outpaced it environment for complexity. Software practices have decayed over the years. Although modern operating systems are far more protective of themselves, there are still a lot of ways left to render a machine useless. New holes open up, faster than old ones are patched. And now of course, there are people actively trying to subvert the system for profit.
If it was just an issue of an exponentially increasing F exceeding our operational thresholds for stability, technological advancements would gradually reduce the issue. We'll eventually find stronger abstractions that help with reducing these problems.
However, indirectly, we've created an even worse problem for ourselves. One that is growing more rapidly than just the normal equipment or algorithmic failures.
As a stand-alone machine, we define the number of problems in a computer as F. If we link two machines by a network, this really doesn't change. The addition of networking hardware and software increases F, but linearly, relative to the new pieces. Failures on one machine, in very limited cases may cascade to another, but that is rare and unlikely. Each machines reacts to its data independently.
If one machine becomes responsible for pushing code and data to another machine, then everything changes. The two machines are now intertwined to some degree. F might not double, but there are all sorts of reasons why the second machine may easily clobber the first. As we see all of the different chips and micro-controllers of any standard PC as a single machine, we must now see all of them for both machines as one machine. Bind them correctly, and they are no longer independent entities. The two become one.
Once the interaction goes beyond a simple question answer protocol, where the exchanged data itself can contains failures, both devices are tied to each other.
With two machines and a rather large increase in F, the behavior of the whole new machine becomes rather less deterministic. Like the NDFAs, you can see the whole as the different permutations of its kids. A big machine containing two independent simulations of other machines. We can still follow the behavior of the boxes, but it is no longer as predictable. Small differences in execution order on either of the boxes may change the results dramatically. Subtle differences that mean non-determinism.
THE NETWORK IS A COMPUTER
Almost all of our modern machines are now interconnected to each other in some way. Internal corporate intranets connect to the Internet, which connect to our home networks. We have constant, easy access to our resources from a huge number of locations.
These increases in networking have revolutionized our machines, giving us unprecedented access to a near infinite amount of data (at least relative to one's lifetime). They helped integrate these tools into every aspect of our daily existence.
Software companies have utilized this connectedness to make it easier than ever to keep our current software up-to-date. There is a mass of interaction, at both the operating system level and the application levels. There are tools great and small for automatically patching, updating, reinstalling and coordinating the software on our machines. Windows XP, Java, Firefox, Acrobat Reader and dozen of other packages check on a frequent basis for the latest and greatest updates.
Even more interesting, if you account for all the virus, spam and phishing, then there is a massive number of sources out there interacting with your machine, trying (if not always successful) to give it new data, but also new code.
In essence, we've tied together our machines on so many different levels that we have effectively taken the millions and millions of previously independent computers and created one big massive giant one. One big massive giant, vulnerable one. With an F that is shooting well out of control.
BAD HAIR DAY
In its simplest form, all it takes to trigger a failure is for one of these updating mechanisms to install a change that renders the machine useless. Although there are checks and balances to prevent such things, history is full of great examples of nasty bugs getting lose in the last final round. It is inevitable.
So now we're becoming increasingly dependent on some poor coder in Redmond not having a bad hair day and accidentally scorching our boxes in a serious flame-out.
We're also dependent on some poor coder in the outskirts of civilization hopefully not having a good hair day and finding some new great way to subvert our machines for evil purposes.
If we started by thinking that F was in the hundreds, once you start counting all of the little possibilities, and how they might intertwine with each other, F grows at a frightening rate. There is a staggering number of reasons why an average desktop computer can now fail. Millions?
Given my experiences both at work and at home, these are not just infinitesimally small probabilities either. Over the last few years I've seen more hardware and software failures for an ever changing set of reasons, then I've seen in my whole career. Yet my usage and purpose and intent with my machines hasn't changed all that significantly. In truth I do use more web applications, and spend more time on my home machine then I used to, but the increases in failures has been way over my increases in usage.
My machine at work fails every time there is an system update to a piece of software I de-installed years ago. The box locks up, hangs, and turns off my database. This occurs multiple times per year. Several of the other auto-updates have crippled my box on occasion as well.
My friend's machine rebooted on a usb key insertion. My wifes machine has seized up for a number of unknown reasons.
If it were just me, I could forget about it, but virtually everybody I know has been entangled with at least one good computer bug story in the last couple of years. I'd start listing them out, but I don't think it necessary to prove the point, if you use your machine significantly, you're probably already aware of this.
In truth, it has been this way for a while.
Way back, I remember a system administrator telling me that Windows NT works fine, so long as you don't touch it. Sadly, he was serious, and defending the platform, which I had been verbally abusing because it keep crashing while running some test software. Nothing about the software should have effected the box, yet it was. Desktop computers have long since had a history of being flakier than any of their earlier cousins.
PHYSICAL DETERMINISM
A world-spanning massive super machine is both an interesting, but also a scary idea. Our little desktop machine is just a tiny piece of this bigger cube, entirely subject to its will, not ours. We only get to use it to complete our tasks if we are lucky. The F for this machine is way too high.
Purchasing a time-share on a massive computer is an interesting prospect, but would the circumstances be helped by achieving a massive bump in quality? If there was better security and better testing, would this change things?
These are very tough questions to answer simply. The short answer, often demanded by a younger, and less patient generation is that tools need to be deterministic, and that our current methods of interconnecting our machines defeats this entirely. In its own philosophy, we could say that the bugs are just the effects, the underlying cause is the lack of determinism.
To really understand that, we must start with a very simple observation. A bulldozer is a tool used by construction workers to move large volumes of earth from one place to another, and to flatten big areas. Bulldozers act as a tool to leverage the power of the machine to accommodate the drivers actions. The machine extends the driver's abilities.
It is entirely possible to build some type of Rube Goldberg contraption that given a setting, puts the bulldozer through a precise set of instructions. Physical, like the machines that help manufacturing lines assemble complex objects, completely non-computerized, just physics. We could build it so that the bulldozer is dropped off in a location, turned on and then it would do a specific operation like clear an area of land, or move a big pile of dirt somewhere. Set the switch to 40x40 and you get a huge square of land precisely flattened.
This type of mechanical automation could be used to remove the necessity of the driver, whom after all is just sitting there operating the machinery. The problem is that, in operating by itself, even with a very fixed simple set of rules, there are always unexpected circumstances that will creep up. In order to prevent serious accidents, someone must monitor the progress of the machine, and it is far better for them to be working with it while this happens, then for them to be sitting on the sidelines in a chair.
You can automate a factory because it is a limited controlled environment, but you'd never be able to automate something for an un-controlled one, common sense and safety keep us from doing so. Even the tiniest of accidents would draw a huge storm of protest.
The tool is best if it deterministically caries out the instructions of the driver, extending their abilities to do manual jobs. For both the factory and the bulldozer, determinism is a crucial aspect.
INTELLECTUAL PURSUITS
While that makes sense in the physical realm, people often differentiate between physical effort and mental effort. It is one of those class hold-overs where physical is seen as less desirable. In reality, most physical jobs demand a huge degree of mental energy, and sometimes this shifts over time. Los Alamos relied on human calculators in the pre-computer days, to do all of the "manual" calculation work, now easily seen as intellectual effort handled easily by a simple calculator. An Olympic class athlete's brain is working overtime trying to control their reactions to a massively precise level, a huge feat in thinking.
Effort of all kinds is really just a mix of some percentage between physical and mental. The two are closer than most people care to admit, some analogies relate the brain to being a muscle for thinking. Just another way to expend effort.
As anybody who has ever tried to automate a complex intellectual task knows, if you can't contain it there are as many possible obstacles to getting it to work correct as their are for the bulldozer. We cannot predict all of the variability of the real world, be it a mountain of dirt in a field, or a mountain of information in a corporation. It might, in some way be possible to account for all things, but realistically we need to assume that the possible failure conditions are infinite. Some are tiny, but it's still infinite.
REACTIVE INTELLIGENCE
In a world of unpredictability, we'd need a extremely complex calculation engine to be able to cope with an infinite variety of errors. Artificial intelligence is the much heralded savior for our woes. The only problem is that some people strongly believe that it is not possible, and have constructed proofs for such a truth. Many people believe that the failure in research to achieve it already is proof enough.
It is an interesting debate, and I'll digress a little into it. I have a tendency to think it is possible, but I think that most people studying it have grossly underestimated its inherent complexity. In a sense, they are looking for a simplified, pure, abstraction that provides this heighten capabilities. An abstraction that fits the neat ordering of our thinking.
I tend to see thinking as being intrinsically messy, and in many cases such as creativity, quite possibly flawed. I've often suggested that creative sparks are failures to keep things properly separated in our minds. A flaw, but a useful one.
A good simple example is the Turing test, a simple way of determining if something is "intelligent" or not. A person blindly interacts with a couple of entities with the intent of finding out which of them is a machine, and which is a human. If the person cannot distinguish between the two, then the computer's behavior has been deemed intelligent.
The problem with this test comes from the original episodes of Star Trek. Given a Vulcan -- an idealized, overly logical race of beings -- as the machine entity in a test against a human, most people would assume the Vulcan to be a computer. The logic and lack of emotion would be the keys. A Turning test for a human and a Vulcan would fail, showing the Vulcan as not being intelligent.
However, our idealized alien, at least in a TV-show script sort of way, is in fact an intelligent being quite capable of building star ships and traveling through the universe, a massively complex and clearly intelligent feat. Although it is only a TV show, it does act as a well constructed thought-based test. There is at least one alien intelligence we can conceive of that would intrinsically fail the Turing test. It would appear as a computer without intelligence, when it clearly wasn't.
I see this as hugely important in that the researchers are out there looking for something idealized. Something pretty, something with structure. They are searching in a specific location in a rather large field, my hunch is that their prey is hiding at rather an opposite location. And, as always, we can't search there, until our current path(s) take us closer to that area. I'd guess that we have to understand the structure of knowledge first, before we can learn to reliably extract out specific relationships.
It's likely to turn out that our own hubris-based definition of intelligence may actually be a big part of the problem.
GOING TO THE DOGS
What we consider as intelligence isn't often so. There are lots of examples, but fear and politics always provides the most interesting ones.
Over time, various media have discovered that shocking scary stories help sell the news. Because people like simple stories, there has been a tendency to report dog attacks, but only if the animal is described fully or partially as a Pit Bull. The term has come to invoke fear, and the common stereotype is of a vicious animal most often owned by drug dealers or other nefarious folk.
It's always a good simple story that gets a reaction. Often the dogs in question aren't even remotely Pit Bulls. The largest size of the breed is around 65-85 pounds, but its not uncommon to see stories about 120 pound "Pit Bulls".
It also doesn't matter that all other dogs bite, there are millions of Pit Bulls, nor that most of the supposed Pit Bulls are actually just mixes of many other breeds. Somehow this one domesticated animal has been granted superior dog capabilities. We've been living closely with the "child's nanny" for over a hundred years, they are the mascot of WWI, the RCA icon, appeared with children in films and have thousands of other cultural references. However, people wanted a villain, and the Pit Bulls were appointed.
Fatalities is where the anti-dog fanatics really like to focus, but they are slow to compare the dog-based numbers to those of other animals such as horses or even cows. Cars and guns kill millions every year, and somehow a tiny number of deaths becomes a growing epidemic. Yes, interaction with animals sometimes ends badly, that has always been true, no matter what animal is under consideration, domestic or wild. Anybody who has ever owned pets of any kind easily understands this.
Hype, misinformation, etc. aren't new, but we live in an age were it is becoming harder and harder to get away with this, and were ironically we are getting more and more of it. So, as a side effect of selling newspapers and nightly news casts, one of North America's most distinctive dog breeds has been used as the scape-goat for all of our angst about nature. Ironic, particularly in a period of intense "green" frenzy. And what more would we expect from this?
Of course some regional government, desperate to look "proactive" seizes the day and passes a breed-specific law banning Pit Bulls from an entire province. Sad, given that the negative hype was initially profit driven. The underlying facts aren't there, or are irrelevant. "Surely anybody who reads the newspaper would know that Pit Bull bites have reached an epidemic status and something needs to be done!"
The twist in all of this, is that we use our intelligence to build up and maintain our collective set of social rules. Because we live in an "intelligent" society we make rules and laws based on our understanding of the world around us. Smart people supposedly come together to lead us.
Ironically, the law didn't even really pass correctly. The dogs they were trying to address were mixes of Pit Bulls, but the law on docket is now restricted to only pure Pit Bulls, dogs with papers. It's all but useless, other than having taken away the newspapers ability to print good "Pit Bull" stories, and keep breeders from selling pups. It's a sad example of irrational fear gone completely wrong.
A new set of laws exist that don't do what they were supposed to, and the only reason they were created was to keep the public believing that a political party was actively trying to solve a problem (even if it was a trivial, non-existent one). It's insane, and clearly not intelligent. Even if you buy into the anti- "Pit Bull" hype, the resulting law itself is completely in-effective, a failure on top of a failure. The pure Pit Bulls have been all been replaced by mixed breeds. Nothings changed. All we have is another useless set of laws drafted by supposedly intelligent beings.
EXCEEDING DILBERTIZATION
This is but one simple example of how we have been building up our rules, our political systems, our organizations and our knowledge over the decades. We're exponentially piling on Dilbert inspired thinking to poorly solved unrelated problems. We have absolutely no way of disproving bad ideas or compressing our increasing mass of mediocre ones. Any new legislation from any existing body is equally as likely to be bad as it is good. And oddly, it is not just subjective taste, there are massive examples of clearly stupid short- or long-term ideas getting passed as though they were somehow intelligent. We live in a depressing era. Good for cartoonist, bad for intelligence.
Given the arbitrary mess of our collective knowledge, its easy to see how we appear as kids just barely beginning to qualitatively think about our surroundings. There was a time without zeros and negative numbers, pretty hard to conceive of now, but it wasn't that long ago. There will come a time when we can show an irrational bit of thought for what it really is. That, I guess, is probably were real intelligence lies, we're only partly there right now.
So what we know take for intelligence is barely such. We react to the world around us, but often at such a guttural level that it should never be confused with intelligent behavior. We still rely heavily on our older emotional capabilities, our intuition and of course superstition as aides in making many of our daily choices. Hardly rational, and barely intelligent.
Computer Science sits on the cusp of this. It is the one discipline that confronts this messy disarray of our knowledge and behavior on a daily basis. We guess what data we want, guess its shape, and then hope it works. The rampant changes in specifications come from the poorness of our guesses. If we were right initially, it wouldn't keep changing.
And our most venerable solution is by going around the problem with artificial intelligence. If we only had that, some people think, then our problems would be solved.
The biggest problem with our actually finding artificial intelligence is that like the Vulcan, Spock from Star Trek, it will simply annoy us, because it's rationale for doing things will be well beyond our own understanding. It will act in an intelligent manner, always.
It is very likely that we are in fact only half-intelligent. Simply a good step down a very long road to hopefully becoming a fully intelligent creature some day. Any good read of a daily newspaper will easily fuel that suspicion.
So, in all likelihood, even if artificial intelligence is discovered, it is likely something we don't want in our lives at this stage in our evolution. Some day perhaps, but not now. It just isn't going to help, and if Hollywood is to be believed, it will make things a lot worse.
If we can't have it, then our only option is to make all of our tools simple and deterministic. The tools have to work, no matter what environment that are placed in. We can't just keep adding rampant complexity and hoping that some magical solution will come along and fix all of the issues.
THE LITTLEST OF DETERMINISM
While I've gone on about the effects of failures, even at the smallest level our tools need to be deterministic as well. Certainly there was enough initial theory for GUIs about making everything on the screen be a direct result of the users actions. What happened to great ideas like avoiding modal behavior?
Good practice that we have been losing over time. Windows move on their own, focus shifts arbitrarily and dialogs pop up unexpectedly. It's an annoying type of sloppiness that degenerates the usefulness of tool.
We shouldn't have to stare at the machine to be able to ascertain its state, our actions alone should accomplish that. In such, a blind person should be in control of a simple program without ever having to look at the screen. The results on the screen should be absolutely deterministically caused by the user's actions. Strange erratic actions are non-deterministic.
Some early operating systems like Oberon did this exceedingly well, but as is often the case in software, with each new generating ignoring the knowledge of the past, much is lost while little is gained.
In general the idea of building non-deterministic tools is easily proven crazy. We clearly wouldn't be happy if the controls on a bulldozer or car, "sort of" worked. Forward mostly went forward, stop kinda stopped, etc.
And how useful would a calculator be if it randomly added or subtracted a few numbers from the result. If the results of 34 + 23 could vary by a couple of positions?
Tools are there to extend what we are doing, and they work when that extension is simple enough and predictable enough for us to have confidence in the results. When it has all become convoluted beyond a simple degree, we may have a tool, but using it is uncomfortable. A stupid semi-automated bulldozer is not an intelligent idea, it is an accident waiting to happen.
AND BACK AROUND AGAIN
Errors, networks, machines, intelligence, features, etc. all are tied together by the necessity of having that deterministic property for a useful tool. We must be able to predict the tools behavior.
We want our intellectual tools to work in the same way as our physical ones do. They should leverage our abilities in a simple and deterministic way, so that we can accomplish so much more. Tools should leverage our efforts.
Our modern computers are increasingly failing on this front. The number of failures is increasing rapidly, as we add new "features" we keep kicking it up to the next higher levels. At some point, the instability exceeds our abilities to construct it and the usefulness of the machine plummets significantly with each new increase in failures.
Our interfaces too have been increasingly failing. We have forgotten those simple attributes that should anchor our designs, things like determinism. We might have big fancy displays, spewing lots of colorful graphics, but if we can't trust what we are seeing, the enhanced presentation is all meaningless.
Artificial intelligence may some day grace us with its presence, but if anything it will complicate matters. We'll still need tools and they'll still need to be deterministic. It's usage will (and should) be limited and tightly controlled, possibly it is a similar dilemma to athletes taking steriods, something that clearly has an effect, but for obvious reasons is entirely non-desirable.
Determinism is a hugely important property of computers, that we've been letting slip away from us in our haste to make prettier systems. Where we think that tight bindings of the systems are making them easier to use, the truth is exactly the opposite. The more instabilities in the machine the more we stop trusting it. It is said that a poor workman blames his tools, but I'd guess that a foolish one uses undependable tools and should probably be blaming them.
Given that we have the foundations in Computer Science to understand why we should be building deterministic systems, our increasing failure to do so is all the more disconcerting. We want simple, but not at the cost of stability, a point where we have already sacrificed way too much.
If you take an abstract view, any given computer has exactly F ways to fail. By failure, I don't mean small annoying bugs, but rather total and complete melt down of the platform to the point where it requires a significant fix: reboot, new hardware, or new software. It is unusable without intervention.
F varies, based on the hardware, operating system and any installed software. A couple of decades ago, with a simple IBM box, DOS 6 and a normal upscale vendor for peripherals, there might have been hundreds of ways for the machine to fail. Any of the core hardware could burn out, software could go wild, or the OS could find a reason to crash. The actual number doesn't matter, we far more interested in the way its changing, and why its changing.
In a sense, on using the machine to complete a task there was one way to success, and F ways to fail. The successful branch is overwhelmingly likely, but the size of F is not insignificant.
DETERMINISM AND ITS NON
There is a dictionary definition for the word determinism that describes it as a philosophical doctrine. A belief in cause and effect. In Computer Science however, we tend to use the term more as an objective property of a system, such as a deterministic finite automata (DFA). The transitions between states in the automata are driven deterministically by the input. There is an observable causal relationship. If the system does exactly what you would expect it to do, no more, and no less, then it is deterministic.
Computers, by their very nature are entirely deterministic. The CPU starts executing a fix series of instructions, such that given the same initial pre-conditions the results will always be the same. That deterministic quality, as we'll get into more detail, is very important for making the computers a useful tool.
Interestingly enough, although their behavior is predicable, computers can be used to simulate non-deterministic behavior. In at least a limited sense, the regular expression operator * eats up an arbitrary number of characters in a string until it is finished or the expression is proven to not match. a*b behaves differently depending on the input string, matching a string like annnnb, but not one like baaaaaa.
A common way of implementing this type of functionality in software is by using a non-deterministic finite automata (NDFA), which is a rather long way of describing an abstract machine with a set of internal states where the transition from one state to another is caused by a non-deterministic reaction to the input. You just don't know when the machine will change state.
You'd think that writing something on a deterministic computer to have non-deterministic behavior would be a complex problem, but they solved that rather early in our history by spawning off heaps of DFAs for each possible state that could exist. These possible transition automata may or may not exist, one input of which may collapse all of them into a single defined state. The NDFA may be any number of possible DFAs, but similar to Quantum probabilities, they all collaps down to one (or none) at the very end.
With this knowledge, it became easy to simulate a NDFA using a dynamic list of DFAs. Non-determinism, it seems, can be easily simulated. Or at least aspects of it.
A neat trick, but it also holds a broader understanding: non-deterministic behavior is easily emulated by determinism machines. Just because the essence of something is predictable, doesn't mean that everything you do with it will be predicable as well.
INCREASING WOES
Over the years, as the software has gotten more complex, the number of possible failures, F, has risen significantly. I believe it is rising exponentially, but proving that is beyond my horizon.
These days, there are many more reasons why modern desktop computers fail. Hardware, while getting cheaper is also less in quality. Software, while more abundant has outpaced it environment for complexity. Software practices have decayed over the years. Although modern operating systems are far more protective of themselves, there are still a lot of ways left to render a machine useless. New holes open up, faster than old ones are patched. And now of course, there are people actively trying to subvert the system for profit.
If it was just an issue of an exponentially increasing F exceeding our operational thresholds for stability, technological advancements would gradually reduce the issue. We'll eventually find stronger abstractions that help with reducing these problems.
However, indirectly, we've created an even worse problem for ourselves. One that is growing more rapidly than just the normal equipment or algorithmic failures.
As a stand-alone machine, we define the number of problems in a computer as F. If we link two machines by a network, this really doesn't change. The addition of networking hardware and software increases F, but linearly, relative to the new pieces. Failures on one machine, in very limited cases may cascade to another, but that is rare and unlikely. Each machines reacts to its data independently.
If one machine becomes responsible for pushing code and data to another machine, then everything changes. The two machines are now intertwined to some degree. F might not double, but there are all sorts of reasons why the second machine may easily clobber the first. As we see all of the different chips and micro-controllers of any standard PC as a single machine, we must now see all of them for both machines as one machine. Bind them correctly, and they are no longer independent entities. The two become one.
Once the interaction goes beyond a simple question answer protocol, where the exchanged data itself can contains failures, both devices are tied to each other.
With two machines and a rather large increase in F, the behavior of the whole new machine becomes rather less deterministic. Like the NDFAs, you can see the whole as the different permutations of its kids. A big machine containing two independent simulations of other machines. We can still follow the behavior of the boxes, but it is no longer as predictable. Small differences in execution order on either of the boxes may change the results dramatically. Subtle differences that mean non-determinism.
THE NETWORK IS A COMPUTER
Almost all of our modern machines are now interconnected to each other in some way. Internal corporate intranets connect to the Internet, which connect to our home networks. We have constant, easy access to our resources from a huge number of locations.
These increases in networking have revolutionized our machines, giving us unprecedented access to a near infinite amount of data (at least relative to one's lifetime). They helped integrate these tools into every aspect of our daily existence.
Software companies have utilized this connectedness to make it easier than ever to keep our current software up-to-date. There is a mass of interaction, at both the operating system level and the application levels. There are tools great and small for automatically patching, updating, reinstalling and coordinating the software on our machines. Windows XP, Java, Firefox, Acrobat Reader and dozen of other packages check on a frequent basis for the latest and greatest updates.
Even more interesting, if you account for all the virus, spam and phishing, then there is a massive number of sources out there interacting with your machine, trying (if not always successful) to give it new data, but also new code.
In essence, we've tied together our machines on so many different levels that we have effectively taken the millions and millions of previously independent computers and created one big massive giant one. One big massive giant, vulnerable one. With an F that is shooting well out of control.
BAD HAIR DAY
In its simplest form, all it takes to trigger a failure is for one of these updating mechanisms to install a change that renders the machine useless. Although there are checks and balances to prevent such things, history is full of great examples of nasty bugs getting lose in the last final round. It is inevitable.
So now we're becoming increasingly dependent on some poor coder in Redmond not having a bad hair day and accidentally scorching our boxes in a serious flame-out.
We're also dependent on some poor coder in the outskirts of civilization hopefully not having a good hair day and finding some new great way to subvert our machines for evil purposes.
If we started by thinking that F was in the hundreds, once you start counting all of the little possibilities, and how they might intertwine with each other, F grows at a frightening rate. There is a staggering number of reasons why an average desktop computer can now fail. Millions?
Given my experiences both at work and at home, these are not just infinitesimally small probabilities either. Over the last few years I've seen more hardware and software failures for an ever changing set of reasons, then I've seen in my whole career. Yet my usage and purpose and intent with my machines hasn't changed all that significantly. In truth I do use more web applications, and spend more time on my home machine then I used to, but the increases in failures has been way over my increases in usage.
My machine at work fails every time there is an system update to a piece of software I de-installed years ago. The box locks up, hangs, and turns off my database. This occurs multiple times per year. Several of the other auto-updates have crippled my box on occasion as well.
My friend's machine rebooted on a usb key insertion. My wifes machine has seized up for a number of unknown reasons.
If it were just me, I could forget about it, but virtually everybody I know has been entangled with at least one good computer bug story in the last couple of years. I'd start listing them out, but I don't think it necessary to prove the point, if you use your machine significantly, you're probably already aware of this.
In truth, it has been this way for a while.
Way back, I remember a system administrator telling me that Windows NT works fine, so long as you don't touch it. Sadly, he was serious, and defending the platform, which I had been verbally abusing because it keep crashing while running some test software. Nothing about the software should have effected the box, yet it was. Desktop computers have long since had a history of being flakier than any of their earlier cousins.
PHYSICAL DETERMINISM
A world-spanning massive super machine is both an interesting, but also a scary idea. Our little desktop machine is just a tiny piece of this bigger cube, entirely subject to its will, not ours. We only get to use it to complete our tasks if we are lucky. The F for this machine is way too high.
Purchasing a time-share on a massive computer is an interesting prospect, but would the circumstances be helped by achieving a massive bump in quality? If there was better security and better testing, would this change things?
These are very tough questions to answer simply. The short answer, often demanded by a younger, and less patient generation is that tools need to be deterministic, and that our current methods of interconnecting our machines defeats this entirely. In its own philosophy, we could say that the bugs are just the effects, the underlying cause is the lack of determinism.
To really understand that, we must start with a very simple observation. A bulldozer is a tool used by construction workers to move large volumes of earth from one place to another, and to flatten big areas. Bulldozers act as a tool to leverage the power of the machine to accommodate the drivers actions. The machine extends the driver's abilities.
It is entirely possible to build some type of Rube Goldberg contraption that given a setting, puts the bulldozer through a precise set of instructions. Physical, like the machines that help manufacturing lines assemble complex objects, completely non-computerized, just physics. We could build it so that the bulldozer is dropped off in a location, turned on and then it would do a specific operation like clear an area of land, or move a big pile of dirt somewhere. Set the switch to 40x40 and you get a huge square of land precisely flattened.
This type of mechanical automation could be used to remove the necessity of the driver, whom after all is just sitting there operating the machinery. The problem is that, in operating by itself, even with a very fixed simple set of rules, there are always unexpected circumstances that will creep up. In order to prevent serious accidents, someone must monitor the progress of the machine, and it is far better for them to be working with it while this happens, then for them to be sitting on the sidelines in a chair.
You can automate a factory because it is a limited controlled environment, but you'd never be able to automate something for an un-controlled one, common sense and safety keep us from doing so. Even the tiniest of accidents would draw a huge storm of protest.
The tool is best if it deterministically caries out the instructions of the driver, extending their abilities to do manual jobs. For both the factory and the bulldozer, determinism is a crucial aspect.
INTELLECTUAL PURSUITS
While that makes sense in the physical realm, people often differentiate between physical effort and mental effort. It is one of those class hold-overs where physical is seen as less desirable. In reality, most physical jobs demand a huge degree of mental energy, and sometimes this shifts over time. Los Alamos relied on human calculators in the pre-computer days, to do all of the "manual" calculation work, now easily seen as intellectual effort handled easily by a simple calculator. An Olympic class athlete's brain is working overtime trying to control their reactions to a massively precise level, a huge feat in thinking.
Effort of all kinds is really just a mix of some percentage between physical and mental. The two are closer than most people care to admit, some analogies relate the brain to being a muscle for thinking. Just another way to expend effort.
As anybody who has ever tried to automate a complex intellectual task knows, if you can't contain it there are as many possible obstacles to getting it to work correct as their are for the bulldozer. We cannot predict all of the variability of the real world, be it a mountain of dirt in a field, or a mountain of information in a corporation. It might, in some way be possible to account for all things, but realistically we need to assume that the possible failure conditions are infinite. Some are tiny, but it's still infinite.
REACTIVE INTELLIGENCE
In a world of unpredictability, we'd need a extremely complex calculation engine to be able to cope with an infinite variety of errors. Artificial intelligence is the much heralded savior for our woes. The only problem is that some people strongly believe that it is not possible, and have constructed proofs for such a truth. Many people believe that the failure in research to achieve it already is proof enough.
It is an interesting debate, and I'll digress a little into it. I have a tendency to think it is possible, but I think that most people studying it have grossly underestimated its inherent complexity. In a sense, they are looking for a simplified, pure, abstraction that provides this heighten capabilities. An abstraction that fits the neat ordering of our thinking.
I tend to see thinking as being intrinsically messy, and in many cases such as creativity, quite possibly flawed. I've often suggested that creative sparks are failures to keep things properly separated in our minds. A flaw, but a useful one.
A good simple example is the Turing test, a simple way of determining if something is "intelligent" or not. A person blindly interacts with a couple of entities with the intent of finding out which of them is a machine, and which is a human. If the person cannot distinguish between the two, then the computer's behavior has been deemed intelligent.
The problem with this test comes from the original episodes of Star Trek. Given a Vulcan -- an idealized, overly logical race of beings -- as the machine entity in a test against a human, most people would assume the Vulcan to be a computer. The logic and lack of emotion would be the keys. A Turning test for a human and a Vulcan would fail, showing the Vulcan as not being intelligent.
However, our idealized alien, at least in a TV-show script sort of way, is in fact an intelligent being quite capable of building star ships and traveling through the universe, a massively complex and clearly intelligent feat. Although it is only a TV show, it does act as a well constructed thought-based test. There is at least one alien intelligence we can conceive of that would intrinsically fail the Turing test. It would appear as a computer without intelligence, when it clearly wasn't.
I see this as hugely important in that the researchers are out there looking for something idealized. Something pretty, something with structure. They are searching in a specific location in a rather large field, my hunch is that their prey is hiding at rather an opposite location. And, as always, we can't search there, until our current path(s) take us closer to that area. I'd guess that we have to understand the structure of knowledge first, before we can learn to reliably extract out specific relationships.
It's likely to turn out that our own hubris-based definition of intelligence may actually be a big part of the problem.
GOING TO THE DOGS
What we consider as intelligence isn't often so. There are lots of examples, but fear and politics always provides the most interesting ones.
Over time, various media have discovered that shocking scary stories help sell the news. Because people like simple stories, there has been a tendency to report dog attacks, but only if the animal is described fully or partially as a Pit Bull. The term has come to invoke fear, and the common stereotype is of a vicious animal most often owned by drug dealers or other nefarious folk.
It's always a good simple story that gets a reaction. Often the dogs in question aren't even remotely Pit Bulls. The largest size of the breed is around 65-85 pounds, but its not uncommon to see stories about 120 pound "Pit Bulls".
It also doesn't matter that all other dogs bite, there are millions of Pit Bulls, nor that most of the supposed Pit Bulls are actually just mixes of many other breeds. Somehow this one domesticated animal has been granted superior dog capabilities. We've been living closely with the "child's nanny" for over a hundred years, they are the mascot of WWI, the RCA icon, appeared with children in films and have thousands of other cultural references. However, people wanted a villain, and the Pit Bulls were appointed.
Fatalities is where the anti-dog fanatics really like to focus, but they are slow to compare the dog-based numbers to those of other animals such as horses or even cows. Cars and guns kill millions every year, and somehow a tiny number of deaths becomes a growing epidemic. Yes, interaction with animals sometimes ends badly, that has always been true, no matter what animal is under consideration, domestic or wild. Anybody who has ever owned pets of any kind easily understands this.
Hype, misinformation, etc. aren't new, but we live in an age were it is becoming harder and harder to get away with this, and were ironically we are getting more and more of it. So, as a side effect of selling newspapers and nightly news casts, one of North America's most distinctive dog breeds has been used as the scape-goat for all of our angst about nature. Ironic, particularly in a period of intense "green" frenzy. And what more would we expect from this?
Of course some regional government, desperate to look "proactive" seizes the day and passes a breed-specific law banning Pit Bulls from an entire province. Sad, given that the negative hype was initially profit driven. The underlying facts aren't there, or are irrelevant. "Surely anybody who reads the newspaper would know that Pit Bull bites have reached an epidemic status and something needs to be done!"
The twist in all of this, is that we use our intelligence to build up and maintain our collective set of social rules. Because we live in an "intelligent" society we make rules and laws based on our understanding of the world around us. Smart people supposedly come together to lead us.
Ironically, the law didn't even really pass correctly. The dogs they were trying to address were mixes of Pit Bulls, but the law on docket is now restricted to only pure Pit Bulls, dogs with papers. It's all but useless, other than having taken away the newspapers ability to print good "Pit Bull" stories, and keep breeders from selling pups. It's a sad example of irrational fear gone completely wrong.
A new set of laws exist that don't do what they were supposed to, and the only reason they were created was to keep the public believing that a political party was actively trying to solve a problem (even if it was a trivial, non-existent one). It's insane, and clearly not intelligent. Even if you buy into the anti- "Pit Bull" hype, the resulting law itself is completely in-effective, a failure on top of a failure. The pure Pit Bulls have been all been replaced by mixed breeds. Nothings changed. All we have is another useless set of laws drafted by supposedly intelligent beings.
EXCEEDING DILBERTIZATION
This is but one simple example of how we have been building up our rules, our political systems, our organizations and our knowledge over the decades. We're exponentially piling on Dilbert inspired thinking to poorly solved unrelated problems. We have absolutely no way of disproving bad ideas or compressing our increasing mass of mediocre ones. Any new legislation from any existing body is equally as likely to be bad as it is good. And oddly, it is not just subjective taste, there are massive examples of clearly stupid short- or long-term ideas getting passed as though they were somehow intelligent. We live in a depressing era. Good for cartoonist, bad for intelligence.
Given the arbitrary mess of our collective knowledge, its easy to see how we appear as kids just barely beginning to qualitatively think about our surroundings. There was a time without zeros and negative numbers, pretty hard to conceive of now, but it wasn't that long ago. There will come a time when we can show an irrational bit of thought for what it really is. That, I guess, is probably were real intelligence lies, we're only partly there right now.
So what we know take for intelligence is barely such. We react to the world around us, but often at such a guttural level that it should never be confused with intelligent behavior. We still rely heavily on our older emotional capabilities, our intuition and of course superstition as aides in making many of our daily choices. Hardly rational, and barely intelligent.
Computer Science sits on the cusp of this. It is the one discipline that confronts this messy disarray of our knowledge and behavior on a daily basis. We guess what data we want, guess its shape, and then hope it works. The rampant changes in specifications come from the poorness of our guesses. If we were right initially, it wouldn't keep changing.
And our most venerable solution is by going around the problem with artificial intelligence. If we only had that, some people think, then our problems would be solved.
The biggest problem with our actually finding artificial intelligence is that like the Vulcan, Spock from Star Trek, it will simply annoy us, because it's rationale for doing things will be well beyond our own understanding. It will act in an intelligent manner, always.
It is very likely that we are in fact only half-intelligent. Simply a good step down a very long road to hopefully becoming a fully intelligent creature some day. Any good read of a daily newspaper will easily fuel that suspicion.
So, in all likelihood, even if artificial intelligence is discovered, it is likely something we don't want in our lives at this stage in our evolution. Some day perhaps, but not now. It just isn't going to help, and if Hollywood is to be believed, it will make things a lot worse.
If we can't have it, then our only option is to make all of our tools simple and deterministic. The tools have to work, no matter what environment that are placed in. We can't just keep adding rampant complexity and hoping that some magical solution will come along and fix all of the issues.
THE LITTLEST OF DETERMINISM
While I've gone on about the effects of failures, even at the smallest level our tools need to be deterministic as well. Certainly there was enough initial theory for GUIs about making everything on the screen be a direct result of the users actions. What happened to great ideas like avoiding modal behavior?
Good practice that we have been losing over time. Windows move on their own, focus shifts arbitrarily and dialogs pop up unexpectedly. It's an annoying type of sloppiness that degenerates the usefulness of tool.
We shouldn't have to stare at the machine to be able to ascertain its state, our actions alone should accomplish that. In such, a blind person should be in control of a simple program without ever having to look at the screen. The results on the screen should be absolutely deterministically caused by the user's actions. Strange erratic actions are non-deterministic.
Some early operating systems like Oberon did this exceedingly well, but as is often the case in software, with each new generating ignoring the knowledge of the past, much is lost while little is gained.
In general the idea of building non-deterministic tools is easily proven crazy. We clearly wouldn't be happy if the controls on a bulldozer or car, "sort of" worked. Forward mostly went forward, stop kinda stopped, etc.
And how useful would a calculator be if it randomly added or subtracted a few numbers from the result. If the results of 34 + 23 could vary by a couple of positions?
Tools are there to extend what we are doing, and they work when that extension is simple enough and predictable enough for us to have confidence in the results. When it has all become convoluted beyond a simple degree, we may have a tool, but using it is uncomfortable. A stupid semi-automated bulldozer is not an intelligent idea, it is an accident waiting to happen.
AND BACK AROUND AGAIN
Errors, networks, machines, intelligence, features, etc. all are tied together by the necessity of having that deterministic property for a useful tool. We must be able to predict the tools behavior.
We want our intellectual tools to work in the same way as our physical ones do. They should leverage our abilities in a simple and deterministic way, so that we can accomplish so much more. Tools should leverage our efforts.
Our modern computers are increasingly failing on this front. The number of failures is increasing rapidly, as we add new "features" we keep kicking it up to the next higher levels. At some point, the instability exceeds our abilities to construct it and the usefulness of the machine plummets significantly with each new increase in failures.
Our interfaces too have been increasingly failing. We have forgotten those simple attributes that should anchor our designs, things like determinism. We might have big fancy displays, spewing lots of colorful graphics, but if we can't trust what we are seeing, the enhanced presentation is all meaningless.
Artificial intelligence may some day grace us with its presence, but if anything it will complicate matters. We'll still need tools and they'll still need to be deterministic. It's usage will (and should) be limited and tightly controlled, possibly it is a similar dilemma to athletes taking steriods, something that clearly has an effect, but for obvious reasons is entirely non-desirable.
Determinism is a hugely important property of computers, that we've been letting slip away from us in our haste to make prettier systems. Where we think that tight bindings of the systems are making them easier to use, the truth is exactly the opposite. The more instabilities in the machine the more we stop trusting it. It is said that a poor workman blames his tools, but I'd guess that a foolish one uses undependable tools and should probably be blaming them.
Given that we have the foundations in Computer Science to understand why we should be building deterministic systems, our increasing failure to do so is all the more disconcerting. We want simple, but not at the cost of stability, a point where we have already sacrificed way too much.
Saturday, September 13, 2008
7 Fabulous Ways to Great Programming
This post is for all of you coding surfers that have ever anonymously filled in "TL;DR".
So you wanna be a great programmer, do ya? All you have to do is follow these seven easy bullet points:
- Stop reading bullet points!
- You heard me, stop reading these stupid bullet points!
- They're not helping, you know.
- They are often just fluffy platitudes.
- Still reading? I thought you'd get wise by now?
- It just shows how useless bullet points actually are ...
"Get on with it", your inner voice is screaming, adding in "just give me the highlites, the bullet points, the summary, dude. I don't need the rest of your stupid rant."
The problem -- in bullet points -- is:
- bullet points only convey "summary" information.
- bullet points are forgettable.
- bullet points are junk food, a kinda McIdea that neither teaches nor satisfies.
If you want the knowledge then you have to get the knowledge, otherwise you know nothing.
Ways to do that:
- Spend hundreds of years writing every possible type of program.
- Apprentice with an experienced programmer.
- Read, read, read and read.
- Take courses, then read.
- More reading.
Well, too bad. So sad, sorry and all of that. You foolishly picked a rather incomplete occupation; programming and software development are barely out of their diapers. We just haven't had centuries to distill the knowledge out of the experience, yet.
Some day, perhaps, but until then we're all struggling with trying to find a voice to share our experiences. Sometimes that provides good solid reading, but sometimes it only comes off as a rant or a ramble, with hidden buried gems piled deep in the subtext.
But, and here's the BIG point:
- There is real knowledge buried in the subtext.
- You can't learn that knowledge unless you read the full text.
- It can't be summarized.
- It's not even necessary the main point.
- Skimming the text misses its real value!
A little knowledge is a dangerous thing. Always has been, always will be. If you sort of know how to drive a car, and you mostly follow some of those "rules" about silly things like lanes and stop signs and such, you're not going to last long. You're just a roving accident waiting to happen.
And oh, if that knowledge, has been even further diluted into a list of platitudes, ack! Seven reasons for "anything" is probably useless to you. It is probably useless to most people, unless it is just acting as a reminder for known ideas. Really. For starters:
- Platitudes say nothing, but sound good.
- They are easily forgotten.
- They fill you up on junk knowledge, when you should have been learning.
"Still, people could distill their crap into 3 easily readable paragraphs, dude." you're insisting.
Possibly, but most bloggers are amateurs, we barely have time in our lives to write this stuff. Half the time we don't even really know what it all means. Not, as you might guess, the explicit text of what is being said, but what it really means in the bigger sense, the holistic view. There is so much buried, hidden between the lines, just waiting to get a voice. Particularly in a field like Computer Science were it is still not fully understood, huge amounts of important knowledge get buried in people's direct experiences. It is hard to fully express that understanding.
In a strange sense, we can only communicate what we explicitly know, but often times the topics are implicit. Sometimes when I am writing, for example, I'll dig at something deeper, but other than just bouncing around it, I don't really have the vocabulary, yet, to express what I am trying to say.
Some of my later works, are just continuations or follow-ups of my earlier writings, each time they are getting a little closer to the real underlying truth.
Pretty much if you read most of the serious essayist bloggers you'll find the same thing. Learning to express something unknown is a truly creative act, a spontaneous one. Of course just repeating well-known platitudes isn't, but then isn't that why they are platitudes in the first place?
Blogging as a medium is a direct way to access a mass amount of early information. It has not been comfortably packaged into neat theories and pretty textbooks. It might be the predecessor of some well structured understanding, but only if you're willing to wait for it to drift into the main stream. In an industry like software development, where so much of what we do is based on intuition and guessing, getting any addition comprehension is a major assistance.
"Fine, so long blogs are often interesting, but badly packaged information; I've got masses of dysfunctional code that I've written that needs urgent fixing. I just don't have time to dig for gems." you moan, adding it "and it has nothing to do with my bugs anyways!".
Initially I said:
- I want to people to be better programmers.
- bullet points suck!
- real knowledge is buried in the subtext.
We could wait for an understanding to trickle down upon us from academia. Some day, there will be cohesive theories and processes that support reliably building complex systems. Someone will eventually discover a better way of coding. But, that process is slow, and at its current rate things are not likely to improve until well after I have finished my career, possibly my lifetime.
Another alternative is to reach out to the industry, and hopefully explore the issues in a way that we all come to learn how to build better code. There are pockets of excellence in programming, but they certainly are not the state of the industry.
The single largest problem with software comes from its inconsistency. It is a dog's breakfast of inoperative ways of handling data, often half finished, and poorly extended. It is a big mess that we are deliberately making bigger.
"Sure, but not my stuff. It just has a few bugs, that's all" you chime.
Here's the rub (as Shakespeare might have said, according to Cole's notes): The single greatest, most important, significant all-powerful encompassing, awesome, extreme, critical, supreme points about programming are:
- Focus!
- Self-discipline.
- Consistency.
If they're out there telling you otherwise, you know they're just blowing smoke. It's simple:
- good programmers write good code.
- good code is neat and simple.
- good code is consistent.
"So what the hell does this cranky nonsense of yours have to do with my code?" you start to ponder.
Here is the easy bit. Very simple. This entry, as the title indicated was about fabulous ways to great programming. The most important of these is the ability to focus for long periods. If you are finding that reading large blog entries is far too taxing, then you are having problems focusing. If you can't get through several pages, then how can you expect to get through five years working on the same massive code base? Serious development is a huge amount of work. A month of HTML is fun, but light-weight coding isn't the same as having written something big and serious.
The problems, the real ones, take a long time to sort out and are complex to build. Inherent in this, is a tremendous amount of focus and consistency. To survive, year after year, you need to be self-disciplined. These attributes are all intertwined.
If you're flailing at the keyboard, or waiting for your boss to "make you" refactor that mess you spazzed into the machine last month, then focus or self-discipline could be the problem.
In point of fact, if every long blog article is automatically "TL;DR" by default, this ADD driven approach is bound to spill over into the other parts of your career. If you can't focus for longer than a few minutes, there is no way your code is going to be "great". Just ain't happening. Programming isn't a multi-tasking opportunity; you sit down for long periods at a time, heavily focused on the work. If you cannot do this, your code may be a hell of lot of other things, some of which may end up entertaining people on WTF, but it unlikely to ever be great, and probably not even good.
Consistency is a mandatory property of good code. Focus and self-discipline are the ways to get it there. Bouncing around on the net, reading only short platitudes and bullet points indicates a possible inability to focus. Leaving comments such as "TL;DF" says more about the reader than I think they would care to admit.
The way to better coding is to spend more time trying to learn. Experience is a way to refine knowledge, but you need to acquire it first, or else you will just wander around in the dark forever. Beyond the standard texts, which are far from complete, the new and often critical knowledge these days is buried in long blog discussions. An apprenticeship would be better, but experience leaves our industry fast, and many people misgauge their own abilities.
Certainly any increase in focus and concentration will filter back into your coding practices. If you keep it up, the things you write will be cleaner and better structured, giving you a fighting chance to remain a programmer for longer than just a few years. Good work comes from good habits. Great work comes from really understanding all of the details and nuances of what you are doing. It is several levels beyond just being able to get it to compile.
For all those that skipped most of the above text, just a quick summery:
- bullet points rock!
- great careers are opening up in marketing.
- happiness is a bigger hard-drive.
- the clothes make the man (or woman).
Subscribe to:
Posts (Atom)