Innovation

For a long time now, I’ve been noticing that the pace of innovation in software has been slowing. I started paying attention to the industry in the mid-80s, so that breadth gives me a pretty good perspective on the (cyclic) changes.

There are a couple of really great links that I thought shed insight into these problems. This first one gives a great general idea about knowledge transfer: 

https://www.youtube.com/watch?v=pW-SOdj4Kkk

And this second one takes the perspective of how it is acting from an economical standpoint:

https://issues.org/behind-technological-hype/

I’ve had a strong sense that ‘software projects’ act as fishbowl allowing us to observe complexity problems in a reduced, constrained scale. Our projects usually have all of the core fuzzy issues including personalities, agendas, irrational behavior, politics as well as the underlying technical ones like constraints, strict logic, discrete components, etc. That is, they exist in that intersection between mathematics and the physical world. This volatile mix plays out all across our societies, in different industries, in different ways, but it is somewhat smaller and easier to observe in software because the projects are often tiny. 

In the mid-90s I saw the lure of commercialization pull away a lot of the pure and applied researchers. There are pockets left out there, but they are all underfunded. Decades of little research eventually resulted in stagnation on moving forward. We basically locked ourselves into construction techniques that haven’t moved for at least 20 years, all while the number of programmers has exploded and the amount of available code that we can leverage grew even larger. Lots of people, lots of code, but we still don’t see innovative products coming out very often, and we’ve seen a huge decrease in the quality of the code produced. 

“And some things that should not have been forgotten were lost. History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge.”

― Galadriel in The Lord of the Rings: The Fellowship of the Ring, The Lord of the Rings

If there are any billionaires out there that would like to help reverse this problem, there are a lot of people like myself that are fountains of innovative ideas but have no serious way to explore them. Once life grants you a few dependencies, and takes away some of your energy, working a full-time, high-stress job to pay the bills is cognitively demanding enough that we need our nights and weekends to recover. Personal research projects tend to take hundreds of times longer as side-projects and are prone to disruptions and so they aren’t pursued often and have a super-high rate of not getting finished. Instead of toiling away on another half-finished idea, I’ve tended to just write them up in the blog and move on. It’s less frustrating.

Of the brighter sources of help is MacArthur fellowship grants, but they are pretty much just focused on academics in the USA. The idea is great though, that someone can grant you a five year, no strings attached, opportunity to pursue your ideas. I think if this type of financing was more easily available, we’d see a lot more people trying out innovative software ideas, and eventually, some of those ideas would result in huge improvements in our industry. 

The reason we don’t see this type of innovation in private companies is that most of the focus, as expected, is on monetization. The biggest and most pressing problems in software development don’t have direct ways of making money. They layout a base platform on which the upper layers may be monetizable, but much like the Internet, if they get bent too early in just making money, that hurts both their construction and usage. We need cooperative platforms first before we can rest competitive games on top of them. Skipping step one tends to craft speculative games, not solid ecosystems. 

If we wanted to lift the software development with innovation, we really have to go backward and explore a lot of vague ideas that are attached to very deep root causes. We suspect that there are better ways of framing our construction processes that would result in faster, more stable systems. We know that as we become more dependent on software, we can’t continue to live with our low-quality construction methods. If we rely on some code, that code has to work within a very tight tolerance, and we currently can’t afford the massive amount of time we know is needed to achieve that type of precision. There are so many fundamental issues that we’ve ignored over decades in our mad rush to just put half-baked products into the market. At some point, this has to change if we want to move forward.

Controlling Development Chaos

A lot of software developers really hate any comparison or analogy that involves building construction. I think the primary reason behind this dislike is the sense that construction workers are viewed as passive players in the whole development project, in the same way that factory workers are seen as nearly-mindless cogs in a production line.

There is some reason to fear this type of comparison, intellectual pursuits are clearly not the same as physical. The rules are different. For whatever reason, we have a great deal of knowledge on how to refine and maximize the efforts of factory workers, but virtually none on how to do the same for white-collar jobs. Trying to take what we've learned from the physical and apply it to intellectual hasn't been particularly effective.

Still, in this post, I start with looking back at a construction analogy and then moving forward to a more sophisticated, and probably more popular analogy. Even if construction is not a perfect fit, building things is always a lot like building things, and we can always learn a lot from the way other people build things.


THE UNLOVED CONSTRUCTION ANALOGY

While I agree that we shouldn't draw two many parallels between software and construction, there are still things about a construction analogy that are 'truish' enough to be useful.

For instance, a large sky-scraper project contains a huge number of specialists, including carpenters, plumbers, electricians and of course general-labourers. At the same time as well are engineers. managers, inspectors and a whole host of other people with interesting roles in keeping the process, and the building from collapsing. Software projects are similar, in that they often need graphic designers, editors, GUI expects, database programmers, system administrators, managers, tool-smiths and all manner of specialists for handling any sophisticated technologies or domain issues. They are multi-disciplinary projects.

Both software development and construction involve building things and both, to some degree or another, change radically depending on the size of the thing you are building. A shed out back is not the same as a house, or an apartment building. A sky-scraper is a completely different deal. Scale has a huge influence on the size and techniques required to bring it all together. One does not easily jump from building sheds in a backyard to building apartment buildings, it is a different set of skills.

The construction of a modern sky-scraper is an amazing project that is pulled together with a degree of smoothness that software developers can only drool at. If we could build as large and sophisticated things, with the same degree of precision and timing, then we would have been able to get beyond our current hit-or-miss random guessing style of development process.

It is worth mentioning too, that while design and code are fixed to some degree, the job of an electrician on a huge site still requires a significant amount of thinking and problem solving. It is not a clear-cut mindless day at the office. While they may be working with their hands, electricians still need to think about what they are doing, the various plans and codes, and how they are going to make it work in the current context. The same is true for all of the other specialists. It takes more mental effort than just being a body, that after all is the specific job assigned to general labourers.


WHY IT FAILS

But still, even if you give all of that its due, construction just doesn't map entirely onto software development. We know this, it is a frequent discussion. Sometimes, I wonder if it really is a case of us just not wanting to accept it; we have a great deal of freedom, and it becomes hard to surrender that, even if it is for the good of the project. Still, there are strong differences.

The biggest single problem with construction is that it is a one-phase deal. You build the building, and then move in and repair it as you go. When we tried this with our waterfall ideas, we found it doesn't match reality. Code is seems, never gets done and is always rusting. Projects just don't end, and when they do, the code goes out of circulation. For software, there are a nearly infinite number of phases, ending only with the lifetime of the code.

The state of the art of building design has progressed over centuries, and mostly it is only slightly tweaked between buildings. In a sense, construction companies redo the same projects over and over again, they have no choice. This repetitiveness is the root of mastering the process. If you do it enough, eventually you'll get good at it. Contrast that with software, where each design for each system is unique and rarely learns from its predecessors. Because we can clone code, we don't want to keep reinventing it (even where in some cases, that might actually produce significantly cheaper or better code). Code builds up, but our experiences do not.

Buildings stay around for a long time. That aspect, as well as safety considerations bends the choices made towards the less risky long-term options. Saving time or money by not following the process, the code or using inferior parts may help in the short run, but because the life span is so long it opens up a lot of risk of getting caught in the future. Most buildings are built correctly. Software, on the other hand, being mostly invisible and uncontrolled doesn't provide much incentive for programmers to make long-term choices, despite the fact that software projects are always long-term projects.

Another significant different is that workers on a construction site have less degrees of freedom then computer programmers. Their roles and jobs are far more rigid. Right or wrong I don't know, but this is a key issue that I want to get back to later. We've always had our freedoms, and they have always been a problem.

For all of the differences, there is also a lot that matches, but not enough that we should try to emulate construction companies, although we should admire their skill and organization. But it is exactly that line of thinking that sent me in search of some other type of better suited analogy.


GOING BEYOND CONSTRUCTION

As often happens with me, I get a little down the road of a thought, and then it gets left behind for a while. In particular, I dropped my pondering of construction analogies so that I could do my usual Friday night routine, which was watching a film with friends. The film -- which I can't remember -- had finished, and we had moved on to the DVD special features.

As the director and actors mutually patted each other on the backs and proclaimed their love of working together, a little thought was brewing in my head. A film, you see is a large complex project, that also brings a large number of multi-disciplinary professions together in order to create something. Most people don't quite see it as the same thing, but if you look at the effort and money that was poured in a mega production like Lord of the Rings, you start to see some similarity to these huge buildings we keep throwing up in our cities. But, it is hugely different.

We deem a film as art, and the director, as an artist. We see most people involved with film as artisans. Even thought, in the end, there are also carpenters, electricians, plumbers, casting agents, cooks, boom operators, grips, computer programmers, and a huge huge host of other professionals involved.

What's interesting with a film, is that even thought the script is often written by one or more writers, and a huge number of producers and managers are involved, the films themselves have always held the particular 'stamp' of the director. In a very real sense, even with all of the 'creatives' vying their wares, the director of a modern major feature film, gets and sets the final 'vision' under which the film will be created. You can tell a lot more about a film by its director, then you can by its writers or actors. In many ways, the director of a film, is far more significant than the architect of a building. An architect shapes the design, but the engineers ensure it is built correctly. A film director often has no such constraints, at least not ones that are 'that' objective (budget excluded). They control the vision.


ANOTHER ANALOGY

But if we go back to the special features, even with all of the influence, the staring actors still often talk about how much artistic freedom they were allowed in playing their parts and contributing to the film. In that way, a good director stays true to their vision, but not necessary at the expense of making all the actors just mindless pawns on a film assembly line. Most films are collaborations of many artists, yet remain true to their director's goals.

So if I am looking for a more realistic role-model, I probably want to be more like a director of a modern film when I am leading a big development project. In that way, I want to have my vision implemented, and I want the final product to clearly bear my stamp, yet I want my fellow developers to be able to contribute to the overall project in a way that makes them feel honored and proud to have been on that particular development team. Like the actors, the coders should feel that they were able to make significant contributions, but without disrupting the overall product.

I do at this point, want to warn programmers about thinking that I am saying they should have some infinite degree of freedom while developing software. Not even the directors have total freedom. There is a difference between creative contributions and just plain old chaos. The requirements of the users, the limits of the technology, and the consistency of the conventions all combine to remove a great deal of the degrees of freedom from coding.

For tools to be useful, they must have certain specific functionality that clearly matches a problem domain. You can't creatively work your way around the actual problems, they are what they are. Our existing technological base is awful, but it is the only one we have. If you have to bend your design for a rigid relational database schema in order to allow for reporting, then there is little choice in that matter. And if every other similar application uses widgets in some annoying brain-dead, but extremely common way, you too must follow that or pay the consequences of annoying your users instead of impressing them. A good software product, one that solves a specific problem is heavily constrained long before it is even sketched on the back of a napkin.

In all, if you are implementing a targeted solution for a specific industry, there isn't an infinite array of possibilities for the development. Like a 'genre' in film making, if you break the rules, you had better understand them first, and given a solid reason for breaking them, or people just take you for someone that doesn't get it.

A film like Lord of the Rings, for example, while intensely creative is also highly restricted. Frodo can't be green, and you can't redefine elfs back into funny little forest creatures. The film would flop if it violated the spirit of the books. In the same way, software is constrained by its intended targets.


FINAL CREDITS

In many ways software is unique, but in many ways it is just another method of building things. Our biggest difference is the immaturity which lingers in our development process. We clearly do not want to grow up and start producing better software. But the issue might have been one about what our newer 'maturer' self would really look like. With terms like architect and and processes like waterfall, we've been quietly, but agonizingly following along behind various construction analogies, but with little effect toward reducing the 'software crisis' first identified in the late sixties. Is it any wonder we are resistant to change? Is there actually anything to change to that will really work?

I think if we really don't want to be factory or construction workers, then we need to look carefully at other groups of skill artisans, and start trying to fit their processes into ours. A director for a major motion picture, and his teams of 'creatives' is such a great parallel because it involves a large group of diverse people putting together a big complex work. Its main weakness is that a film is a short-term project, once completed it is over. Because of that, most choices of how to get the work done are not made for the long term. If we did that, it would be a disaster in software. Still, mixing this type of analogy with a construction based one, which is a much longer-term vision, produces something in the middle. We are neither, but we could be both.

I'm probably late with this analogy, as I suspect given the credits on the back of most video games, that at least one segment of the software industry has been quietly following this analogy all along. Still, a unified single vision is the key to bring out high-quality products, and a film director does it better than any other industry. Major motion pictures, at least according to the DVD special features, are fine examples of creative and organizational projects. We should marvel at how they work, and how they manage to bring together so many diverse disciplines into one final amazing product.

Next time someone asks me about what title I want for leading a project, it may be something like: commercial software director. Can an Oscar be that far behind?

Testing for Battleships

In one of my earlier blog entries, I casually suggested that testing was like playing a game of 'Battleship'. 

It is an odd analogy, but as I pondered it a bit more I realized that there were interesting parallels between playing Battleship and testing, many that are worth exploring. 

Often, it is exactly our change of perspective that allows us to see things with fresh eyes; even if it is tired and well-tread ground. 

AN INITIAL DEFINITION 

For those of you not entirely familiar with it, Battleship was a popular kid's game, a long long time ago, in a galaxy far far away. 

Wikipedia has a great description of it: http://en.wikipedia.org/wiki/Battleship_%28game%29 

The version I grew up with was non-electronic and consisted of two plastic cases each containing a couple of grids, the 'ships', and a huge number of little red and white pegs. 

In the first phase of the game, both players take their five 'ships' of varying sizes and place them on the horizontal grid. After that, one after the other, the players take turns calling out 'shots' by the grid coordinates. 

The coordinates are alphabetical on one axis, and numerical on the other. A shot to C5 for instance is the third position across and the fifth one down. 

If the shot hits a ship, the defending player has to say 'it is a hit'. The calling player puts a red peg in the vertical grid to mark the spot. Otherwise, just so they don't call the same spot again, a white peg is used. 

The game continues until one player has sunk all of the other players 'ships'. It is a race between who sinks who's fleet first. 

The big strategy of the game is to find ways to 'hide' the ships. For instance, lining them up against each other, so it is hard to see where one starts and one ends. Sometimes hiding them on the outskirts of the grid is great, sometimes not. Once the game gets going, it becomes a contest between the two players as to who figures out all of the other locations first. 

Mostly, the game ends long before all of the pegs have been entered into the board. From my rather random and fuzzy memory, I'd say usually around two thirds of the board was full at the end. Sometimes more, sometimes less. 

OBSERVATIONS 

What does this have to do with testing? 

Well, interestingly enough, testing is kinda like playing half a game of Battleship against the code. You can see all of the functionality in the system as a giant grid, a vast sea that contains a number of hidden bugs. 

The tester's job is to place tests (pegs) at various points in the grid to see if there is a bug there or not. Each test -- or part of the test -- acts as a little peg that reveals a possible bug. For a test suite, one after another, the testers keep adding pegs until all of the the testing is complete. 

Battleship ends when one player has sunk all of the other player's ships. Testing ends, when the time is up or you've run out of little pegs. It is also a race condition, but usually against time, resources, or patience. 

In the real game, the 'ships' come in multiple sizes that are well known. Bugs on the other hand come individually or in clumps and there is very little rational behind either. In fact, there is a totally unknown number hidden in the grid that can be anywhere from zero to almost NxN bugs. 

Now, most people might think testing occurs with a huge number of pegs, but if you really look at the coverage on a line-by-line, input-by-input basis, the grid is absolutely huge, while the pegs are few. 

Even if you are going to push the whole development team into an intense round of final system-wide super integration testing, it is still a pitifully small number of pegs verses the actual size of the grid. One big different between the game and real testing is that the grid grows with each release. It keeps getting larger all of the time. The game is ever changing. Even when you think you have mastered it, it will still occasionally catch you by surprise. 

Although for the analogy I picture the functionality as an N by N grid, the real topology of any system, depending on representation, is some multi-dimensional arbitrary structure, where the sea-scape itself holds many clues into the increased presence or likelihood of bugs. 

What I really like about this analogy is that it gets across the point that testing is hit-or-miss, and by definition not everything is tested. People tend to view testing as a deterministic process; if you do enough testing, then you will catch all of the bugs. In fact, I've seen 'intensely' tested code, that has run in production for years and years, still yield up a bug or two. 

There are always a huge number of bugs and they are always buried deep. Unless a peg hits them directly with the right input, the presence of the bug is unknown. 

TESTING STRATEGIES 

Over the years I've seen many 'interesting' strategies for testing. 

Sometimes it is all manual, sometimes it is heavily automated. Some testing focuses on catching new problems, so of it is more bias towards regression. 

Generally, though the most common thing about testing is that most of it is done on the easier bits. The hard stuff rarely gets touched, even though it is the most likely source of bugs. 

In that sense, developers quickly use up all their pegs clumped together in a close location on the grid, while ignoring the rest of the board. They tend to put the most pegs into the spots where the least bugs will exist. 

Unlike the game, in testing the board itself offers tonnes of clues on how to test more effectively, but most developers prefer to disassociate themselves from that knowledge. As if, somehow, if we don't get smart about what we test, then the testing will be more thorough, which would be a possibly workable strategy only if you have a huge mountain of pegs. If you have way too few pegs, it's effect is probably the opposite. 

Software developers hate testing, so it is usually handled very badly. Most big integration tests for example are just replications of unit-tests, done that way to save time. An oddly ineffectual use of time, if the tests both refer back to the same position on the grid. 

Underneath, some bugs may have moved around but we still need to test for them in their final position. Sticking the first peg in, then removing it, then adding it again is unnecessary effort. 

Interestingly enough is that for whole sections of the board, if there haven't been any changes to the code, then the same bugs remain hidden in the same positions. So executing the same peg pattern, or regression testing, will produce the exact same results as last time: nothing. 

No changes to the code + no changes to the tests = no new bugs, but also a huge waste of time. 

If you get the sense of this game, then spending a great deal of time sticking the pegs into the boards only to not find any bugs seems as though it may have been a waste of time. Programmer's like to test to 'confirm' the absence of bugs, but really for most code, particularly if it is new -- less than twenty years old -- not finding bugs more likely means the testing is ineffective, not that the code is good. It ain't. So that one last pass through the tests, with the expectation of not finding anything wrong is really a big waste of time. 

You regression test to make sure none of the older bugs have reappeared, but that only makes sense if the code was actually effected. Retesting the same code multiple times, even when it hasn't changed is a standard practice, but still crazy. 

Clearly this also implies that any second round of testing should be based on the result of any changes, and a first round of testing. Parts of the board are not worth retesting, and it is important to know which parts of the board are which to avoid wasting resources. 

PATTERNS 

I can remember when I was a kid, I'd come up with 'patterns' to try and optimize my peg placement for Battleship. You know, like hitting every other spot, or doing a giant X or cross or something. I wanted a secret formula for always winning. These never worked, and I would generally be pushed back to messing up the pattern because of the nature of the game. 

Once you've identified a red peg, you want to finish off the ship quickly. Then you get back to searching. Overall the best strategy was to randomly subdivide as much as possible. Big gaps allow big things to hide. Random shots, distributed more or less evenly across the board tended to be more fruitful. 

One thing for certain about Battleship was that no matter who you were playing against, there was some underlying structure to the way they placed the ships. 

As the game progressed, the strategy for finding the next ships was based on the success of the current strategy. None of the fixed patterns worked because they were too static. Although, no one would actually be able to explain why they placed the pieces the way they did -- to them it was random -- it just wasn't. 

Different opponents tended towards different strategies, which after a game or two became fairly obvious. In a weird, but rather indirect way, the same is true for code. The bugs in the system are not really random, they come from an extremely complex pattern that will always seem random to human beings, but there is some ordered structure. 

The most effective ways to find them are by using instinct to work through the parts of the system that are most likely to have problems, or to insure that their aren't problems where they might be really devastating. On fixing a bug, retesting the entire system is a huge waste of resources. It isn't hard in most systems to narrow the board down to a smaller one that might have been effected by the change in code. 

Finding the true 'impact' of a change may seem expensive, but it is considerably cheaper than retesting everything. In that regard, a well-crafted architecture, one that isn't just a big ball of mud certainly helps in drawing lines around the possible effects of a software change. It makes understanding the impact of a change a lot easier. Obviously, if you can reduce the size of the board, you increase the chances of finding any other hidden ships on the field, particularly if you didn't significantly decrease the number of pegs. 

SUMMARY 

There are lots of things we can learn from studying an analogy. They allow us to morph the problem into some other perspective, and then explore the relationships and consequences in some other way. 

Pushing testing along side a board-game shows the importance of being responsive in the testing process, and not just trying to issue fixed patterns across a huge space. A static approach to testing fails repetitively and is more expensive. A dynamic approach is a more likely way to utilize the available resources. 

Clearly shrinking the board but not heavily reducing the pegs will increase the likelihood of finding problems. Also, not just retesting all of the old spots will help to find new hidden dangers. 

Understanding the topography of the functionality will help in high-light areas that need to be specifically addressed. There are lots of strategies for making better use of the resources. If you see the life of a product as a long series of games against the code, you get an awful large number of chances to test for underlying problems. 

If you squander that on a massive stack of simple-but-useless regression tests that cover the same unchanging territory again and again, it is no wonder that the product keeps getting released with a significantly large number of problems. Time and resources are not making headway into the real underlying territory. If you craft tests as absolute sets of instructions, then recrafting them for new games is hard and a huge amount of work. 

Since we hate testing, we shy away from any work in that area. As such, even projects that do everything 'correctly' and have the 'proper' suites of regression tests and always test 'before' each and every release, let large qualities of bugs through. In fact, the ones that followed the established best practices are probably of the average or worse quality because the programmers tend to shift their minds into neutral during the testing phase. It is the guys with no resources that are anxious about not wasting what little effort they have, that are getting the most bang for their buck.

In Object-Orientation

"Familiarity breeds contempt" is a common cliche. It sums up an overall attitude, but for technology I don't think it is that simple.

I am certainly aware that the more we work with something, the more we come to really understand its weaknesses. To the same degree, we tend to overlook the most familiar flaws just because we don't want to admit their existence. We tend towards self-imposed blindness, right up to the instant before we replace the old with the new. It was perfect last week; this week it is legacy.

Software development has once again come to that state where we are searching for the next great thing. We are at a familiar set of cross-roads, one that will possibly take us to the next level.

In the past, as we have evolved, we have also dumped out "the baby with the bath-water", so to speak. We'd go two steps forward, and then one and three-quarters of a step back. Progress has been slow and difficult. Each new generation of programmers has rebuilt the same underlying foundations, often with only the slightest of improvements, and most of the old familiar problems.

To get around this, I think we need to be more honest about our technologies. Sure, they sometimes work and have great attributes, but ultimately we should not be afraid to explore their dark sides as well. All things have flaws, and technology often has more than most. In that, nothing should really be unassailable. If we close our minds, and pretend that it is perfect, we'll make very little progress, if any.

For this blog entry, I wanted to look at the Object-Oriented family of programming philosophies. They have a long and distinguished history, and have become a significant programming paradigm. Often, possibly because of their history, most younger programmers just assume that they work for everything, and that they are the only valid approach to programming. They clearly excel in some areas, but they are not as well rounded as most people want to believe.


EVOLUTION

Although it is hard to establish, the principles of Object-Oriented (OO) programming appear to be based around Abstract Data Types (ADT). The formalization of ADTs comes in and around the the genesis of the first Object-Oriented programming language Simula, in 1962. In particular, The Art of Computer Programming, by Donald Knuth which explores the then commonly known data structures, was also started in 1962 although the first volume wasn't published until 1968. Certainly, whatever the initial relationship between the two, they are very close, yet they are very different.

Abstract Data Types as a movement is about using data structures as the basis of the code. Around these data structures, the programmer writes a series of primitive, atomic functions that are restricted to just accessing the structure. This is very similar to Object-Orientation, except that in ADTs it is a "philosophy of implementation" -- completely language independent -- while in OO it is a fundamental abstraction buried deep within the programming language.

One can easily see that OO is the extension of the ADT ideas into the syntax and semantics of the programming languages. The main difference being that ADTs are a style of programming, while OO is a family of language abstractions.

Presumably, by embedding the ideas into the language, it makes it more difficult for the programmers to create unstructured spaghetti code. The language pushes the programmer towards writing better structured code, making it more awkward to do the wrong thing. The compiler or interpreter assists the programmer in preventing bad structure.

While ADTs are similar to OO -- you set up a data-type and then build some access functions to it -- at a higher level there is no specific philosophy. The philosophy only covers the low-level data structures, but absolutely nothing is said about the rest of the code. Practice, however is to use structured code to layout each of the high-level algorithms that are used to drive the functionality.

This means there is a huge difference between ADTs and OO. Outside of the data structures, you can fall back into pure procedural style programming in ADTs. The result is that ADT style programming looks similar Object-Oriented at the low-level, but is really structured as 'algorithms' at the high-level. A well-structured ADT program consists of a number of data-structures, the access functions and a series of higher-level algorithms that work the overall logic of the system. Since data drives most of the logic in most programs, the non-data parts of the code are usually control loops, interfaces to dispatch functionality or glue code to interfaces between different control loops. Which ever way, they can be coded in the simplest most obvious fashion, since there are no structural requirements.

That ambiguity in defining the style of the code at a high level in ADTs is very important. This means that there is a natural separation between the data and the algorithms, and they get coded slightly differently. But we'll get back to this later.


CODING SECRETS

Software development often hinges on simplifications, so it is not surprising that we really only want one consistent abstraction in our programming languages.

Because of this, we spend a lot of time arguing about which approach is better: a fixed language, that is strongly-typed or one that is loosely-typed. We spend a lot time arguing about syntax, and a lot of time arguing about the basic semantics. Mostly we spend a lot time arguing about whether or not the language should be flexible and loose, or restricted and strict. It is a central issue in most programming discussions.

If you come an impasse enough times, sooner or later you need to examine why you keep returning to the same spot. Truthfully, we work in two different levels with our implementations. At the higher level, the instructions are more important. At the low level it is the data.

At the low level we want to make sure the data that we are assembling is 'exactly' what we want.

When I was building a PDF rendering engine in Perl, for example, I added in an extra layer of complexity. The engine was designed to build up a complicated page as a data-structure and then traverse it, printing each element out to a file. This type of processing is really simple in a strongly typed language, but can get really messy in a loosely typed one. To get around Perl's loose semantics I wrapped each piece of data in a small hash table with an explicit type. With the addition of a couple of really simple access functions, this had the great quality of emulating a strongly typed syntax, both making the code easier to write, but also guaranteeing less errors. It was a very simple structuring that also made it really easy to extend the original code.

At the higher level we want to focus on the instructions and their order, data is not important. Batch languages and text processing tools are usually loosely typed because it makes more sense. The data is really minimally important, but the order of the functions is critical.

As another example, I was building a dynamic interface engine in Java. In this case, I wasn't really interested in what the data was, only that at the last moment it was getting converted into whatever data-type I needed for display or storage. This type of problem is usually trivial in a loosely-typed language, so in Java, I added in a class to 'partially' loosely type the data. Effectively, I was concerned with 'removing' via a single call, any of the attributes of the type of the data. It does not matter what the data is, only what it should be. Again it is a simple structuring, but one that effectively eliminated a huge amount of nearly redundant code and convoluted error handling.

With enough coding experience in different languages, you come to realize that being strict at a low-level is a great quality, but being flexible at the high level is also important. It is easy enough to get around the bias of the language, but it does count as some additional complexity that might prove confusing to other developers. It is no wonder that the various arguments about which type of language is superior are never conclusively put to rest, it is more than a simple trade-off. There is an inherent duality related to the depth of the programming. The arguments for and against, then are not really subjective as much as they are just trying to compare two completely different things.

Well-rounded implementations need both strong-typing and loose-typing. It matches how we see the solutions.


INTERNAL MODELS

Another fundamental problem we face, comes from not realizing that there are actually two different models of the solution at work in most pieces of software.

The way a user needs to work in the problem domain is not a simple transformation from the way the data is naturally structured. The two models often don't even have a simple one-to-one mapping, instead the user model is an internal abstraction that is convenient for the users, while the data model is a detail-oriented format that needs to be consistent and strictly checked.

A classic example is from a discussion between Jim Coplien and Bob Martin:

http://www.infoq.com/interviews/coplien-martin-tdd

Specifically Jim said:

"It's not like the savings account is some money sitting on the shelf on a bank somewhere, even though that is the user perspective, and you've just got to know that there are these relatively intricate structures in the foundations of a banking system to support the tax people and the actuaries and all these other folks, that you can't get to in an incremental way."

In that part of the conversation Jim talked about how we see our interaction with the bank as a saving account, but underneath due to regulatory concerns the real model is far more complex. This example does a great job of outlining the 'users' perspective of the software, as opposed to the underlying structure of the data. Depending on the usage, a bank's customers deal with the bank tellers operating the software. The customer's view of their accounts is the same one that the system's users -- the tellers -- need, even if underneath, the administrators and regulators in the bank have completely different perspectives on the data.

While these views need to be tied to each other, they need not be simple or even rational. The user's model of the system is from their own perspective, and they expect -- rightly so -- that the computer should simplify their view-point. The computer is a powerful tool, and one of its strengths is to be able to 'transform' the model into something simpler allowing the user to more easily interact with it, and then transform it back to something that can be stored and later mined.

It is a very common problem for many systems to have the software developers insist that there is only one 'true' way to look at the data. The way it needs to be structured in the database. It is not uncommon to see forms tools, generators or other coding philosophies that are built directly on the underlying data model.

This makes it possible to generate code and it cuts down on the required work, but rarely is the data model structured in the same way as the user needs. The results are expected. The user's can't internally map the data to their perspective, so the applications are extremely awkward.

A single model type of design works only for very simple applications where there is little mapping between the user model and the data model. Mostly that is sample applications and very simple functions. While there are some applications of this sort, most domain specific systems require a significant degree of complex mapping, often maintaining state, something the computer can and should be able to do.

The essence of this, is that the 'users' have a model that is independent of the 'data' model. E.g. The interface translates between the real underlying structure of the data, and the perspective that the user 'needs' to complete their work. More importantly the models are not one-to-one, or we would be able to build effective translators. This duality and inconsistent mapping is why code generators and forms tools don't work. You can't build a usable application from the data model, because it is only loosely tied to the user model.


OBJECT ORIENTED

So what does this have to do with Object-Oriented programming?

The structure of an Object-Oriented language makes it relatively easy to model the structure of the data in the system. At the lower data level, the semantics of the language allow it to be used to build complex systems. You create all the objects that are needed to model the data, or the user abstraction. Object-Oriented languages are at their height when they are encoding easily structured objects into the language.

The classic example is a GUI system where the objects in the system are mapped directly to the visual objects on the screen, that type of mapping means that when there are problems it is really easy to go back to the screen and find the errors. The key here, is that one-to-one link between the visual elements and the objects. The point of going to a lot of extra effort, is to make it easy to find and fix problems. You pay for the extra effort in structuring the OO code by getting reduced effort in debugging it.

At the higher level, all of the OO languages totally fall apart. We add mass amounts of the artificial complexity into the works just to be able to structure the problem in an Object-Oriented manner. Ultimately this makes the programs fragile and prone to breakage. Ideas such as inversion of control, and dependency injection are counter-intuitive to the actual coding problems. They become too many moving parts.

Now, instead of the problem being obvious, you may need to spend a significant amount of time in a debugger tracing through essentially arbitrarily structured objects. Constructs like Design Patterns help, but the essence of the implementation has still moved far away from the real underlying problem. The developer is quickly buried in a heap of technical complexity and abstractions.

If what you wanted was a simple control loop, a traversal or some hooks into a series of functions, it would be nice if the simplicity of the requirement actually matched the simplicity of the code.

Even more disconcerting, we must leave our primary language to deal with building and packaging problems. We use various other languages to really deal with the problems at the highest level, splitting the development problem domain into pieces. Fragmentation just adds more complexity.

Our build, package and deploy scripts depend on shell, ant or make. We could write the build mechanics in our higher level language, but writing that type of building code in Java for example, is so much more awkward than writing it in ant. If we see it that clearly for those higher-level problems, does it not seem as obvious for the higher-level in the rest of the system?

Another critical problem with the Object-Oriented model is the confusion between the user model, and data model and the persistent storage. We keep hoping that we can knock out a significant amount of the code, if can simplify this all down to one unique thing. The Object-Oriented approach pushes us towards only one single representation of the data in the system. A noble goal, but only if it works.

The problem is that in heading in that direction, we tend to get stingy with the changes. E.g the first version of the application gets built with one big internal model. It works, but the user's quickly find it awkward and start making changes. At different depths in the code, the two models start to appear, but they are so overlapped it is impossible to separate them. As integration concerns grow, because more people want access to the data, the persistence model starts to drift away as well.

Pretty soon there are three distinct ways of looking at the underlying business logic, but nobody has realized it. Instead the code gets patched over and over again, often toggling between leaning towards one model, and then back again towards another. Creating an endless amount of work. And allowing the inconsistencies tp create an endless amount of bugs.

If the architecture identifies and encapsulates the different models from each other, the overall system is more stable. In fact the code is far easier to write. The only big questions comes about with deciding the real differences between the user model and the data model. When a conflict is discovered, how does one actually know that the models should differ from each other or that both are wrong?


SUMMARY

Ironically, even though they are probably older and less popular, ADTs were better equipped for handling many implementation problems. Not because of what they are, but because of what they are not. Their main weakness was that they were not enforced by the language, so it was easy to ignore them. On the plus side, they solved the same low-level problem that Object-Oriented did, but allowed for better structuring at the higher level. It was just that it was up to the self-discipline of the programmer to implement it correctly.

I built a huge commercial system in Perl, which has rudimentary Object-Oriented support. At first I tried to utilize the OO features, but rapidly I fell back into a pure ADT style. The reason was that I found the code was way simpler. In OO we spend too much time 'jamming' the code into the 'right' way. That mismatch makes for fragile code with lots of bugs. If you want to increase your quality, you have to start with trying to get the 'computer' to do more work for you, and then you have to make it more obvious were the problems lay. If you can scan the code and see the bugs, it is far easier to work with.

We keep looking of the 'perfect' language. The one-size-fits-all idea that covers over both the high and low level aspects of programming. Not surprisingly, because they are essentially two different problems, we've not been able to find something that covers the problem entirely. Instead of admitting to our problems, we prefer to get caught up in endless arguments about which partial solution is better than the others.

The good qualities of ADTs ended up going into the Object-Oriented language design, but the cost of getting one consistent way of doing things was a paradigm where it is very easy to end up with very fragile high-level construction. Perhaps that's why C++ was so widely adopted. While it allowed OO design, it could still function as a non-OO language allowing a more natural way to express the solution. Used with caution, this allows programmers to avoid forcing the code into an artificial mechanism, not that anyprogrammers who have relied on this would ever admit it.

The lesson here I guess is that the good things that make Object-Oriented popular, are also the same things that make it convoluted to some degree. If we know we have this duality, then for the next language we design for the masses, we should account for this. It is not particularly complicated to go back to ADTs and use them to redefine another newer paradigm, this time making it different at the low and high levels. Our own need for one consistent approach seems to be driving our technologies into becoming excessively complicated. We simplify for the wrong variables.

We don't want to pick the 'right' way to do it anymore, we want to pick the one that means we are most likely to get the tool built. Is a single consistent type mechanism better than actually making it easier to get working code?

Testing Perspectives

I just finished listening to a discussion between Jim Coplien and Bob Martin:

http://www.infoq.com/interviews/coplien-martin-tdd

and there were some really interesting things in that conversation.

The most stunning was that Jim Coplien works with systems that are hundreds of millions of lines of code. That is a massive amount of code. Shouldn't a higher paradigm like Object-Oriented save one from such a difficult fate? How long does it take to assemble that much code? Why hasn't millions of it been refactored?

The next is that even though Jim was dead-on about his observations of having to start close to the underlying understanding of the actual domain in his example of writing software for a bank -- otherwise you'll spend 40 years reinventing a well-known wheel -- that center of the conversation just died. If you know how you need to structure your system, why wouldn't you go straight there with the minimum amount of work, especially if you have huge time constraints. Approaches such as CDD or TDD seem to be taking the long way around the block.

Jim's point about developers without an architecture painting themselves into a corner is true irregardless of any discussion on Object-Oriented or Agile issues. It is a long standing tradition in Computer Science for aimless meanderings to finish off a project badly, that hasn't changed in 40 years. It didn't work before OO, YAGNI, TDD or any other acronym and it still won't work after. If you don't start close to where you need to be, you probably won't ever get there unless you are really lucky.

Bob's description of TDD seems more like a 'game' where you are pitting two different representations of the same thing against itself. All the more so when you realize that Jim is correct about unit testing just being a game of 'sink the battleship' where you randomly fire specific tests over a large surface hoping to hit something, before running out of time and shipping the code. Testing is always about getting lucky. And doesn't context switching between tests and code tired one out more quickly? For speed, I've always liked to batch up the work as much as possible.

Bob's thesis that it was unprofessional to 'not' unit-test every line of code before a release was absolutely crazy. And not possible. Mass buckets of code get released by 'professionals' all of the time where the 'coverage' of the testing is probably way, way less than 30%. That's just a guess, but an easy one. TDD doesn't even stop that; given all of the areas were you can't use TDD (GUIs and databases?), huge amounts of untested code will get loose. Untested code will always get loose.

Even if you unit-tested each and every line to 110%, the combination all of the pieces will open up unexpected and undesirable results; which if you didn't have some overall architecture to tie it all together would be a particularly huge and likely fatal mess. You can't skip integration testing, it isn't safe. If you have to do a lot of it at the end, then wouldn't it be better to minimize the stuff in the middle or at the start?

Further, TDD is not a main-stream process right now, even if it is getting lots of coverage from the 'Agile Press'. Most code is still written using waterfall methods, for most industries. Agile itself isn't even main-stream, and TDD is only a tiny piece of it. Saying it is so, doesn't make it any truer, and it leaves a bad taste in the mouth.

The weirdest part for me is that both of these guys are arguing about how to make code more stable, or at least that's what I hope they are arguing about, otherwise what is the point? All this low-level testing is nice, but it doesn't invalidate the higher level full-system testing that 'must' be done to end-to-end test the various pathways through the code. When is low-level testing effective and when is it just doubling up the workload for no apparent reason?

So, I add a process to my house-building technique that makes sure each and every brick is super-accurate. Now they are all exactly the same size given some huge level of precision. That's nice, but does that really help me build a better house?

A Silly Little Storm

This posting looks into a small and common problem, but one that frequently causes larger issues. Even the tiniest of problems, which starts as a small rock in a stream disrupts behind it an increasingly larger segment of water. This type of turbulence exceeds the sum of itself as more rocks combine together to cause larger and larger ripples. With enough rocks you get rapids; the whole being much larger than its pieces.

Sometimes when I am commenting on other writer's blog entries I just want to end my comment with a little bit of shameless self-promotion. Usually my name and then a link back to my blog to make it easy for readers to stop by and visit me:

Paul.
http://theprogrammersparadox.blogspot.com

These are my favorite last two lines in most comments.

Of course, to avoid mistyping the URL I generally go back to my blog's root page, highlight the URL, copy it and then paste that into the comment box a couple of times. Pasting decreases the likelihood of typos; usually a good thing.

Someone pointed out that in Blogger I should wrap the URL in an anchor tag so that it is easier for any readers to click on the link. No problem. That seems easy enough. The simplest way to accomplish this is to type in the starting tag '<a href=', paste in the URL, close it with '>'; then paste in the URL again -- to make it the visible text as well -- and finally add in the closing '</a>'. How hard is that?

In Blogger, when I go to the root page for my blog, I get an extra forward slash '/' character because it is at the 'top' level, and not pointing to any specific entry. If I selected a blog entry then the URL ends with the '.html' extension because it is directly referencing a file. But normally if I want the top level URL for my blog, I'll go to the main page and copy the URL with its extra forward slash at the end.

Funny enough, in Blogger there is this stupid little 'issue' where an anchor tag in the form of "<a href=XX />link</a>" that has an extra '/' at the end of the open tag works perfectly in the preview, but is slightly modified when actually posted. In the preview it appears as if the forward slash is ignored -- everything looks fine, which is no doubt a browser dependent behavior -- and FireFox interprets the tag as expected.

But once you post the comment:

<a href=http://theprogrammersparadox.blogspot.com/>
http://theprogrammersparadox.blogspot.com/</a>

it gets turned into:

<A HREF="http://theprogrammersparadox.blogspot.com" REL="nofollow"/>

Visually the hyperlink text disappears and any of the following text becomes the link instead. As well, the newlines get turned into breaks and there are paragraph markers inserted at the beginning and end of each paragraph.

I am guessing, but when you actually post the comment it is sent to the database for storage. On the way in or out, but usually in, the comment text is modified by some well-meaning, but obviously destructive code. That 'function' is looking to perform some modifications on the comment text; from above we see it added an REL option to the anchor tag; it also converted the tag to uppercase. Because it sees the anchor tag as unary, it seems to strip the orphaned closing '</a>' tag. It probably does lots of other stuff as well. The parsing for that function saw the close forward slash and assumed that it was part of the HTML tag syntax, unlike the browser that sees it as part of the earlier directory path syntax. The browser's parsing behavior is described as 'greedy' because it tries for the longest possible token, which in this case is the full directory path with the appended forward slash. The internal function on the other hand is probably ignoring the directory path syntax and is just looking directly for '<', '>' and any '/'s to modify them. Since it isn't fully parsing the HTML it doesn't have to understand it as well as the browser does.

So, wham, bam, I copy the URL into the text a couple of times, check it in the preview and kapow, I post it. "Oops, what happened there?" Suddenly the text and the link looks very different from the preview. If you combine both of the above issues that simple little extra '/' picked up in the URL causes quite an annoying problem, but not one that is detectable in preview. Everything looks great until you actually post. A very embarrassing error.

You may have noticed that I didn't wrap the href argument in double quotes. That certainly would have fixed the problem, but why toss in an extra set of double quotes when the preview has already shown that not having them will work properly. It is just extra stuff that doesn't seem necessary. Dropping the trailing '/' works too. But the 'problem' isn't what is actually typed, the problem is the preview. It is incorrectly verifying that stuff is working, when clearly it is not.

The cause here beyond my own sloppiness starts with two different points in history where at least two different programmers assigned two different meanings to the same underlying forward slash character: one as the separator between directory names in a path string, and the other as a modifier to the tag syntax to indicate a unary tag or an ending tag depending on placement. These two distinct meanings 'overload' the meaning of the character so that when combined in this specific circumstance the true meaning is 'nearly' ambiguous; one block of code sees it one way, while the other sees it differently. When we 'paste' together different 'idioms' that have been crafted over time, we often get impedance mismatches. Little blots of dangerous complexity.

Added to that, two different pieces of code are parsing the same HTML fragment very differently. However close Blogger's HTML parsing routine is to FireFox, it is different enough to cause problems. I could try testing this in Internet Explorer, but it doesn't really matter to me who is right and who is not following the standard. FireFox is being greedy, Blogger is not. More likely is that to greedily parse the directory path, Blogger's code would have to go to a huge amount of extra work, most of which was an unnecessary level of complexity beyond what they actually needed to manipulate the comments. They won't ever parse the HTML fragment correctly, because they do not need to go to that level of understanding.

What is important is that the same basic 'data' is being interpreted in multiple ways by multiple different programs and that while it is not ambiguous, it might as well be, since any two coders cannot and will not parse it identically. Partial parsing will always produce ambiguities. Overloading will too. The meaning of our data is so often relative to how we handle it in our code. We fail to separate the two; implicitly buring knowledge into some piece of code.

Topping it all off, some programmer implemented the preview function without fully understanding what he or she was doing. If there are 'translations' done in other parts of the system then it is exactly these 'translations' that need to exist in the preview function. Without them preview is useless. Incomplete. A 'mostly' preview function is a sloppy dangerous featurette that mainly misleads people and diminishes the tool. In the end had this been implemented correctly, none of the other issues would matter or be of any consequence. I would have seen the broken link in the preview and fixed it. Although earlier problems set up the circumstance, the 'buck' stops at the final and most obvious place.

We might quickly dismiss this type of issue as trivial -- perhaps I should just learn to quote my arguments correctly -- if it wasn't for the fact that this type of thing is standard all over our software. It is one hell of a turbulent stream of never-ending little issues that together keep this stuff from being trustworthy. I have the greatest mental tool known to mankind on my desktop, but I can't trust it to cut-and-paste properly? We've so many old legacy problems acting as rocks in the stream that our users are constantly drowning in rapids of our own making. And so often, all we ever do is tell them that they shouldn't have added that extra forward slash; "everybody knows you can't have extra forward slashes"; and we wonder sometimes why so many people hate computers?

A related theory is that you can judge the overall complexity of a 'machine' by the size of the effects caused by changes to its small details. In a simple system, small changes have only a small impact, but as the overall complexity grows chaos becomes more prominent so that small issues have progressively larger and larger impacts. The inherent complexity of the problem leads us to a base change effect of a specific size, but with every increasing dose of artificial complexity those effects become magnified. In a big system if you can draw some conclusions about the size of the initial problems vs. the size of their effects, over time you might get some gauge on how quickly the artificial complexity is accelerating within the system. This is important if you are interested in how many more changes you could make to the existing system without applying some type of major cleanup before it becomes likely that the changes will become impossible to make. Once it has passed some threshold, no significant change can be made to an overly complex system without it causing sever and damaging side-effects.

UPDATE: I'm not even going to try to put words to the amount of frustration I just experienced trying to move this piece between Google Docs and Blogger. The formatting, anchor tags, are not happy; consider yourself lucky if you can even read this...

From Our Inner Depths

The biggest problem with the World Wide Web is that it is just too hard to find anything "unique" anymore. In an instant, you can gather together at least fifty hits for just about anything. Are there any two word phrases left that don't at least get one hit? It is hard to imagine.

Despite the existence of many other published uses, I'm still going to proposed a new definition for a pair of old words. Sometimes you can't let prior history hold you back, particularly when the terminology is so appropriate.

If you've been using software for a while, you've probably been there. It is version X of the program; some function you relied upon, while still in the system no longer works correctly. Sometimes it a bug, sometimes it is a legacy problem, but often it is just because the new programmers maintaining the code were clueless. They had no idea how the system is really used in the real world, so they removed or broke something vital. They get so busy adding new crappy features and ignoring the existing bugs that they don't realize how much they are diminishing the code base.

Software -- we know -- starts out needing a lot of features. In its life, often the big programs have many, perhaps hundreds of developers pouring over the code and enhancing it. As the development grows, the quality and functionality get better and better, until at some point it reaches its maximum. Never a good thing. It is an inverted curve, going up often quite quickly, but after the high there is no where else to go but down. Down, down and down.

Now you'd think in this day and age we'd know when not to mess with a good thing, but the horrible little side-effect of capitalism is that nothing can ever remain static. Ever. If it ain't growing, then it must be dying. Or at least not living up to its full revenue generating potential; often the same thing in business school.

And so, all of our favorite software packages 'arc'. Reaching their highs in some early release, only to be followed by a long slow steady decline into the abyss. If you've hung around long enough, you know this to be true. What goes up; as they like to say...

Back to my career in creative terminology.

You should always honor those that paved the way, it is only fitting. As we take in consideration the above issue about the lifespan of software we need to fall back a bit and consider the user's perspective. Each of us who have aged a bit, has had the misfortune of being bitten by some new dis-functionality in the software. While that may be interesting -- often depending on how pressed we are for time -- our reaction to finding this out is less than amiable. Like automotive drivers completely losing it on a busy highway, I've seen many instances of software users succumbing to "Word Rage" in my day. When the user shakes their fist at the screen, curses the maker of the product and vows to set forth as much negative karma as possible on those responsible for the atrocity, you know you are seeing a genuine instance.

Word Rage. Hits when that #$%$% auto-formating stuff re-arranges a perfectly good document. It hits when you try to turn off those brain-dead auto typing crapifiers. When the menus are truncated. When that hyperlink nonsense doesn't work. Any of it. When the backup copies are no longer saved, or the save files are no longer readable. When the documentation tells you nothing, and there are fifty million stupid meaningless options that are choking up the dialog; obscuring that one stupid thing that needs to be unset; again. Whenever that program carelessly wastes hours and hours of your time, while you were only trying to do something both obvious and practical that any decent minded programmer should have though necessary. Word Rage!

But alas, while I coined the term after the masters of inciting it, many many other software packages produce equally dramatic effects. Just recently, I found that the plugin I used in a well know photo manipulation program wouldn't work with the built-in batch capabilities. I couldn't create an action for changing either the height or width that depended on the orientation and that the plugin I used for saving wouldn't work in batch either. Of the four simple things I needed to automate, three of them concealed bugs, or at very least serious weaknesses. The coders allowed the necessary capabilities to be divested into the plugin facility, but failed to integrate that into the batch capabilities; making batch useless. At least we get nice spiffy plugins; that are only partly useless. Way to go guys!

I could spend all day ranting about this sort of stuff; there is enough for perhaps years and years worth of blogging. I just doubt anyone would read it; it gets old, fast.

Our software is full of inconsistencies, bad ideas, stupid mistakes and poorly thought out designs that make it all rather irritating to try and get anything done. I find I don't have any Word Rage if and only if I'm not actually trying to get anything accomplished. The stupid machines work fine, just so long as you don't touch em. In them good ole days we didn't have a lot of functionality, but at least if it existed you could probably trust it. Now we have the world, but we waste more time fiddling with silly problems. The tools often bite their masters.

Lately I've been swearing off technology; insisting that I'm going to switch to woodworking instead. I need to rely on something that works properly. When you get to the point were you think being a Luddite would actually help make "better" tools, you know you've been enraged once too often.

Mind the Gap

Sometimes to get a clearer perspective on a problem we need to leave its proximity and venture a little deeper into the woods.

For this particular post, we need to go way out into the forest before we can come back and make sense of what we are seeing. With a little patience and time, I think you'll find that the rewards from exploring are well worth the stumbling around in the bushes and trees.

As an abstraction, pure mathematics is nothing short of absolute perfection. It exists in its own idealistic dimension with no interference from our world in any way.

As such, arithmetic for example is always consistent no matter where you are in the universe. It never changes, it is never wrong, it cannot be reduced to anything lower and it cannot be refactored into anything simpler. Quite an accomplishment and a model of perfection. The same is true of many other branches of pure mathematics.

As more of the real world gets involved in the equations -- such as physics -- the details become messier, but with respect to what they subscribe to, it is still close to perfect. The details cause complexity, but it is inherent in the problem domain, and thus acceptable.

Going out even further to some mathematically based soft science such as Economics we see even more real world detail, but also another type of complexity that is derived directly from humanity. I'd call it "errors", but that's not really fair. It is better described as the "not perfectness" that is driven by human personalities, history, interaction and thinking. It is working knowledge that is full of artificial complexity; where some reduction always exists that could simplify it. Unlike inherent complexity, it is not necessary.

So, we have this point of perfection that we often drift way from because of the complexity of the underlying details and also because of our own introduced complexity. These points are all important.

There is nothing we can do about real world complexity. It is as complex as it needs to be in order to function. That complexity comes directly from the problem domain and is unassailable.

The complexity we have created ourselves, however, is more interesting. Underneath it lies some minimal real world complexity, but more because of how we got there the complexity we are dealing with is significantly larger. This leads to a "gap" between the perfect solution and the one that we currently have. It is an important gap, one that bears more digging and understanding.

Looking at software, Computer Science revolves around building tools to manipulate data. In both our understanding of the data and our understanding of the tools themselves we have significant room for introducing problems.

With a bit of practice, it is not hard to imagine the perfect tool manipulating the perfect data representation. We are always driven to complete tasks, so on a task by task basis we can close our eyes and picture the shortest path to getting the task finished. We can compare this perfect version of what we are trying to accomplish to the tools and data representations that we actually have intuitively measuring the gap between the two. And it is a surprisingly large gap.

I am often very aware of just how big of a gap. I was lucky to work on some very complex but extremely well written systems, so I have a good sense of how close we can really come to a perfection solution. I can guess at the size of the minimum gap, and while some gap must always exist the minimum can be way smaller than most people realize. There is a huge latitude for our software to improve, without actually changing any of the obvious functionality.

We don't need fancy new features, we just things that work properly.

Over time the trend that I have seen is for our software to get larger and more sophisticated. It is clearing improving, but at the same time however, the gap has also been steadily growing; wiping away many of the improvements.

Given our current level of technological sophistication, today's software should be far better than it actually is. We have the foundations now to do some amazing things but we don't have the understanding of the process to implement them. We have a huge gap. Our software is much farther way from its potential than it was twenty years ago. A trend that is only getting worse with time.

The size and growth rate of the gap are important. It not only gives us an indication of how well we have done with our science, but also what our future directions will be. Given that the gap is growing faster than ever, my virus-laden PC that is choked by bad software and malicious attempts at crime and profit comes as a real embarrassment to the software industry. In the past we have been lucky and the expectation of the users has diminished, but one should really brace themselves for a backlash. We can only make so many excuses for the poor behavior of our systems before people get wise.

For instance, a virus obviously isn't necessary to make a computer function properly. As such, it is clearly unnecessary complexity that comes from the wrong side of the gap. If things had been written better, creating a virus would be impossible. The gap widened to allow spyware and viruses to became reality; then it widened again to make anti-software a necessity. Now we consume mass amounts of human effort on problems that shouldn't exist.

Computers are deterministic machines. Specifically what I mean is that they behave in completely predictable ways and with some digging all of the visible actions of the machine should be explainable. It is surprising then to encounter experienced professionals who tell stories of their machines "just acting weird". If it were one or two people we might write it off as a lack of training or understanding, but who amongst us has not experienced at least one recent unexplained behavior with their PC?

More to the point, for professionals working for years, hasn't this been getting more and more common? All of these instabilities come from our side of the gap. Being deterministic, our computer systems could be entirely built to be explainable. They should just function. Weird behavior is just another aspect of the gap.

The gap is big and getting bigger, but is it really important? When we build tools, we need our foundations to be stable so that we understand the effects of the tool. Building on a shaky foundation makes for a bad implementation. The gap at the base only gets wider as you build on top of it. Nothing you build can shrink it. If it is too wide, you can't even bridge it. The usefulness of a computer and the quality of our tools is dependent on the size of the gap.

So, standing back here in the forest, we see where we are currently with our latest and greatest technologies. Somewhere over the next hill, down into the valley and up the other side is the place where we could be if we could find a way to minimize the existing gap. We can see it from here, but we can never get there until we admit that 'here' is not where we want to be. If we stay were we are the things we build only get farther away from our potential. We amuse ourselves with clever bits of visual trickery, but progress doesn't mean more dancing baloney, it really means closing the gap. Something we should do as soon as possible.

Acts of Madness

Just a quick posting:

What kinda madness would convince people to base Open Source projects on proprietary technology, particularly when the technology belongs to a company best-known for riping off other people's technologies?

How much pain do people self-inflict because they fail to learn lessons from history?

Apropos for this: "if you lie down with dogs, you get up with fleas".

Wasted my Time

No doubt it is my own foolishness. My expectations are too high.

When I blog, I use Yahoo Notepad to write the draft entries. That gives me portability, and access to the draft articles wherever I am. The idea that I can access my writing from literally anywhere in the world is amazing. Something wonderful provided for me by the computer.

I was struggling with a particularly hard to write piece, where I was trying to take the reader backwards towards using first principles. Not easy to begin with, and even harder for a novice writer like myself.

I was happy with what I had done. I probably would have posted it.

After agonizing for a couple of hours, I hit the update button. Something I've done a million times before. This time though, because I had been logged in for so long, Yahoo Notepad decided to ask me to confirm my userid. The session had timed-out. Ok. No worries. This problem has caught me in the past, and in Firefox if I hit the back button, I can get back to the original screen with the textbox, which from there I can copy and paste into a text file, and use it to update the note after I've logged in again.

I could test it, but if I remember correctly, in Yahoo Notepad, after the login screen, the data in the textbox is lost. It goes through as a post, I think, and cannot get passed back into the next query, so even if the program continues from the point it left off, the right data will not be updated, and possibly the file would end up being blanked if they tried. I could be wrong about this, hitting the back button and copying out the data worked well in the past, I seem to remember. The myths we build up about our software is often interesting in themselves.

This time, when I hit the back button to my horror the contents of the textbox where the original contents, not my updated version. My notepad file is quite large. It contains a lot of draft material, so I suspect that the size of the textbox forces Firefox not to save the data. So, going back won't work. I went forward too, but that didn't seem to work either. Although by the time I tried that, the panic had already set in. All of my labor, all of my work was gone!? Vanished. Kaput. Ahhhhhh.

When the anger subsided, and I stop swearing and pounding on things I really starting thinking about it.

In the early days of PCs, in Word Perfect we developed a disciple where we hit save file function every 10 minutes or so. It was necessary because the machines of the day were so flaky, that you were always running the risk of losing work. 10 minute intervals minimized the amount of lost work.

In those days, our technology was new and had problems. If someone lost their work, we told them it was their fault. They clearly weren't saving enough. We pushed the obvious problems with the technology back onto the users. Later a big advancement was to performed this neuroses automatically. The computer could take care of constantly saving the file for you.

A while back, Microsoft Word crashed, and I lost a couple of hours worth of work. I though it had been doing the ten minute backup thing, but that option was turned off. Not sure why, could have been a local thing or an network share problem, or something equally uninteresting.

Hardware crashes are not as common anymore, although it might just be bad luck, but I've had more problems with hardware in the last couple of years than I have for the prior decade. At work, for instance I've had four machines die in less than a year. There seems to be a noticeable decline in the dependency of modern hardware.

I'd figured Word was set up correctly, so I never bothered to check it myself. It wasn't. But I should have checked, they said. It was pushed back to me as the user, to have notice that Word was no longer creating backups. This brought on a twisted sense of deja vu and a 'sad realization'.

A computer is an extremely powerful tool; it can do so much. It has been around for decades and can automate a huge number of tasks.

It is sophisticate enough that we could build a system that could not lose any of your work beyond a few seconds, right up to and including a major catastrophic event like and asteroid colliding with earth. Since we don't have a backup planet, there is little we could do about that, but we do have technology designed to withstand a nuclear war, major storms, earth quakes and we do know how to create fault tolerance redundant distributed implementations.

Disk storage is cheap enough that we can keep all of the key strokes we've ever entered into the computer for our entire lives; allowing us to rebuild any or all of our labors.

Our technology should not allow us to ever lose work. We have progressed beyond that point.

Computers have a huge amount of potential, unknown to even the most savvy of us, but we have not been able to tap it. We write the same stuff over and over, repeating the same mistakes and never really learning from the past. At its best, a web application is possibly the worst ever technology invented. One giant mess of poorly implemented ideas and political maneuvering. A great big ball of mud. We have at the foundation, a great technology in the Internet, that we've build a huge mess on top, the World Wide Web. We connect to it with our easily modifiable PCs, that are complete untrustworthy and getting worse with each new release. Web applications are improving, but they are still extremely crude. And very disappointing. One step forward, and one step back.

But our problems aren't technological, they aren't something we need to invent or fix. We have all of the technology we need, all of the understanding. Our problems lie with us, the people building the technology, with our culture, and our need to be free, inconsistent and messy, and with the way we organize ourselves.

For decades now we have approached the same problems in the same manner, and consistently the results have been flawed, yet we do it again and again. I really don't think it is difficult to build working systems, but it does seem extremely difficult to organize people in a way that they can build working systems.

We have the technology, if only we could figure out a way to utilize it to our advantage. Until then, I guess I'll just have to lower my expectations and get used to losing my work every so often. It is only a couple of hours...