Creatively Speaking

Somewhere in my youth, I came into possession of a very simplified model of the human brain. I'm not sure from where, it was so long ago its roots were lost in time.

The idea is simple, the brain consist of three, and only three things. First is memory. We can memorize stuff. By that, I mean that we take some fact, associate it with a number of keys and then, on demand, we can return the fact as needed. It is committed to memory. It becomes a piece of knowledge, something we know. I'll quietly ignore long-term vs. short-term memory for this model.

The next thing is understanding. That is, we understand something. For this, I mean that we can take some set of facts that we learned, and at the appropriate time use them for some piece of higher reasoning. If you've just memorized a fact, that's nice, but until you understand it, you can't really take advantage of it. In some sense, understanding is really just how that set of facts is linked to other sets of facts that can in some way drive or change your behavior. You 'know' what an stove burner is (fact), and you 'understand' that once its turned on, it is hot and will burn you.

The very last thing is creativity. This, conceptually is a little more complicated. If a fact is a thing you remember, and understanding is how it is linked, creativity in some offbeat way is how you can create 'dynamic' linkages between non-related facts or understandings. You might have a pencil for instance, and you know how to use it to draw shapes and textures, but at the time you are working you're feeling blue, which is some intangible emotion. If you want to use the pencil and your ability to draw to express your emotion state by drawing a dark moody image of yourself, you are taking your pencil, and your self-view, both facts, applying your understanding of drawing and then 'combining' that with your current emotional state to 'create' something that on its own would not come into existence for anyone else. You are creatively expressing your emotions through a self-portrait with a moody overtone. The work is not reproducible.

Creativity then is this ability to combine two entirely unrelated things, in arbitrary, random, or mostly irrational ways. The base things are either facts or understandings.


CONSEQUENCES

A consequence of this simplified model is that the abilities of a human on one level are quite simple. We build up knowledge -- linking it to other bits -- in ways that we find useful. Indirectly, as if it were a computer program run amok, one can see creativity as some type of inherent 'flaw' in our brains. After all, connecting two things that "shouldn't" be connected is not rational or deterministic. It's like a computer programming randomly changing the data, it is outside of the norm. The brain shorts out, or something.

From a natural point of view, this model allows one to see human progress often as this long history of people gradually creating new ideas and memes by either observing them in nature or randomly combining together un-related bits. In a direct sense, all of our of knowledge beyond what is quantifiable, is accidental. We stumble into discovering it.

That's a little hard for many people to swallow, so it likely shows that this model is far to simplified to be absolutely correct, but still quite interesting. To get to the really high concepts, they have to be based on lower ones, that somewhere down the way have to be based on observable things, or creative leaps.

I like this model because it provides nice tidy explanations for many things. For instance, why somebody with 'photographic' memory does so well in their early university courses, but then has trouble later on; they never really understood the original stuff, just memorized it. The latter courses require understanding as well as memorization. You can't build on a foundation that you don't have.

Intelligence is not just our ability to remember facts, understanding and creativity play a huge role as well. So that means a brilliant person is one, then who has a lot of facts, and understands them well. While a genius is an intensely creative person, who many not actually sit on that big a reservoir of understandings or facts; but they can just combine what little they have into interesting new bits. There are a lot of brilliant people, while creativity, is a rarer thing.

This model harmonizes 'intelligence' and shows that one cannot easily compare their three values against other people's three values. With three dimensions, you cannot easily rank your intelligence. Often you can meet people who can quote tremendous amounts of knowledge, but seem to have trouble really putting the pieces together. Your typical absent minded professor may be incredible forgetful, but full of deep understandings. It is a great perspective on our abilities.


COMPUTER SCIENCE

Now, all of that being interesting, what does it have to do with computer science? I read a excellent write up recently of someone talking about how they believe that math is actually creative. It is know as Lockhart's Lament and it is at:

http://www.maa.org/devlin/LockhartsLament.pdf

This to me this is a hugely misunderstood characterization of what it means to be creative, and of what mathematics is, and why it is so special. In a way, relating it back towards creativity takes away from the true underlying awe and beauty of mathematics. It trivializes it.

In a very real sense, mathematics is an infinite set of axioms over an abstract space. It does not exist in our real world and because of its rigour, which could not exist either in this world, it is purely abstract. Math is sort of this pure crystallize abstract structure that we can perceive if we concentrate on it long enough. It exists, but without substance. A truly amazing thing.

As such, all math exists and has always existed. It always will, with or without us.

The branches, axioms, primitives etc. of the various different types of math are our expressions of the that abstract foundation. It is out there, right now. All of our existing math -- a drop in the overall bucket -- and all of the stuff we still do not know. Mathematicians set out on a quest to 'find' new mathematics. They find it, prove it is rigorous, and then explain it to the rest of us.

In that sense, mathematics as we know it, is a huge set of facts and a significant amount of understanding about those facts. In its abstractness, it is consistent, and contained. It is not contaminated by the real world, where entropy and chaos dominate. It exists in a pure state, perfect, rigorous and eternal.

Knowledge and understanding then easily define our existing view of mathematics. They exist, so there is little that is inherently creative in their existence. Learning mathematics is not a creative exercise. You study the existing works, memorize the facts and work on problems until the understanding of the facts kicks in. If you are being led down the garden path, then as exciting as it is, learning about some new set of axioms and how to use them really isn't your creative spark, is it?

If learning mathematics is not particularly creative, then the big question must be one about 'finding' the ideas initially; whether or not that is a really 'creative' endeavour?


DOWN ANOTHER HOLE

To digress just a little bit. Software development is in its very own way a form of expressing a set of instructions to a computer that in its abstractness and rigour are similar to the act of creating a mathematical proof. Expression, be it about what you had for lunch, the weather or some specific algorithm for searching is a form of communication. The underlying topic may change, but choosing English, French, C or Java for your expression is still a way of communicating something. For English, it is non-rigorous and it goes to another human being. For C, it is very rigorous and it goes to a computer, but if both are about algorithms, then they are just different ways of expressing the same underlying thing. We only choose to express vastly different things in the different mediums.

There is, of course, a question about the expression of intangible quantities. It is one of those that Richard Gabriel in his book "Patterns of Software" says architect Christopher Alexander called: "the quality without a name". We have to categorize the world into those things that are determinable, tangible in some way, and those that are not. Tangible usually means, some form of physical existence. For the intangible, mostly it comes to us, personally in some what that is spiritual to some degree. By that I do not evoke any higher meaning, other than to describe 'spiritual' as a mood that is specified by humans. It is one of our many emotions. In Gabriel's book he talks about the feelings one gets when they go into these huge architectural wonders, often churches, but not always. It moves people in some intangible way. That unnameable quality then to my mind is just another intangible human emotion. Perhaps awe, or something similar.

That, fortunately makes for a nice and tidy taxonomy, since I've often described fine art -- for example -- as having an embedded emotional context, which is what separates it from merely being graphic design. If it stirs you in some way, it is art. If it is just pretty, then it is only graphic design. The same is true for music, it moves you or it is 'pop' music.

What we can perceive with our intelligence is tangible, and what we can experience with our emotions is intangible.

And so, in my earlier example with the drawing, the self portrait combines two very distinct things, one quite tangible, the representation of a human -- with the other, which is intangible -- the emotion of feeling somewhat blue. The creative 'short' or spark that drives that is not natural to most people, so although afterwards we (some of us) can see what the artist was getting at, and are moved by it. Although left in the same circumstances, we -- not being that specific artist -- would not have been creative in that same way.


MATHEMATICS

Coming back to mathematics: we know that the math is out there, and that it is abstract, yet not intangible. It is a thing, and if you search hard enough, eventually you will come upon it. I am not saying that in any way it is easy, or that it would take humanity anything less than thousands of years of culture before they might stumble on a complexity concept like negative numbers, which is exactly what happened. It also took a long time for us to figure out zero, complex numbers, calculus, etc. Some humans clearly can get there faster, but the road that needs to be traveled needs to be traveled by all, collectively.

We could not have complex branches of mathematics for instance without having an understanding of negative numbers. The intuitive leap that one and only one person can make, can only be so far. We are limited by our life spans. The imaginary of standing on shoulders of giants is exactly correct if you realize that the giants themselves are only just other people standing on other people's shoulders. And on it goes.

We get left then with the understanding that finding new forms of mathematics is a search, and it is one that comes from a limited distance from our last existing searches. With each new generation we can cover new ground, but only just barely. In that, although for one human, they learn the existing knowledge and then the existing understandings, they can only start searching from that point on. To them it is an open un-explored field with hidden surprises, to humanity however, it is just part of an ever going permutation through all of the localized space relative to the last set of solved problems. So, in the end, the best that a mathematician can do -- in that sense -- is to find or get to their discovery before the hoard of people following them do. Speed is important, but so is being in the right place at the right time, as the field is constantly filling up with others overturning the same rocks and prodding the same bushes too. If you don't discover it, they eventually will.

All of this really comes back to whether or not the idea of someone prodding around in a big field looking for interesting bits of mathematics is in fact being creative or not. And, to some very large degree, if mathematicians are creative in what they find, then by virtual of their similarity, computer scientists -- at least for the very first time they are writing something really unknown -- are equally as creative.

As if you needed more twists, we still need to try and relate this situation back to something more physical.

The simplest mapping would be to look at a graphic artist. They are by virtual of their training able to produce things that are visually appealing. Yet, as we are awash in pretty graphic design, you know that their work is not stirring in any fashion.

It looks nice, but it doesn't not give rise to emotions. That's true because while they have taken their facts, and taken their understanding -- in this case abilities and skills at things like drawing or matching pleasing colors -- they have not crossed the gulf and gone on to combine these with some intangible emotional context or any other thing that is out of the ordinary. Their skills may be beyond what some in the 17th century would have been able to do -- appearing back then to be 'art' -- but here and now, in this time they are applying the things they learned in school to a specific end. Understanding, yes, creativity, probably not.


BACK AROUND AGAIN

So going back to our mathematicians, as they ponder and search throughout the field, are they really being creative, or are they not just applying the skills thy were taught towards the goal of identifying previously unknown bits of mathematics? There may, I admit be some truly creative spark there that helps particularly gifted mathematicians to get to a specific point in the field faster than others and allows them to identify their thing rapidly. How else, can you explain Einstein? Although his leaps where related to a specific science, not general abstract mathematics, the distance that he appears to have travelled vs. his contemporaries at the time seems to be enormous. But again, what he found was there plainly for all to see, if they wanted to, or were willing to look. Relativity always was, and always will be. You just have to want to see it. But still, the leap was big and it took twenty years for his fellows to catch up with him and get to that particular place.

And so, for me, I'd have to say that most of the leaps that are going in mathematics are not leaps, but a natural progression of steps. And that these steps are an expected consequence of the knowledge and understanding that come from studying the field and really understanding it. That sometimes, but very rarely, a few make creative'ish giant leaps, but in general people are just working towards the goals of their field. If any one famous mathematician had died early, it may have taken us longer to get to their understanding, but we would still get there. The results might be slightly different, possibly composed of a different set of primitives, but it will still be functionally equivalent. As mathematicians create proofs, so too do computer programmers create code. It is exceptionally less complex, and far more mundane, but still shares the same type of relationship.

The code exists, we just have to find that sequence of instructions that expresses the solution to the problem in the most elegant fashion. So, we have to ask again: is programming really creative? The answer would have to be rarely. Only at that moment where we are combining two unrelated things in a truly unique way are we actually allowing our brains to take creative leaps. Otherwise we are just applying our understanding of how to code.

If your just doing another day's worth of problem solving, then those are exactly the skills programmer's should have and were taught. Software design has creative moments, and often the design is left unspecified so that some of the effort is required by the programmers, but mostly the business domain, technical domain and best practices remove a significant amount of the degrees of freedom. The interfaces share the same creative tones as graphic design. Most of it is really the application of an understanding.

Is software development art? Never. There is virtually no place in software for the expression of intangible emotions, I can barely imagine what that might look like. Tools are just tools. But, to slightly confuse the issue, the content of the programming could be art. That is, you could create a masterpiece of art using code as the means of expression. Games designers and many of the early pioneers of tools like flash succeeded or came close.

But really, a tool that moves you to tears is somewhat counter-productive, yes? It would be difficult to be both stirring and functional. Although, I do have to admit that some big company software products have caused me to cry, but really, that is only due to dealing with their crap, and my inability to be able to find a better replacement. The code did not stir the emotions, the angst of dealing with it did.

Controlling Development Chaos

A lot of software developers really hate any comparison or analogy that involves building construction. I think the primary reason behind this dislike is the sense that construction workers are viewed as passive players in the whole development project, in the same way that factory workers are seen as nearly-mindless cogs in a production line.

There is some reason to fear this type of comparison, intellectual pursuits are clearly not the same as physical. The rules are different. For whatever reason, we have a great deal of knowledge on how to refine and maximize the efforts of factory workers, but virtually none on how to do the same for white-collar jobs. Trying to take what we've learned from the physical and apply it to intellectual hasn't been particularly effective.

Still, in this post, I start with looking back at a construction analogy and then moving forward to a more sophisticated, and probably more popular analogy. Even if construction is not a perfect fit, building things is always a lot like building things, and we can always learn a lot from the way other people build things.


THE UNLOVED CONSTRUCTION ANALOGY

While I agree that we shouldn't draw two many parallels between software and construction, there are still things about a construction analogy that are 'truish' enough to be useful.

For instance, a large sky-scraper project contains a huge number of specialists, including carpenters, plumbers, electricians and of course general-labourers. At the same time as well are engineers. managers, inspectors and a whole host of other people with interesting roles in keeping the process, and the building from collapsing. Software projects are similar, in that they often need graphic designers, editors, GUI expects, database programmers, system administrators, managers, tool-smiths and all manner of specialists for handling any sophisticated technologies or domain issues. They are multi-disciplinary projects.

Both software development and construction involve building things and both, to some degree or another, change radically depending on the size of the thing you are building. A shed out back is not the same as a house, or an apartment building. A sky-scraper is a completely different deal. Scale has a huge influence on the size and techniques required to bring it all together. One does not easily jump from building sheds in a backyard to building apartment buildings, it is a different set of skills.

The construction of a modern sky-scraper is an amazing project that is pulled together with a degree of smoothness that software developers can only drool at. If we could build as large and sophisticated things, with the same degree of precision and timing, then we would have been able to get beyond our current hit-or-miss random guessing style of development process.

It is worth mentioning too, that while design and code are fixed to some degree, the job of an electrician on a huge site still requires a significant amount of thinking and problem solving. It is not a clear-cut mindless day at the office. While they may be working with their hands, electricians still need to think about what they are doing, the various plans and codes, and how they are going to make it work in the current context. The same is true for all of the other specialists. It takes more mental effort than just being a body, that after all is the specific job assigned to general labourers.


WHY IT FAILS

But still, even if you give all of that its due, construction just doesn't map entirely onto software development. We know this, it is a frequent discussion. Sometimes, I wonder if it really is a case of us just not wanting to accept it; we have a great deal of freedom, and it becomes hard to surrender that, even if it is for the good of the project. Still, there are strong differences.

The biggest single problem with construction is that it is a one-phase deal. You build the building, and then move in and repair it as you go. When we tried this with our waterfall ideas, we found it doesn't match reality. Code is seems, never gets done and is always rusting. Projects just don't end, and when they do, the code goes out of circulation. For software, there are a nearly infinite number of phases, ending only with the lifetime of the code.

The state of the art of building design has progressed over centuries, and mostly it is only slightly tweaked between buildings. In a sense, construction companies redo the same projects over and over again, they have no choice. This repetitiveness is the root of mastering the process. If you do it enough, eventually you'll get good at it. Contrast that with software, where each design for each system is unique and rarely learns from its predecessors. Because we can clone code, we don't want to keep reinventing it (even where in some cases, that might actually produce significantly cheaper or better code). Code builds up, but our experiences do not.

Buildings stay around for a long time. That aspect, as well as safety considerations bends the choices made towards the less risky long-term options. Saving time or money by not following the process, the code or using inferior parts may help in the short run, but because the life span is so long it opens up a lot of risk of getting caught in the future. Most buildings are built correctly. Software, on the other hand, being mostly invisible and uncontrolled doesn't provide much incentive for programmers to make long-term choices, despite the fact that software projects are always long-term projects.

Another significant different is that workers on a construction site have less degrees of freedom then computer programmers. Their roles and jobs are far more rigid. Right or wrong I don't know, but this is a key issue that I want to get back to later. We've always had our freedoms, and they have always been a problem.

For all of the differences, there is also a lot that matches, but not enough that we should try to emulate construction companies, although we should admire their skill and organization. But it is exactly that line of thinking that sent me in search of some other type of better suited analogy.


GOING BEYOND CONSTRUCTION

As often happens with me, I get a little down the road of a thought, and then it gets left behind for a while. In particular, I dropped my pondering of construction analogies so that I could do my usual Friday night routine, which was watching a film with friends. The film -- which I can't remember -- had finished, and we had moved on to the DVD special features.

As the director and actors mutually patted each other on the backs and proclaimed their love of working together, a little thought was brewing in my head. A film, you see is a large complex project, that also brings a large number of multi-disciplinary professions together in order to create something. Most people don't quite see it as the same thing, but if you look at the effort and money that was poured in a mega production like Lord of the Rings, you start to see some similarity to these huge buildings we keep throwing up in our cities. But, it is hugely different.

We deem a film as art, and the director, as an artist. We see most people involved with film as artisans. Even thought, in the end, there are also carpenters, electricians, plumbers, casting agents, cooks, boom operators, grips, computer programmers, and a huge huge host of other professionals involved.

What's interesting with a film, is that even thought the script is often written by one or more writers, and a huge number of producers and managers are involved, the films themselves have always held the particular 'stamp' of the director. In a very real sense, even with all of the 'creatives' vying their wares, the director of a modern major feature film, gets and sets the final 'vision' under which the film will be created. You can tell a lot more about a film by its director, then you can by its writers or actors. In many ways, the director of a film, is far more significant than the architect of a building. An architect shapes the design, but the engineers ensure it is built correctly. A film director often has no such constraints, at least not ones that are 'that' objective (budget excluded). They control the vision.


ANOTHER ANALOGY

But if we go back to the special features, even with all of the influence, the staring actors still often talk about how much artistic freedom they were allowed in playing their parts and contributing to the film. In that way, a good director stays true to their vision, but not necessary at the expense of making all the actors just mindless pawns on a film assembly line. Most films are collaborations of many artists, yet remain true to their director's goals.

So if I am looking for a more realistic role-model, I probably want to be more like a director of a modern film when I am leading a big development project. In that way, I want to have my vision implemented, and I want the final product to clearly bear my stamp, yet I want my fellow developers to be able to contribute to the overall project in a way that makes them feel honored and proud to have been on that particular development team. Like the actors, the coders should feel that they were able to make significant contributions, but without disrupting the overall product.

I do at this point, want to warn programmers about thinking that I am saying they should have some infinite degree of freedom while developing software. Not even the directors have total freedom. There is a difference between creative contributions and just plain old chaos. The requirements of the users, the limits of the technology, and the consistency of the conventions all combine to remove a great deal of the degrees of freedom from coding.

For tools to be useful, they must have certain specific functionality that clearly matches a problem domain. You can't creatively work your way around the actual problems, they are what they are. Our existing technological base is awful, but it is the only one we have. If you have to bend your design for a rigid relational database schema in order to allow for reporting, then there is little choice in that matter. And if every other similar application uses widgets in some annoying brain-dead, but extremely common way, you too must follow that or pay the consequences of annoying your users instead of impressing them. A good software product, one that solves a specific problem is heavily constrained long before it is even sketched on the back of a napkin.

In all, if you are implementing a targeted solution for a specific industry, there isn't an infinite array of possibilities for the development. Like a 'genre' in film making, if you break the rules, you had better understand them first, and given a solid reason for breaking them, or people just take you for someone that doesn't get it.

A film like Lord of the Rings, for example, while intensely creative is also highly restricted. Frodo can't be green, and you can't redefine elfs back into funny little forest creatures. The film would flop if it violated the spirit of the books. In the same way, software is constrained by its intended targets.


FINAL CREDITS

In many ways software is unique, but in many ways it is just another method of building things. Our biggest difference is the immaturity which lingers in our development process. We clearly do not want to grow up and start producing better software. But the issue might have been one about what our newer 'maturer' self would really look like. With terms like architect and and processes like waterfall, we've been quietly, but agonizingly following along behind various construction analogies, but with little effect toward reducing the 'software crisis' first identified in the late sixties. Is it any wonder we are resistant to change? Is there actually anything to change to that will really work?

I think if we really don't want to be factory or construction workers, then we need to look carefully at other groups of skill artisans, and start trying to fit their processes into ours. A director for a major motion picture, and his teams of 'creatives' is such a great parallel because it involves a large group of diverse people putting together a big complex work. Its main weakness is that a film is a short-term project, once completed it is over. Because of that, most choices of how to get the work done are not made for the long term. If we did that, it would be a disaster in software. Still, mixing this type of analogy with a construction based one, which is a much longer-term vision, produces something in the middle. We are neither, but we could be both.

I'm probably late with this analogy, as I suspect given the credits on the back of most video games, that at least one segment of the software industry has been quietly following this analogy all along. Still, a unified single vision is the key to bring out high-quality products, and a film director does it better than any other industry. Major motion pictures, at least according to the DVD special features, are fine examples of creative and organizational projects. We should marvel at how they work, and how they manage to bring together so many diverse disciplines into one final amazing product.

Next time someone asks me about what title I want for leading a project, it may be something like: commercial software director. Can an Oscar be that far behind?

Testing for Battleships

In one of my earlier blog entries, I casually suggested that testing was like playing a game of 'Battleship'. 

It is an odd analogy, but as I pondered it a bit more I realized that there were interesting parallels between playing Battleship and testing, many that are worth exploring. 

Often, it is exactly our change of perspective that allows us to see things with fresh eyes; even if it is tired and well-tread ground. 

AN INITIAL DEFINITION 

For those of you not entirely familiar with it, Battleship was a popular kid's game, a long long time ago, in a galaxy far far away. 

Wikipedia has a great description of it: http://en.wikipedia.org/wiki/Battleship_%28game%29 

The version I grew up with was non-electronic and consisted of two plastic cases each containing a couple of grids, the 'ships', and a huge number of little red and white pegs. 

In the first phase of the game, both players take their five 'ships' of varying sizes and place them on the horizontal grid. After that, one after the other, the players take turns calling out 'shots' by the grid coordinates. 

The coordinates are alphabetical on one axis, and numerical on the other. A shot to C5 for instance is the third position across and the fifth one down. 

If the shot hits a ship, the defending player has to say 'it is a hit'. The calling player puts a red peg in the vertical grid to mark the spot. Otherwise, just so they don't call the same spot again, a white peg is used. 

The game continues until one player has sunk all of the other players 'ships'. It is a race between who sinks who's fleet first. 

The big strategy of the game is to find ways to 'hide' the ships. For instance, lining them up against each other, so it is hard to see where one starts and one ends. Sometimes hiding them on the outskirts of the grid is great, sometimes not. Once the game gets going, it becomes a contest between the two players as to who figures out all of the other locations first. 

Mostly, the game ends long before all of the pegs have been entered into the board. From my rather random and fuzzy memory, I'd say usually around two thirds of the board was full at the end. Sometimes more, sometimes less. 

OBSERVATIONS 

What does this have to do with testing? 

Well, interestingly enough, testing is kinda like playing half a game of Battleship against the code. You can see all of the functionality in the system as a giant grid, a vast sea that contains a number of hidden bugs. 

The tester's job is to place tests (pegs) at various points in the grid to see if there is a bug there or not. Each test -- or part of the test -- acts as a little peg that reveals a possible bug. For a test suite, one after another, the testers keep adding pegs until all of the the testing is complete. 

Battleship ends when one player has sunk all of the other player's ships. Testing ends, when the time is up or you've run out of little pegs. It is also a race condition, but usually against time, resources, or patience. 

In the real game, the 'ships' come in multiple sizes that are well known. Bugs on the other hand come individually or in clumps and there is very little rational behind either. In fact, there is a totally unknown number hidden in the grid that can be anywhere from zero to almost NxN bugs. 

Now, most people might think testing occurs with a huge number of pegs, but if you really look at the coverage on a line-by-line, input-by-input basis, the grid is absolutely huge, while the pegs are few. 

Even if you are going to push the whole development team into an intense round of final system-wide super integration testing, it is still a pitifully small number of pegs verses the actual size of the grid. One big different between the game and real testing is that the grid grows with each release. It keeps getting larger all of the time. The game is ever changing. Even when you think you have mastered it, it will still occasionally catch you by surprise. 

Although for the analogy I picture the functionality as an N by N grid, the real topology of any system, depending on representation, is some multi-dimensional arbitrary structure, where the sea-scape itself holds many clues into the increased presence or likelihood of bugs. 

What I really like about this analogy is that it gets across the point that testing is hit-or-miss, and by definition not everything is tested. People tend to view testing as a deterministic process; if you do enough testing, then you will catch all of the bugs. In fact, I've seen 'intensely' tested code, that has run in production for years and years, still yield up a bug or two. 

There are always a huge number of bugs and they are always buried deep. Unless a peg hits them directly with the right input, the presence of the bug is unknown. 

TESTING STRATEGIES 

Over the years I've seen many 'interesting' strategies for testing. 

Sometimes it is all manual, sometimes it is heavily automated. Some testing focuses on catching new problems, so of it is more bias towards regression. 

Generally, though the most common thing about testing is that most of it is done on the easier bits. The hard stuff rarely gets touched, even though it is the most likely source of bugs. 

In that sense, developers quickly use up all their pegs clumped together in a close location on the grid, while ignoring the rest of the board. They tend to put the most pegs into the spots where the least bugs will exist. 

Unlike the game, in testing the board itself offers tonnes of clues on how to test more effectively, but most developers prefer to disassociate themselves from that knowledge. As if, somehow, if we don't get smart about what we test, then the testing will be more thorough, which would be a possibly workable strategy only if you have a huge mountain of pegs. If you have way too few pegs, it's effect is probably the opposite. 

Software developers hate testing, so it is usually handled very badly. Most big integration tests for example are just replications of unit-tests, done that way to save time. An oddly ineffectual use of time, if the tests both refer back to the same position on the grid. 

Underneath, some bugs may have moved around but we still need to test for them in their final position. Sticking the first peg in, then removing it, then adding it again is unnecessary effort. 

Interestingly enough is that for whole sections of the board, if there haven't been any changes to the code, then the same bugs remain hidden in the same positions. So executing the same peg pattern, or regression testing, will produce the exact same results as last time: nothing. 

No changes to the code + no changes to the tests = no new bugs, but also a huge waste of time. 

If you get the sense of this game, then spending a great deal of time sticking the pegs into the boards only to not find any bugs seems as though it may have been a waste of time. Programmer's like to test to 'confirm' the absence of bugs, but really for most code, particularly if it is new -- less than twenty years old -- not finding bugs more likely means the testing is ineffective, not that the code is good. It ain't. So that one last pass through the tests, with the expectation of not finding anything wrong is really a big waste of time. 

You regression test to make sure none of the older bugs have reappeared, but that only makes sense if the code was actually effected. Retesting the same code multiple times, even when it hasn't changed is a standard practice, but still crazy. 

Clearly this also implies that any second round of testing should be based on the result of any changes, and a first round of testing. Parts of the board are not worth retesting, and it is important to know which parts of the board are which to avoid wasting resources. 

PATTERNS 

I can remember when I was a kid, I'd come up with 'patterns' to try and optimize my peg placement for Battleship. You know, like hitting every other spot, or doing a giant X or cross or something. I wanted a secret formula for always winning. These never worked, and I would generally be pushed back to messing up the pattern because of the nature of the game. 

Once you've identified a red peg, you want to finish off the ship quickly. Then you get back to searching. Overall the best strategy was to randomly subdivide as much as possible. Big gaps allow big things to hide. Random shots, distributed more or less evenly across the board tended to be more fruitful. 

One thing for certain about Battleship was that no matter who you were playing against, there was some underlying structure to the way they placed the ships. 

As the game progressed, the strategy for finding the next ships was based on the success of the current strategy. None of the fixed patterns worked because they were too static. Although, no one would actually be able to explain why they placed the pieces the way they did -- to them it was random -- it just wasn't. 

Different opponents tended towards different strategies, which after a game or two became fairly obvious. In a weird, but rather indirect way, the same is true for code. The bugs in the system are not really random, they come from an extremely complex pattern that will always seem random to human beings, but there is some ordered structure. 

The most effective ways to find them are by using instinct to work through the parts of the system that are most likely to have problems, or to insure that their aren't problems where they might be really devastating. On fixing a bug, retesting the entire system is a huge waste of resources. It isn't hard in most systems to narrow the board down to a smaller one that might have been effected by the change in code. 

Finding the true 'impact' of a change may seem expensive, but it is considerably cheaper than retesting everything. In that regard, a well-crafted architecture, one that isn't just a big ball of mud certainly helps in drawing lines around the possible effects of a software change. It makes understanding the impact of a change a lot easier. Obviously, if you can reduce the size of the board, you increase the chances of finding any other hidden ships on the field, particularly if you didn't significantly decrease the number of pegs. 

SUMMARY 

There are lots of things we can learn from studying an analogy. They allow us to morph the problem into some other perspective, and then explore the relationships and consequences in some other way. 

Pushing testing along side a board-game shows the importance of being responsive in the testing process, and not just trying to issue fixed patterns across a huge space. A static approach to testing fails repetitively and is more expensive. A dynamic approach is a more likely way to utilize the available resources. 

Clearly shrinking the board but not heavily reducing the pegs will increase the likelihood of finding problems. Also, not just retesting all of the old spots will help to find new hidden dangers. 

Understanding the topography of the functionality will help in high-light areas that need to be specifically addressed. There are lots of strategies for making better use of the resources. If you see the life of a product as a long series of games against the code, you get an awful large number of chances to test for underlying problems. 

If you squander that on a massive stack of simple-but-useless regression tests that cover the same unchanging territory again and again, it is no wonder that the product keeps getting released with a significantly large number of problems. Time and resources are not making headway into the real underlying territory. If you craft tests as absolute sets of instructions, then recrafting them for new games is hard and a huge amount of work. 

Since we hate testing, we shy away from any work in that area. As such, even projects that do everything 'correctly' and have the 'proper' suites of regression tests and always test 'before' each and every release, let large qualities of bugs through. In fact, the ones that followed the established best practices are probably of the average or worse quality because the programmers tend to shift their minds into neutral during the testing phase. It is the guys with no resources that are anxious about not wasting what little effort they have, that are getting the most bang for their buck.