My brain uses an efficient algorithm for garbage collection. Sometimes, it can be a little too efficient. I'll be in the middle of using some fact and then "poof" it is gone.
"Gee, thanx brain! Now I'll have to ask for that phone number again, ... for the third time". Wetware can be such a pain.
Besides the obvious disadvantages of having such quick fact cleanup there are in fact a few bright sides to limited short-term retention. One important side effect is that I am constantly forced to go back to first principles, mostly because I've forgotten everything else. Re-examining the same ground over and over breeds familiarity. You not only come to understand it, but you also come to simplify that understanding; one of the harder things to do well. Because I can remember so little, everything must be packed as tightly as possible or else it is forgotten.
That perspective is interesting if I used it to examine something such as software development. We always build up our knowledge overtime, sometimes to the degree where we make things more complex than necessary. Sorting back through our understanding is an important exercise.
Building software can be a very complicated pursuit, but underneath creating software is actually fairly simple. We are doing nothing more than just building tools to manipulate data. In spit of all of the fancy features and appearances, when you get right down to it, the only things a computer can actually do are view, add, delete and edit data.
Playing music on a portable mp3 player is a good example. The computer copies the data from its internal storage over to some circuitry that generates the currents for the sounds. We hear the currents vibrating through the speakers, but the computer's actual responsibility stopped just after it copied over the information. When it is no longer just a collection of binary bits, it is no longer a computer that is doing the work.
We can go deeper. When you get right down to it, all software can be seen as nothing more than just algorithms and data working together within some framework. At the higher level we perceive this as software applications with various functionality, but that is an illusion. Software is deceptively simple.
We can take advantage of this simplicity when we build systems. Orienting our understanding of what we build based on the data it manipulates simplifies the overall design for a system. The data is the key. There can be a great deal of functionality, but it is always restricted by the underlying range of available data.You have to get the data into the system first, before you can start manipulating it.
One key problem with badly written software code is the amount of repeated data in the system. Big balls of mud and poor architectures are identified by the redundant copying and translating of the same data around the system in a huge varieties of ways. This not only wastes CPU and memory, but it also makes understanding and enhancing the system difficult. A sense of complexity can be measured from the volume of redundant code within a system. If, with enough time you could delete half the code, yet keep the same functionality you have a system in need of some serious help.
Understanding the importance of data not only effects the code, but it translates all of the way up to the architecture itself. The blueprints that define any system need only consist of outlining the data, algorithms and interfaces within the system. Nothing more.
In most cases, the majority of the algorithms in system are well defined and come from text books. Probably less than 20% require anything other than just a brief reference, such as "that list is 'sorted'".
Interfaces are frequently the worst handled part of modern software development, generally because only the key ones are documented. The rest are just slapped together.
Anywhere that people interact with the computer should be considered an interface, which not only includes GUIs and command-lines, but also configuration files, libraries and your database schema. Rampant inconsistency is a huge problem these days. Consistency at all levels for an interface reduces its size by at least an order of magnitude. Most interfaces would be just as expressive, at less than a quarter of their size.
For large projects the most natural lines for subdividing the work are the natural boundaries of the data. Those boundary lines within the system -- either horizontal (components) or vertical (layers) -- themselves should be considered interfaces. Specifying them as part of the architecture ensures that the overall coherency in the system remains intact.
Within parts of the system there may also be boundary lines to delineate different functionally independent components. This helps for iterative development as well as supporting team efforts. If the lines are clear, it is easy to pick a specific set of functionality, such as a resource manager, and replace it with another more advanced one. Growing the system over a series of iterations reduces a large amount of development risk at the cost of a relatively small amount of extra work. When compared to going completely off course and having to start over, iterative development is extremely cost effective.
Orientating the development around the data not only makes the code cleaner and more advanced; it also provides the lines to break down and encapsulate the components. But, even more critically, it makes analyzing the underlying problem domain for the tool much simpler as well.
When you understand that the users need a tool to manipulate only specific subsets of data in very specific ways, that drives the way you analyze the problem. You don't get caught worrying about issues that are outside the boundary of the available data. They are outside of the software tool's scope.
In looking at the users, you see the limited amount of data they have available. They want to do things with their data, but only a limited number of those things makes sense and are practical. Technology problems become so much simpler when seen this way. If you can't acquire the data to support the functionality, you could never implement that code. You don't need to get lost in a sea of things that are impossible.
Getting back to data. The great fear with programmers specifying their systems before they write them is that they feel this will make their jobs trivial. Some think that they can stumble around in the dark, and eventually get to the right system. Other just want to build fast and light in the hopes that over time they find the right tools.
Truthfully, you cannot solve a complex problem without first thinking deeply about it. The fruits of your thinking are the design of the system. If you don't want to waste your time, you need to have worked through all of the critical issues first. If you don't want to waste your thinking, you need to get that information down into a design.
An endlessly large list of requirements and functionality isn't a coherent design.
A decent architecture of data, algorithms and interfaces shouldn't be a massive document. In fact, for simple systems, it might even be just a few pages. You can refer to standards and conventions to save time in re-specifying facts which are well known. Nobody reads the obvious, so why waste time in writing it.
Such a design, if it breaks the system down into clear parts with well-known lines, doesn't remove the joy of programming. Instead it instills confidence in the initial labors, and confidence that the project will be able to succeed with few surprises.
Instead of making programming a drudgery, it actually makes it more fun and less painful. I've learned this the hard way many a time, but thanks to my efficient garbage collection, I've also forgotten it a few times already.
The simplicity underlying software is the key to being able to build increasingly complex systems. Too often we get lost in the little details without being able to see the overall structure. Most developers still oriented their understanding of software around its implemented functionality, but this just illuminates the inner workings in its most complex fashion. The data for most systems is simple and it absolutely bounds the code, so orienting your perspective on it simplifies your issues and broadens your overall understanding. We really don't want to make our lives more complicated than they need to be; that threatens our ability to be successful. I've learned that with my limited memory the only real problem I can afford to face in a huge system is the complexity of the system itself. To keep it manageable I need to continually simplify it. To keep it simple I often need to go back to first principles and find the right perspective. It is amazing how much of the complexity can just fall away with the right vantage point.
Monday, October 22, 2007
Saturday, October 13, 2007
Wasted my Time
No doubt it is my own foolishness. My expectations are too high.
When I blog, I use Yahoo Notepad to write the draft entries. That gives me portability, and access to the draft articles wherever I am. The idea that I can access my writing from literally anywhere in the world is amazing. Something wonderful provided for me by the computer.
I was struggling with a particularly hard to write piece, where I was trying to take the reader backwards towards using first principles. Not easy to begin with, and even harder for a novice writer like myself.
I was happy with what I had done. I probably would have posted it.
After agonizing for a couple of hours, I hit the update button. Something I've done a million times before. This time though, because I had been logged in for so long, Yahoo Notepad decided to ask me to confirm my userid. The session had timed-out. Ok. No worries. This problem has caught me in the past, and in Firefox if I hit the back button, I can get back to the original screen with the textbox, which from there I can copy and paste into a text file, and use it to update the note after I've logged in again.
I could test it, but if I remember correctly, in Yahoo Notepad, after the login screen, the data in the textbox is lost. It goes through as a post, I think, and cannot get passed back into the next query, so even if the program continues from the point it left off, the right data will not be updated, and possibly the file would end up being blanked if they tried. I could be wrong about this, hitting the back button and copying out the data worked well in the past, I seem to remember. The myths we build up about our software is often interesting in themselves.
This time, when I hit the back button to my horror the contents of the textbox where the original contents, not my updated version. My notepad file is quite large. It contains a lot of draft material, so I suspect that the size of the textbox forces Firefox not to save the data. So, going back won't work. I went forward too, but that didn't seem to work either. Although by the time I tried that, the panic had already set in. All of my labor, all of my work was gone!? Vanished. Kaput. Ahhhhhh.
When the anger subsided, and I stop swearing and pounding on things I really starting thinking about it.
In the early days of PCs, in Word Perfect we developed a disciple where we hit save file function every 10 minutes or so. It was necessary because the machines of the day were so flaky, that you were always running the risk of losing work. 10 minute intervals minimized the amount of lost work.
In those days, our technology was new and had problems. If someone lost their work, we told them it was their fault. They clearly weren't saving enough. We pushed the obvious problems with the technology back onto the users. Later a big advancement was to performed this neuroses automatically. The computer could take care of constantly saving the file for you.
A while back, Microsoft Word crashed, and I lost a couple of hours worth of work. I though it had been doing the ten minute backup thing, but that option was turned off. Not sure why, could have been a local thing or an network share problem, or something equally uninteresting.
Hardware crashes are not as common anymore, although it might just be bad luck, but I've had more problems with hardware in the last couple of years than I have for the prior decade. At work, for instance I've had four machines die in less than a year. There seems to be a noticeable decline in the dependency of modern hardware.
I'd figured Word was set up correctly, so I never bothered to check it myself. It wasn't. But I should have checked, they said. It was pushed back to me as the user, to have notice that Word was no longer creating backups. This brought on a twisted sense of deja vu and a 'sad realization'.
A computer is an extremely powerful tool; it can do so much. It has been around for decades and can automate a huge number of tasks.
It is sophisticate enough that we could build a system that could not lose any of your work beyond a few seconds, right up to and including a major catastrophic event like and asteroid colliding with earth. Since we don't have a backup planet, there is little we could do about that, but we do have technology designed to withstand a nuclear war, major storms, earth quakes and we do know how to create fault tolerance redundant distributed implementations.
Disk storage is cheap enough that we can keep all of the key strokes we've ever entered into the computer for our entire lives; allowing us to rebuild any or all of our labors.
Our technology should not allow us to ever lose work. We have progressed beyond that point.
Computers have a huge amount of potential, unknown to even the most savvy of us, but we have not been able to tap it. We write the same stuff over and over, repeating the same mistakes and never really learning from the past. At its best, a web application is possibly the worst ever technology invented. One giant mess of poorly implemented ideas and political maneuvering. A great big ball of mud. We have at the foundation, a great technology in the Internet, that we've build a huge mess on top, the World Wide Web. We connect to it with our easily modifiable PCs, that are complete untrustworthy and getting worse with each new release. Web applications are improving, but they are still extremely crude. And very disappointing. One step forward, and one step back.
But our problems aren't technological, they aren't something we need to invent or fix. We have all of the technology we need, all of the understanding. Our problems lie with us, the people building the technology, with our culture, and our need to be free, inconsistent and messy, and with the way we organize ourselves.
For decades now we have approached the same problems in the same manner, and consistently the results have been flawed, yet we do it again and again. I really don't think it is difficult to build working systems, but it does seem extremely difficult to organize people in a way that they can build working systems.
We have the technology, if only we could figure out a way to utilize it to our advantage. Until then, I guess I'll just have to lower my expectations and get used to losing my work every so often. It is only a couple of hours...
When I blog, I use Yahoo Notepad to write the draft entries. That gives me portability, and access to the draft articles wherever I am. The idea that I can access my writing from literally anywhere in the world is amazing. Something wonderful provided for me by the computer.
I was struggling with a particularly hard to write piece, where I was trying to take the reader backwards towards using first principles. Not easy to begin with, and even harder for a novice writer like myself.
I was happy with what I had done. I probably would have posted it.
After agonizing for a couple of hours, I hit the update button. Something I've done a million times before. This time though, because I had been logged in for so long, Yahoo Notepad decided to ask me to confirm my userid. The session had timed-out. Ok. No worries. This problem has caught me in the past, and in Firefox if I hit the back button, I can get back to the original screen with the textbox, which from there I can copy and paste into a text file, and use it to update the note after I've logged in again.
I could test it, but if I remember correctly, in Yahoo Notepad, after the login screen, the data in the textbox is lost. It goes through as a post, I think, and cannot get passed back into the next query, so even if the program continues from the point it left off, the right data will not be updated, and possibly the file would end up being blanked if they tried. I could be wrong about this, hitting the back button and copying out the data worked well in the past, I seem to remember. The myths we build up about our software is often interesting in themselves.
This time, when I hit the back button to my horror the contents of the textbox where the original contents, not my updated version. My notepad file is quite large. It contains a lot of draft material, so I suspect that the size of the textbox forces Firefox not to save the data. So, going back won't work. I went forward too, but that didn't seem to work either. Although by the time I tried that, the panic had already set in. All of my labor, all of my work was gone!? Vanished. Kaput. Ahhhhhh.
When the anger subsided, and I stop swearing and pounding on things I really starting thinking about it.
In the early days of PCs, in Word Perfect we developed a disciple where we hit save file function every 10 minutes or so. It was necessary because the machines of the day were so flaky, that you were always running the risk of losing work. 10 minute intervals minimized the amount of lost work.
In those days, our technology was new and had problems. If someone lost their work, we told them it was their fault. They clearly weren't saving enough. We pushed the obvious problems with the technology back onto the users. Later a big advancement was to performed this neuroses automatically. The computer could take care of constantly saving the file for you.
A while back, Microsoft Word crashed, and I lost a couple of hours worth of work. I though it had been doing the ten minute backup thing, but that option was turned off. Not sure why, could have been a local thing or an network share problem, or something equally uninteresting.
Hardware crashes are not as common anymore, although it might just be bad luck, but I've had more problems with hardware in the last couple of years than I have for the prior decade. At work, for instance I've had four machines die in less than a year. There seems to be a noticeable decline in the dependency of modern hardware.
I'd figured Word was set up correctly, so I never bothered to check it myself. It wasn't. But I should have checked, they said. It was pushed back to me as the user, to have notice that Word was no longer creating backups. This brought on a twisted sense of deja vu and a 'sad realization'.
A computer is an extremely powerful tool; it can do so much. It has been around for decades and can automate a huge number of tasks.
It is sophisticate enough that we could build a system that could not lose any of your work beyond a few seconds, right up to and including a major catastrophic event like and asteroid colliding with earth. Since we don't have a backup planet, there is little we could do about that, but we do have technology designed to withstand a nuclear war, major storms, earth quakes and we do know how to create fault tolerance redundant distributed implementations.
Disk storage is cheap enough that we can keep all of the key strokes we've ever entered into the computer for our entire lives; allowing us to rebuild any or all of our labors.
Our technology should not allow us to ever lose work. We have progressed beyond that point.
Computers have a huge amount of potential, unknown to even the most savvy of us, but we have not been able to tap it. We write the same stuff over and over, repeating the same mistakes and never really learning from the past. At its best, a web application is possibly the worst ever technology invented. One giant mess of poorly implemented ideas and political maneuvering. A great big ball of mud. We have at the foundation, a great technology in the Internet, that we've build a huge mess on top, the World Wide Web. We connect to it with our easily modifiable PCs, that are complete untrustworthy and getting worse with each new release. Web applications are improving, but they are still extremely crude. And very disappointing. One step forward, and one step back.
But our problems aren't technological, they aren't something we need to invent or fix. We have all of the technology we need, all of the understanding. Our problems lie with us, the people building the technology, with our culture, and our need to be free, inconsistent and messy, and with the way we organize ourselves.
For decades now we have approached the same problems in the same manner, and consistently the results have been flawed, yet we do it again and again. I really don't think it is difficult to build working systems, but it does seem extremely difficult to organize people in a way that they can build working systems.
We have the technology, if only we could figure out a way to utilize it to our advantage. Until then, I guess I'll just have to lower my expectations and get used to losing my work every so often. It is only a couple of hours...
Wednesday, October 3, 2007
Lost in Processing
Sometimes it is like a long treacherous march through an endlessly dark and damp jungle. Whenever you think your nearing the end, another obstacle pops up to block your way. Behind every corner is only more pain, suffering and another corner. Every step of the way leaves you wondering if this time, it isn't really a death march. Will it end; can it go on forever?
How many software development efforts feel like an aimless trek through an untamed forest? A haphazard stumble through hostile terrain?
The goal when we start development is to try and find the shortest path towards getting it completed.
Wondering around aimlessly only increases the stress and makes it more difficult. Staying on course during development comes first from knowing the path, but also from following it. A map through the jungle is obviously a prerequisite. But that will only get you part of the way there, the second half and often more dangerous part of the journey comes from testing.
Most developers share the same fears: that initial one about being able to get it all going, and that final one about bringing it all together. Sometimes, in a particularly rough implementation, the last few months can feel like you are stuffing straw into one side of a burlap sack, only to watch it fall out the other side, again, and again, and again. If you haven't already lost it, by the time you get around to testing your probably pretty close. The jungle feels like it is closing in on you.
You might be a little frazzled going into testing, but you shouldn't let that be an excuse for turning off your brain. Testing, you see, is often were we lose the most, and gain the least. Development may seem to drag on endlessly, but testing is when you get totally lost.
It is worse too, when you widen your definition of 'flaws'; bugs you see, aren't just functions that don't quite work as expected, they also include spelling mistakes, grammar problems, awkward interfaces, ugly colors, poor documentation and anything else that ultimately distracts the users from an otherwise perfect system.
Often we try to narrow the definition as we get farther down the road, but that doesn't really help.
With an expanded view towards problems, if you really counted all of the flaws and inconsistencies for most software, the bug list would be huge. Yet, for the majority of bugs, the fixes don't take long to correct. For whatever reason -- most often fatigue -- we choose to turn a blind eye towards the ugly bits at the end of the development, even when they are simple to fix. Fear of making it worse is usually the culprit.
The standard problem with testing is inconsistency. The test cases get attention late in the game, and they always over-test the easy sections of the code, while ignoring the complicated parts. To be honest, most testing ranges from poor to totally ineffective. And most developers don't want to admit it.
Trouble starts early with the writing of the test plans.
It is commonly known that programmers make horrible testers. They test what they find interesting, which is usually the parts of the code they did well. They skip over what they dislike, mostly the sections of code with the higher bug counts.
This results in spending more time looking for bugs in the places they are least likely to be. If, as programmers we continually fail at testing, they by what madness does it make sense for us to write the test cases and test plans? We're only marginally better at thinking up cases for our own work, then we are at actually executing them properly.
Too add it it, some programmer's feel that uncovering a bug is a bad thing.
They want the program to get a bug free pass through in testing. They'll even pre-check the code, using up lots of time, and check all of the tests just to insure that there are no bugs. Fear of fixing bugs in the end seems to drive this type of behavior. If it were that the code was perfect, why bother to test it? Usually they burn through too much time in the simple sections trying to be perfect.
By the time testers execute the plans, they are ineffective because the underlying problems have been fixed, or because they really don't test anything significant. Sometimes, the same test suites will even be executed multiple times in a row, as if that increases the odds of finding anything. Hardly effective.
Testing is a necessary part of the process, so we want to maximize the time spent to get the biggest effect.
In that sense, we actually want to find bugs, particularly if we shortcut the work earlier to save time for other pieces of work. We don't want to find too many bugs, and we don't want to be forced back into another round of development, but we do want to maximize the effort. Finding bugs, after all shows that testing is actually working. Not finding bugs could either be interpreted as a good or a bad thing. More often its a bad thing.
I was once called a defeatist for suggesting that bugs will always exist in all code, but accepting it makes it more likely that you'll handle the bugs better. There is nothing you can do to stop bugs. Accepting that some flaws will get into the release makes it possible to handle them rationally.
If you don't plan on fixing bugs after the final release you are being too optimistic. In all my years of developing, the best I ever did was a four or five year period of just having just one bug in a complex production system. The secret behind this unusual success was lots of time; probably too much of it. Time that isn't generally available for most development projects.
Accepting that there are some arbitrary number of bugs left, helps to utilize the limited resources. It becomes a challenge to find them as quickly as you can with the least amount of effort. Even in a well tested system, a decent round of testing might reveal five or six. Digging deeper may reveal a couple more, but there will still be a couple of them that are shipped. We don't have an infinitely large amount of time, so bugs will exist. Knowing that, the only question is: which ones are the most acceptable?
Nothing is worse, then a stupid bug showing up in an obvious place, so if only for appearances sake, we always want to make sure that the main user pathways through the system are bug free. We can broaden the search out from there depending on the time and quality constraints.
When we do find a bug, there are always two choices, we can accept it, or we can fix it. Accepting it is generally cheaper because fixing it can reveal more bugs and require more testing. However, if you can precisely gauge the impact of the fix then you can retest only the smallest possible area; which is always cheaper than retesting everything. The impact of a fix in a well written program is not hard to determine.
Getting back to that previous project were we had virtually no bugs: what made it work so well was that the testers were two rather bored programmers from another project. They took up the challenge of finding the flaws in the system, and they did so with enthusiasm. The type of enthusiasm that the original programmers -- after months and months of pounding out code as fast as they can -- no longer had the ability to show.
The secret to great testing that I learned from that experience lies in handing off the testing to someone with fresh eyes. To really leverage independence, the testers themselves need to work out the test suite from first principles. They must do it directly from the design documentation because the programmers are tainted. The testers should be able to make certain assumptions about the type of system that fits into the documented problems and solutions. Not only does it supply good system testing, but it also validates that the design matches the system.
The time for independent testing is marginally longer, but if the results come back as several different sets of issues, the value for this type of assessment is worth it. Development can fix some issues, and schedule others for later. The results of different tests can be set back in several passes, allowing the developers to gauge the impact and make the changes early for specific types of problems; effectivily layering the test results. Testing could start with the deeper functionality and work its way out, taking care to insure that the order does not cause significant retesting.
There are various different types of tests that can be applied in this way, ranging in depth and effectiveness verses cost.
Rough testing could, for instance be applied in the middle of development to insure that the current direction is reasonable, and that the code is mostly usable. Walk-throughs could check for interface consistency and ease of use. In a large GUI program, specific pathways could be tested independently of each other. Sections of the system could receive extra testing because they are mission critical, or hard to recover from.
Interaction between the test teams and the developers can occur to validate some of the test plans and possibly later to help explain some of the problems, but it should be left to a minimum. The testers have an expectation that the system will behave in a specific way, it is there job to document any deviation. The developers need to assess the deviations and decide the underlying cause. If the testing is thorough, not everything found will be dealt with.
Wondering around aimlessly in a jungle is never fun. Knowing how long you have to suffer can help make it seem less painful. Still, even the shortest path for development can be a long journey. We need to be wise about how we spend our time. After a long and painful first half for the development, the programmers are tired and not functioning too well. Traversing the second half with a fresh team of testers -- while giving the developers a bit of breathing time -- increases the likelihood that the project will arrive straight at its destination. Where we know we continually have problems, we need to find solutions that fix the original issues without causing worse ones as a side effect. That puts us on the shortest path.
How many software development efforts feel like an aimless trek through an untamed forest? A haphazard stumble through hostile terrain?
The goal when we start development is to try and find the shortest path towards getting it completed.
Wondering around aimlessly only increases the stress and makes it more difficult. Staying on course during development comes first from knowing the path, but also from following it. A map through the jungle is obviously a prerequisite. But that will only get you part of the way there, the second half and often more dangerous part of the journey comes from testing.
Most developers share the same fears: that initial one about being able to get it all going, and that final one about bringing it all together. Sometimes, in a particularly rough implementation, the last few months can feel like you are stuffing straw into one side of a burlap sack, only to watch it fall out the other side, again, and again, and again. If you haven't already lost it, by the time you get around to testing your probably pretty close. The jungle feels like it is closing in on you.
You might be a little frazzled going into testing, but you shouldn't let that be an excuse for turning off your brain. Testing, you see, is often were we lose the most, and gain the least. Development may seem to drag on endlessly, but testing is when you get totally lost.
It is worse too, when you widen your definition of 'flaws'; bugs you see, aren't just functions that don't quite work as expected, they also include spelling mistakes, grammar problems, awkward interfaces, ugly colors, poor documentation and anything else that ultimately distracts the users from an otherwise perfect system.
Often we try to narrow the definition as we get farther down the road, but that doesn't really help.
With an expanded view towards problems, if you really counted all of the flaws and inconsistencies for most software, the bug list would be huge. Yet, for the majority of bugs, the fixes don't take long to correct. For whatever reason -- most often fatigue -- we choose to turn a blind eye towards the ugly bits at the end of the development, even when they are simple to fix. Fear of making it worse is usually the culprit.
The standard problem with testing is inconsistency. The test cases get attention late in the game, and they always over-test the easy sections of the code, while ignoring the complicated parts. To be honest, most testing ranges from poor to totally ineffective. And most developers don't want to admit it.
Trouble starts early with the writing of the test plans.
It is commonly known that programmers make horrible testers. They test what they find interesting, which is usually the parts of the code they did well. They skip over what they dislike, mostly the sections of code with the higher bug counts.
This results in spending more time looking for bugs in the places they are least likely to be. If, as programmers we continually fail at testing, they by what madness does it make sense for us to write the test cases and test plans? We're only marginally better at thinking up cases for our own work, then we are at actually executing them properly.
Too add it it, some programmer's feel that uncovering a bug is a bad thing.
They want the program to get a bug free pass through in testing. They'll even pre-check the code, using up lots of time, and check all of the tests just to insure that there are no bugs. Fear of fixing bugs in the end seems to drive this type of behavior. If it were that the code was perfect, why bother to test it? Usually they burn through too much time in the simple sections trying to be perfect.
By the time testers execute the plans, they are ineffective because the underlying problems have been fixed, or because they really don't test anything significant. Sometimes, the same test suites will even be executed multiple times in a row, as if that increases the odds of finding anything. Hardly effective.
Testing is a necessary part of the process, so we want to maximize the time spent to get the biggest effect.
In that sense, we actually want to find bugs, particularly if we shortcut the work earlier to save time for other pieces of work. We don't want to find too many bugs, and we don't want to be forced back into another round of development, but we do want to maximize the effort. Finding bugs, after all shows that testing is actually working. Not finding bugs could either be interpreted as a good or a bad thing. More often its a bad thing.
I was once called a defeatist for suggesting that bugs will always exist in all code, but accepting it makes it more likely that you'll handle the bugs better. There is nothing you can do to stop bugs. Accepting that some flaws will get into the release makes it possible to handle them rationally.
If you don't plan on fixing bugs after the final release you are being too optimistic. In all my years of developing, the best I ever did was a four or five year period of just having just one bug in a complex production system. The secret behind this unusual success was lots of time; probably too much of it. Time that isn't generally available for most development projects.
Accepting that there are some arbitrary number of bugs left, helps to utilize the limited resources. It becomes a challenge to find them as quickly as you can with the least amount of effort. Even in a well tested system, a decent round of testing might reveal five or six. Digging deeper may reveal a couple more, but there will still be a couple of them that are shipped. We don't have an infinitely large amount of time, so bugs will exist. Knowing that, the only question is: which ones are the most acceptable?
Nothing is worse, then a stupid bug showing up in an obvious place, so if only for appearances sake, we always want to make sure that the main user pathways through the system are bug free. We can broaden the search out from there depending on the time and quality constraints.
When we do find a bug, there are always two choices, we can accept it, or we can fix it. Accepting it is generally cheaper because fixing it can reveal more bugs and require more testing. However, if you can precisely gauge the impact of the fix then you can retest only the smallest possible area; which is always cheaper than retesting everything. The impact of a fix in a well written program is not hard to determine.
Getting back to that previous project were we had virtually no bugs: what made it work so well was that the testers were two rather bored programmers from another project. They took up the challenge of finding the flaws in the system, and they did so with enthusiasm. The type of enthusiasm that the original programmers -- after months and months of pounding out code as fast as they can -- no longer had the ability to show.
The secret to great testing that I learned from that experience lies in handing off the testing to someone with fresh eyes. To really leverage independence, the testers themselves need to work out the test suite from first principles. They must do it directly from the design documentation because the programmers are tainted. The testers should be able to make certain assumptions about the type of system that fits into the documented problems and solutions. Not only does it supply good system testing, but it also validates that the design matches the system.
The time for independent testing is marginally longer, but if the results come back as several different sets of issues, the value for this type of assessment is worth it. Development can fix some issues, and schedule others for later. The results of different tests can be set back in several passes, allowing the developers to gauge the impact and make the changes early for specific types of problems; effectivily layering the test results. Testing could start with the deeper functionality and work its way out, taking care to insure that the order does not cause significant retesting.
There are various different types of tests that can be applied in this way, ranging in depth and effectiveness verses cost.
Rough testing could, for instance be applied in the middle of development to insure that the current direction is reasonable, and that the code is mostly usable. Walk-throughs could check for interface consistency and ease of use. In a large GUI program, specific pathways could be tested independently of each other. Sections of the system could receive extra testing because they are mission critical, or hard to recover from.
Interaction between the test teams and the developers can occur to validate some of the test plans and possibly later to help explain some of the problems, but it should be left to a minimum. The testers have an expectation that the system will behave in a specific way, it is there job to document any deviation. The developers need to assess the deviations and decide the underlying cause. If the testing is thorough, not everything found will be dealt with.
Wondering around aimlessly in a jungle is never fun. Knowing how long you have to suffer can help make it seem less painful. Still, even the shortest path for development can be a long journey. We need to be wise about how we spend our time. After a long and painful first half for the development, the programmers are tired and not functioning too well. Traversing the second half with a fresh team of testers -- while giving the developers a bit of breathing time -- increases the likelihood that the project will arrive straight at its destination. Where we know we continually have problems, we need to find solutions that fix the original issues without causing worse ones as a side effect. That puts us on the shortest path.
Subscribe to:
Posts (Atom)