Wasted my Time

No doubt it is my own foolishness. My expectations are too high.

When I blog, I use Yahoo Notepad to write the draft entries. That gives me portability, and access to the draft articles wherever I am. The idea that I can access my writing from literally anywhere in the world is amazing. Something wonderful provided for me by the computer.

I was struggling with a particularly hard to write piece, where I was trying to take the reader backwards towards using first principles. Not easy to begin with, and even harder for a novice writer like myself.

I was happy with what I had done. I probably would have posted it.

After agonizing for a couple of hours, I hit the update button. Something I've done a million times before. This time though, because I had been logged in for so long, Yahoo Notepad decided to ask me to confirm my userid. The session had timed-out. Ok. No worries. This problem has caught me in the past, and in Firefox if I hit the back button, I can get back to the original screen with the textbox, which from there I can copy and paste into a text file, and use it to update the note after I've logged in again.

I could test it, but if I remember correctly, in Yahoo Notepad, after the login screen, the data in the textbox is lost. It goes through as a post, I think, and cannot get passed back into the next query, so even if the program continues from the point it left off, the right data will not be updated, and possibly the file would end up being blanked if they tried. I could be wrong about this, hitting the back button and copying out the data worked well in the past, I seem to remember. The myths we build up about our software is often interesting in themselves.

This time, when I hit the back button to my horror the contents of the textbox where the original contents, not my updated version. My notepad file is quite large. It contains a lot of draft material, so I suspect that the size of the textbox forces Firefox not to save the data. So, going back won't work. I went forward too, but that didn't seem to work either. Although by the time I tried that, the panic had already set in. All of my labor, all of my work was gone!? Vanished. Kaput. Ahhhhhh.

When the anger subsided, and I stop swearing and pounding on things I really starting thinking about it.

In the early days of PCs, in Word Perfect we developed a disciple where we hit save file function every 10 minutes or so. It was necessary because the machines of the day were so flaky, that you were always running the risk of losing work. 10 minute intervals minimized the amount of lost work.

In those days, our technology was new and had problems. If someone lost their work, we told them it was their fault. They clearly weren't saving enough. We pushed the obvious problems with the technology back onto the users. Later a big advancement was to performed this neuroses automatically. The computer could take care of constantly saving the file for you.

A while back, Microsoft Word crashed, and I lost a couple of hours worth of work. I though it had been doing the ten minute backup thing, but that option was turned off. Not sure why, could have been a local thing or an network share problem, or something equally uninteresting.

Hardware crashes are not as common anymore, although it might just be bad luck, but I've had more problems with hardware in the last couple of years than I have for the prior decade. At work, for instance I've had four machines die in less than a year. There seems to be a noticeable decline in the dependency of modern hardware.

I'd figured Word was set up correctly, so I never bothered to check it myself. It wasn't. But I should have checked, they said. It was pushed back to me as the user, to have notice that Word was no longer creating backups. This brought on a twisted sense of deja vu and a 'sad realization'.

A computer is an extremely powerful tool; it can do so much. It has been around for decades and can automate a huge number of tasks.

It is sophisticate enough that we could build a system that could not lose any of your work beyond a few seconds, right up to and including a major catastrophic event like and asteroid colliding with earth. Since we don't have a backup planet, there is little we could do about that, but we do have technology designed to withstand a nuclear war, major storms, earth quakes and we do know how to create fault tolerance redundant distributed implementations.

Disk storage is cheap enough that we can keep all of the key strokes we've ever entered into the computer for our entire lives; allowing us to rebuild any or all of our labors.

Our technology should not allow us to ever lose work. We have progressed beyond that point.

Computers have a huge amount of potential, unknown to even the most savvy of us, but we have not been able to tap it. We write the same stuff over and over, repeating the same mistakes and never really learning from the past. At its best, a web application is possibly the worst ever technology invented. One giant mess of poorly implemented ideas and political maneuvering. A great big ball of mud. We have at the foundation, a great technology in the Internet, that we've build a huge mess on top, the World Wide Web. We connect to it with our easily modifiable PCs, that are complete untrustworthy and getting worse with each new release. Web applications are improving, but they are still extremely crude. And very disappointing. One step forward, and one step back.

But our problems aren't technological, they aren't something we need to invent or fix. We have all of the technology we need, all of the understanding. Our problems lie with us, the people building the technology, with our culture, and our need to be free, inconsistent and messy, and with the way we organize ourselves.

For decades now we have approached the same problems in the same manner, and consistently the results have been flawed, yet we do it again and again. I really don't think it is difficult to build working systems, but it does seem extremely difficult to organize people in a way that they can build working systems.

We have the technology, if only we could figure out a way to utilize it to our advantage. Until then, I guess I'll just have to lower my expectations and get used to losing my work every so often. It is only a couple of hours...

Lost in Processing

Sometimes it is like a long treacherous march through an endlessly dark and damp jungle. Whenever you think your nearing the end, another obstacle pops up to block your way. Behind every corner is only more pain, suffering and another corner. Every step of the way leaves you wondering if this time, it isn't really a death march. Will it end; can it go on forever?

How many software development efforts feel like an aimless trek through an untamed forest? A haphazard stumble through hostile terrain?

The goal when we start development is to try and find the shortest path towards getting it completed.

Wondering around aimlessly only increases the stress and makes it more difficult. Staying on course during development comes first from knowing the path, but also from following it. A map through the jungle is obviously a prerequisite. But that will only get you part of the way there, the second half and often more dangerous part of the journey comes from testing.

Most developers share the same fears: that initial one about being able to get it all going, and that final one about bringing it all together. Sometimes, in a particularly rough implementation, the last few months can feel like you are stuffing straw into one side of a burlap sack, only to watch it fall out the other side, again, and again, and again. If you haven't already lost it, by the time you get around to testing your probably pretty close. The jungle feels like it is closing in on you.

You might be a little frazzled going into testing, but you shouldn't let that be an excuse for turning off your brain. Testing, you see, is often were we lose the most, and gain the least. Development may seem to drag on endlessly, but testing is when you get totally lost.

It is worse too, when you widen your definition of 'flaws'; bugs you see, aren't just functions that don't quite work as expected, they also include spelling mistakes, grammar problems, awkward interfaces, ugly colors, poor documentation and anything else that ultimately distracts the users from an otherwise perfect system.

Often we try to narrow the definition as we get farther down the road, but that doesn't really help.

With an expanded view towards problems, if you really counted all of the flaws and inconsistencies for most software, the bug list would be huge. Yet, for the majority of bugs, the fixes don't take long to correct. For whatever reason -- most often fatigue -- we choose to turn a blind eye towards the ugly bits at the end of the development, even when they are simple to fix. Fear of making it worse is usually the culprit.

The standard problem with testing is inconsistency. The test cases get attention late in the game, and they always over-test the easy sections of the code, while ignoring the complicated parts. To be honest, most testing ranges from poor to totally ineffective. And most developers don't want to admit it.

Trouble starts early with the writing of the test plans.

It is commonly known that programmers make horrible testers. They test what they find interesting, which is usually the parts of the code they did well. They skip over what they dislike, mostly the sections of code with the higher bug counts.

This results in spending more time looking for bugs in the places they are least likely to be. If, as programmers we continually fail at testing, they by what madness does it make sense for us to write the test cases and test plans? We're only marginally better at thinking up cases for our own work, then we are at actually executing them properly.

Too add it it, some programmer's feel that uncovering a bug is a bad thing.

They want the program to get a bug free pass through in testing. They'll even pre-check the code, using up lots of time, and check all of the tests just to insure that there are no bugs. Fear of fixing bugs in the end seems to drive this type of behavior. If it were that the code was perfect, why bother to test it? Usually they burn through too much time in the simple sections trying to be perfect.

By the time testers execute the plans, they are ineffective because the underlying problems have been fixed, or because they really don't test anything significant. Sometimes, the same test suites will even be executed multiple times in a row, as if that increases the odds of finding anything. Hardly effective.

Testing is a necessary part of the process, so we want to maximize the time spent to get the biggest effect.

In that sense, we actually want to find bugs, particularly if we shortcut the work earlier to save time for other pieces of work. We don't want to find too many bugs, and we don't want to be forced back into another round of development, but we do want to maximize the effort. Finding bugs, after all shows that testing is actually working. Not finding bugs could either be interpreted as a good or a bad thing. More often its a bad thing.

I was once called a defeatist for suggesting that bugs will always exist in all code, but accepting it makes it more likely that you'll handle the bugs better. There is nothing you can do to stop bugs. Accepting that some flaws will get into the release makes it possible to handle them rationally.

If you don't plan on fixing bugs after the final release you are being too optimistic. In all my years of developing, the best I ever did was a four or five year period of just having just one bug in a complex production system. The secret behind this unusual success was lots of time; probably too much of it. Time that isn't generally available for most development projects.

Accepting that there are some arbitrary number of bugs left, helps to utilize the limited resources. It becomes a challenge to find them as quickly as you can with the least amount of effort. Even in a well tested system, a decent round of testing might reveal five or six. Digging deeper may reveal a couple more, but there will still be a couple of them that are shipped. We don't have an infinitely large amount of time, so bugs will exist. Knowing that, the only question is: which ones are the most acceptable?

Nothing is worse, then a stupid bug showing up in an obvious place, so if only for appearances sake, we always want to make sure that the main user pathways through the system are bug free. We can broaden the search out from there depending on the time and quality constraints.

When we do find a bug, there are always two choices, we can accept it, or we can fix it. Accepting it is generally cheaper because fixing it can reveal more bugs and require more testing. However, if you can precisely gauge the impact of the fix then you can retest only the smallest possible area; which is always cheaper than retesting everything. The impact of a fix in a well written program is not hard to determine.

Getting back to that previous project were we had virtually no bugs: what made it work so well was that the testers were two rather bored programmers from another project. They took up the challenge of finding the flaws in the system, and they did so with enthusiasm. The type of enthusiasm that the original programmers -- after months and months of pounding out code as fast as they can -- no longer had the ability to show.

The secret to great testing that I learned from that experience lies in handing off the testing to someone with fresh eyes. To really leverage independence, the testers themselves need to work out the test suite from first principles. They must do it directly from the design documentation because the programmers are tainted. The testers should be able to make certain assumptions about the type of system that fits into the documented problems and solutions. Not only does it supply good system testing, but it also validates that the design matches the system.

The time for independent testing is marginally longer, but if the results come back as several different sets of issues, the value for this type of assessment is worth it. Development can fix some issues, and schedule others for later. The results of different tests can be set back in several passes, allowing the developers to gauge the impact and make the changes early for specific types of problems; effectivily layering the test results. Testing could start with the deeper functionality and work its way out, taking care to insure that the order does not cause significant retesting.

There are various different types of tests that can be applied in this way, ranging in depth and effectiveness verses cost.

Rough testing could, for instance be applied in the middle of development to insure that the current direction is reasonable, and that the code is mostly usable. Walk-throughs could check for interface consistency and ease of use. In a large GUI program, specific pathways could be tested independently of each other. Sections of the system could receive extra testing because they are mission critical, or hard to recover from.

Interaction between the test teams and the developers can occur to validate some of the test plans and possibly later to help explain some of the problems, but it should be left to a minimum. The testers have an expectation that the system will behave in a specific way, it is there job to document any deviation. The developers need to assess the deviations and decide the underlying cause. If the testing is thorough, not everything found will be dealt with.

Wondering around aimlessly in a jungle is never fun. Knowing how long you have to suffer can help make it seem less painful. Still, even the shortest path for development can be a long journey. We need to be wise about how we spend our time. After a long and painful first half for the development, the programmers are tired and not functioning too well. Traversing the second half with a fresh team of testers -- while giving the developers a bit of breathing time -- increases the likelihood that the project will arrive straight at its destination. Where we know we continually have problems, we need to find solutions that fix the original issues without causing worse ones as a side effect. That puts us on the shortest path.

Architectural Blueprints

We are awash in an endless sea of data. Collecting it is easy. Storing it is gradually getting cheaper.

All of this information we gather serves us absolutely no purpose if we are unable to structure it and then summarize its meaning. With so much data, if we can't abstract our representation into some geometrically simple form of presentation, then we can never understand what it is that we have collected.

Crunching that much information is the job of a massive system. The type of behemoth that probably doesn't exist, but that most of us can image.

You can't build systems in that caliber by yourself. We can't even build them with a team. We have the technologies, but not the methodologies for this type of work. Large systems require large groups of developers to come together.

Directing a big group of people towards a successful end is not an easy task. When that task involves building something complicated with lots of dependencies, it becomes ever harder. Software, although virtual and seemingly 'lightweight' is plagued by a surprising amount of complexity. Hidden complexity that is usually underestimated.

By yourself, it may be easy to conceive of a simple design for some software program and over a few months or even a few years belt out the implementation. But the problem changes when you need to include other people, and it really changes when you include them right from the beginning.

To build big, you need a big group of people, there is no choice.

To get a group of programmers together, and to get them all building the same system, you need some form of control. You are unlikely -- for instance -- to get twenty great programmers; some of them will be more skilled than others. You are unlikely to get twenty mind readers; most of them require communication to direct them towards their work. You can't just fire the ones you don't like. The ones that don't measure up to your standards. You'll be left by yourself and you just aren't fast enough to build big systems.

So you need some way of aligning their goals towards a common vision. You need an architecture, and you need it in a form that is easily sharable. Blueprints. You need some type of blueprints that are:

    - Simple, yet abstract.
    - Small enough to understand.
    - Unambiguous, so it gets done correctly.

It is not that taking input from all of the programmers is a bad thing, or should be discouraged, but that 'design by committee' rarely produces elegant results. Where it matters, at the various 'levels' in the design you want consistency, so you want the same designer's influence. Where it could be considered independent, you could have multiple designers.

The quality of the design depends on its underlying consistency. The quality of what you build depends on the number of 'cooks'. Too many cooks spoil the broth.

That is why for many buildings -- at least where they actually care about appearance -- there is an architect. It is that one 'vision' that binds it all together, in a way that it is coherent.

The core of a blueprint is just enough information contained in an abstract sense that at least one person can cram it all into their mind and visualize it. The size of the abstraction is crucial, as you cannot easily validate it if it is too large.

A good architecture is an abstract design for the thing to be built, and its blueprints are the graphical (and textual) instantiation of that abstraction. They need just enough detail to go forward in an unambiguous manner. They needn't contain every little detail, but those details that are absent should be constrained in some way, for example by well known standards.

Having blueprints that you can share with the team is only the beginning. A good architecture does more than just iterate out the coding elements, it can also be used for:

    - Iterative Development Support.
    - Bug Reporting Triage.
    - Testing.

In the architecture, you draw both horizontal and vertical 'lines'. The system is decomposed into layers and components. When these pieces are truly encapsulated from each other, you get a stability to the project. Choosing to upgrade to a more professional resource management layer for example, can be done all by itself with no effect on the rest of the code. For each iteration, you can pick another layer or component and upgrade its depth and scope; growing the system piece by piece. If you know where you want to go in the future, this is a reliable way of getting there, one step at a time.

As well, with discreet well-understood 'lines', it becomes easier to quickly call the location of a bug. This attribute cuts down on support, as one person can accurately triage incoming bugs towards the right resources without having to get the whole team involved. A significant time saving.

The architectural lines effect testing as well, but I'll leave that as the subject of a follow up post. It is too complex to fit into this entry.

When you lay down the blueprints for an architecture you set a structure over the project. This type of framing gives stability, but it also allows for you to delegate responsibilities for the pieces to various sub-teams. If you've gone with a 'big ball of mud' for an architecture, the sub-teams will inevitably overlap with each other. That causes confusion and strife.

While we can envision huge systems capable of processing mass amounts of data, coordinating our efforts as a team towards a single development goal has always been a cultural problem for programmers. You can see the possibilities, but by not working together, all you get is frustration. Unlocking that potential comes from finding a way to communicate a consistent unambiguous design to a team of eager developers. We don't want to be mindless drones, but when that little extra freedom we crave threatens to derail the project, it also brings with it that horrible stress of possibly failing.

Blueprints that really work would alleviate many of the anxieties. If we get past this problem, then we can surf the sea of data and do some really interesting work, instead of reinventing the same simple applications over and over again.