Tuesday, February 26, 2008

Testing Perspectives

I just finished listening to a discussion between Jim Coplien and Bob Martin:


and there were some really interesting things in that conversation.

The most stunning was that Jim Coplien works with systems that are hundreds of millions of lines of code. That is a massive amount of code. Shouldn't a higher paradigm like Object-Oriented save one from such a difficult fate? How long does it take to assemble that much code? Why hasn't millions of it been refactored?

The next is that even though Jim was dead-on about his observations of having to start close to the underlying understanding of the actual domain in his example of writing software for a bank -- otherwise you'll spend 40 years reinventing a well-known wheel -- that center of the conversation just died. If you know how you need to structure your system, why wouldn't you go straight there with the minimum amount of work, especially if you have huge time constraints. Approaches such as CDD or TDD seem to be taking the long way around the block.

Jim's point about developers without an architecture painting themselves into a corner is true irregardless of any discussion on Object-Oriented or Agile issues. It is a long standing tradition in Computer Science for aimless meanderings to finish off a project badly, that hasn't changed in 40 years. It didn't work before OO, YAGNI, TDD or any other acronym and it still won't work after. If you don't start close to where you need to be, you probably won't ever get there unless you are really lucky.

Bob's description of TDD seems more like a 'game' where you are pitting two different representations of the same thing against itself. All the more so when you realize that Jim is correct about unit testing just being a game of 'sink the battleship' where you randomly fire specific tests over a large surface hoping to hit something, before running out of time and shipping the code. Testing is always about getting lucky. And doesn't context switching between tests and code tired one out more quickly? For speed, I've always liked to batch up the work as much as possible.

Bob's thesis that it was unprofessional to 'not' unit-test every line of code before a release was absolutely crazy. And not possible. Mass buckets of code get released by 'professionals' all of the time where the 'coverage' of the testing is probably way, way less than 30%. That's just a guess, but an easy one. TDD doesn't even stop that; given all of the areas were you can't use TDD (GUIs and databases?), huge amounts of untested code will get loose. Untested code will always get loose.

Even if you unit-tested each and every line to 110%, the combination all of the pieces will open up unexpected and undesirable results; which if you didn't have some overall architecture to tie it all together would be a particularly huge and likely fatal mess. You can't skip integration testing, it isn't safe. If you have to do a lot of it at the end, then wouldn't it be better to minimize the stuff in the middle or at the start?

Further, TDD is not a main-stream process right now, even if it is getting lots of coverage from the 'Agile Press'. Most code is still written using waterfall methods, for most industries. Agile itself isn't even main-stream, and TDD is only a tiny piece of it. Saying it is so, doesn't make it any truer, and it leaves a bad taste in the mouth.

The weirdest part for me is that both of these guys are arguing about how to make code more stable, or at least that's what I hope they are arguing about, otherwise what is the point? All this low-level testing is nice, but it doesn't invalidate the higher level full-system testing that 'must' be done to end-to-end test the various pathways through the code. When is low-level testing effective and when is it just doubling up the workload for no apparent reason?

So, I add a process to my house-building technique that makes sure each and every brick is super-accurate. Now they are all exactly the same size given some huge level of precision. That's nice, but does that really help me build a better house?