Industrial Strength Requirements

A while ago I was in a discussion with one of my friends. He said something interesting that really caught my attention. His code was in testing and had turned up some corner-cases. While developing the code he became aware of these issues, but as he said “they weren’t in the requirements”, so he didn’t address them.

I found myself surprised. There is -- I strongly believe -- for all code an inherent set of basic requirements that goes without saying. The users don’t need to specify them, they just come in with the expectation that the code will be able to withstand normal wear and tear. That it will be usable.

We don’t, for instance, need to specify that a GUI should be easy to use. That requirement is fundamental, it is right there in the definition of GUI, and we certainly don’t need people to iterate what that means in general.

We don’t need people to specify that a batch script should be restartable and idempotent. Both qualities go back decades and are important to insuring that the system is operationally sound.

We don’t need people to specify that installation should be simple and repeatable. Those were lessons learned a long time ago.

I could go on, there is a massive amount of these well-known ‘best practices’, ‘conventions’, ‘acceptable behavior’, etc. We already know a huge amount about the proper way to build software so that it lives up to its usage; so that it is industrial strength.

A set of requirements from the users, managers, or whoever, is a set of things that are merged with this existing knowledge base. These default requirements. And for experienced developers, although there are no official standards for these necessities, we know what they are and why they are important.

When we are commissioned to write or enhance a code base,  these fundamentals need to be in place. They don’t need to be iterated by non-experts, they just need to be there.

On occasion, a few of the stakeholder requirements may contradict the default ones. That happens, particularly when stakeholders don’t have enough experience with software. In such collisions, the default requirements hold precedence, and it is up to the programmers to bring this issue to the stakeholders. If there are good solid reasons for altering the defaults and someone make a reasonable case, then it can be done in that way, but otherwise we’re professional software developers so we need to build things that work. So the defaults are important.

For me, in my friends situation, I would have gone back early with the corner-cases and the way I intended to solve the problems. I’d be open for debate, but I prefer to go in with a viable solution, it makes it faster to get an agreement. If it’s a domain issue, since I’m not the expert, I would be extremely flexible to what they need. With a technical issue, I would insure that any alternatives don’t compromise the system. But in either case, one should never violate good practices unless there is a valid reason to do so. The most important aspect is that the issue should come up for discussion, and that the resolution should be noted and followed.

Requirements in that sense, are those things above and beyond the fact that we are building some industrial-strength software. Even before the project was a glimmer in some-one’s eye there is a huge amount of basic work necessary to make it successful. We shouldn’t dump the responsibility for knowing what that is on the stakeholders, it’s part of our job.

How Many Programmers are there?

I just saw this interesting blog post, here, that states that there are over 9 million Java programmers. That number doesn’t ring true with me, my last recollection -- from who knows where -- was that there was roughly a million or so programmers in all languages/technologies.

For this type of number, I’d like it to be restricted to people that are making a “living” off of programming, doing it on a full-time or nearly full-time basis. The number of hobbyists and students is interesting, but these numbers fluctuate widely with popularity. There are also a lot of 'coders' in operations and support roles, but they tend to write small works that smooth over bumps or glue together existing pieces. It's the number of people doing the heavy lifting that I am curious about.

No doubt it is a difficult number to find. You can’t just measure downloads, or up and coming job ads. And there is a huge segment of older programmers out there making a good living from the mature technologies like COBOL and mainframes, but most of them have a near-zero web presence. Still, they are an important group because their technologies run a good majority of the mission critical applications and they often have more refined engineering skills and processes.

If anyone has any links, or comments, I’d be really interested. I’ve seen a lot of resources, but I’ve never found one that I felt really captured this number correctly. 

Economy of Expression

One striking features that I’ve always found with elegant code is how it manages do so much with so little. In its expression it is dramatically stingy, yet in its functionality it is extremely broad and applicable to a large number of similar problems.

Software ultimately is just a very long sequence of instructions for a computer to follow. When building up these sequences, we can code them each very precisely to the specific instance of a specific problem, or we can take a step back and then generalize them to allow for a wider range of reuse.

All computer languages do this by providing sets of standard libraries that accomplish common tasks. Many of these libraries find simple, elegant abstractions that allow for a huge range of functionality by providing an underlying set of consistent primitives, on which programmers can build larger more specific blocks of code. Get the primitives right and the next layer gets cleaner and easier to build. Get them wrong, and the eccentricities percolate into the code above, making it more difficult.

Great examples of reuse in code understand this well. Instead of pounding out an endless series of nearly duplicated lumps of code, the programmers have built layer after layer of clean, well-thought out Lego-like blocks that get assembled into ever higher functionality.

The key to doing this isn’t by allowing for a mass of configurable options, or by forcing a large amount of separate declarative bindings. It isn’t by allowing lots of arguments to the methods, or by wiring everything up with arbitrarily convoluted underlying rules. Rather, when done well, it usually comes from a simple clean abstraction, iron-like consistency and a serious amount of economy of expression in both the internals and in how it is used externally.

That is, a simple and consistent abstraction yields a small number of primitives that are easily interwoven at a specific level. And handled correctly, it encapsulates all of the nasty details while providing an obvious set of default behaviors. Simple things are simple to do. At its best, the calling code reads very straight-forward and is naturally self-documenting. You don’t have to know the underlying details or read reams of incoherent documentation to get a correct sense of the underlying behavior. You can use it, it is obvious, and you can move on.

This often happens with well-written libraries, but the principle can be applied to all software code. A large application may have many layers, but each layer stands on its own as a simple, readable and easily understandable work.

One way to judge the degree of success in achieving this is to give the work to another programmer that has never seen it before. If they can get a fairly good grasp of what the code is doing nearly instantly, then it has these qualities. If they are confused, or turned around, or need extra documentation then the code does not speak for itself very clearly. Or the abstraction is too convoluted (or the programmer is too junior for this type of abstraction).

To get really strong economy of expression the primitives must be a small number, not overlapping and they must cover all of the possible operations. A non-programming example is +, -, * and /. With just four operators, one can do a large amount of arithmetic. AND, OR and NOT is another example of a consistent family of primitives (as is NAND and NOR). Applying these primitives to objects (mathematical in this case) in various combinations, provides for a wide range of applications. There are no oddly overlapping operators like addOneAndMultipleByTwo, since at the level they are getting used, it would be unnecessary and confusing.

Simple primitives, well-thought-out lead to very clean implementations underneath. They allow any layer built on top to be clean and readable. They simplify, encapsulate and self-document some underlying complexity, while providing a strong base for building on.

When done in a large system, at each level, it makes it really easy to view each level individually, with its inherent complexity, and understand how it operates with respect to the level below. It makes it easy to change, without having to worry about unintended side-effects. It makes it easy to extend, to cover new functionality. This, by definition, is the essence of elegance. Clean and simple with no weird bits or thingys that will get posted to WTF.  Done well, further development work on an elegant system becomes interesting, fun and fast as opposed to being time-consuming, painful and dangerous.