Organization

There is one simple, yet fundamental rule for organization: if someone hasn’t explicitly organized it, then it is disorganized.

I believe that one can infer this from the properties of our physical reality. We know for instance, that entropy always wins. What starts as chaos, ends in chaos. Order is a temporary state of affairs. So, without explicit action, order is highly unlikely.

This rule is especially important when building software. A reasonably large system may consist of millions of individual details, all of which need to be embedded into the system. The scope of any development project includes all aspects of analysis, design, implementation, testing and operations. It includes the process of completing the work as well as the details that go into it. There may be regulatory or licensing issues floating about as well. Each area may not appear daunting, but when taken together there is an awful lot between the genesis of an idea and actual utilization of it for practical purposes.

All of these different aspects need coherent organization. They need to be collected, sorted, categorized and normalized so that they are available to the builders, operators and often, users of the system.

In a small project, you can easily get away with a casual approach to managing these details. If there are three or four parts to something, they don’t need any explicit arrangement. But as the size of the project increases, the number of parts grows extremely fast and thus juggling, say a few hundred unorganized parts easily results in a significant number of unexpected problems. And these problems may cascade into other problems.

This effect of scale often means that sloppy habits learned from small projects generally lead to overconfidence in big ones. Just because some aspect didn’t require organization when the project was a mere 20,000 lines, doesn’t mean that it won’t when it balloons to a few hundred thousand lines. And millions of lines requires something else altogether.

As the project size increases, the effects of disorganization become more pronounced. They require more effort to control, and can balloon into more formidable problems if left unchecked for too long. Thus in serious software development, scale is everything. It’s the first thing you investigate, and it’s always the most significant aspect of any large project. The only way to tame this rampant complexity growth is via organization.

Realistically, it doesn’t matter how it is organized, so long as that organization is consistent and sustainable. But it does matter that the organizational system spans the entire scope of the effort. You can’t just organize a sub-part of the project or process and hope that it will somehow magically propagate to the other areas. You have to cover over all of the details, and all of the work getting done, and you have to insure consistent application.

And it also matters how deep the organization goes. You may have some higher level methodology, and be very organized right at the bottom in terms of the code, but if what’s left in the middle is left untouched, eventually it too will cause significant problems. Every part of the process, architecture and sub-problems needs some coherent organizing principle.

Once the dust settles, what usually brings down ambitious software projects, or grinds them to a standstill, is an explosion of complexity. Fundamentally there is nothing that you can do about that, other then partition it carefully and encapsulate it into manageable sub-parts. But it is possible to prevent any secondary complexity caused by disorganization. And it is this avoidable artificial complexity that generally spirals out of control. When stopped the project becomes tractable, but when ignored it combinatorially explodes as the project expands. A little bit of organization goes a long way ... and enough of it may save the project from a premature death ...