Organization

“A place for everything, everything in its place.”

Benjamin Franklin

As Benjamin Franklin pointed out there are two parts to organization. The first is that absolute everything needs to fit somewhere. With software this really translates to having a solid 'reason' for every little bit in the system, be it config files, methods, data etc. and a reason for its location. It all needs its place, and often for that it also needs some level of categorization. "It doesn't matter" is synonymous with "it's not organized".

This includes names, conventions, even coding styles. Everything. Each tiny piece of work, each little change should all have its place. It should all have a reason for being where it is, as it is.

The second half of the quote is just as important. What's the point of having an organizational scheme if its not being used. As new things are added, they need to be added into their place. Sometimes it’s clear where they belong, but often times it hasn't been explicitly documented. Assuming that lack of documentation equates to lack of organization (and as such it is a free for all) is a common failing amongst inexperienced coders. Things can be organized with having an accompanying manual.

Failing to keep a development project organized is the beginning of the end. Disorganization is probably the worst aspect of technical debt because it is a direct path into various forms of spaghetti. And spaghetti is a quagmire for time, either to fix bugs or to extend out the functionality.

Part of software development culture over the last couple of decades has been a strong desire for 'freedom'. Freedom is great, but used improperly it just becomes an excuse for creating disorganization. For instance, utilizing the full freedom to create a new screen in a much fancier way than the existing ones is really just breaking the organization of what is already there. It's like helping someone dry the dishes in their kitchen, but then insisting on placing the clean stuff in any other location than where it actually belongs. It isn't really helpful is it?

It's true that in some regards a well-organized existing development project is a boring one. If the work isn't exceeding the existing organizational bounds then it can really just fit in like all the other pieces. But success in development doesn't come from making any one small piece of the system great, it's an all or nothing deal. The system is only as good as its crappiest screen or stupidest functionality. Thus changes to enhance it or its organization can't be selective. The whole thing needs to be fixed or it’s actually just making it worse.

If you're evaluating an existing development project, disorganization is easily the best indicator of the future. A small project can get away with disorganization, but anything larger absolutely needs organization to survive. If it is lacking or failing, then the whole effort is in deep trouble.

Deep Underground

There are two distinct sides to software development. Since software is a solution to someone’s problem -- a tool to help them in some way -- it lies in a specific business or ‘domain’. Its purpose is tied directly to helping automate, track and organize very specific information. But the software itself is constructed using parts like computers, networks, libraries, programming languages, etc. so it is composed of and limited by ‘technology’. Both the domain and the technology are necessary ingredients in being able to help users, rather than make their issues worse.

If a software development project goes wrong by mostly ignoring the domain issues we usually say that it was built in an ‘Ivory Tower’. It’s a nice metaphor to describe how the developers may have completed the technical side, but they were too far up and away from the domain problems. The software is useless because the creators didn’t bother to dig into the details.

But what about the opposite problem? What if the developers pay close attention to the domain, but fail entirely to properly handle the technical issues?

Since that is fundamentally the opposite of an Ivory Tower we can flip the terminology. An ‘Ebony Dungeon’ project then is one that delves deep into the heart of the domain, but so deeply that it ignores the technology side. We often see this in in-house projects where the business or domain experts exert too much influence over the process, techniques and construction of the software.

Most domains have to tie themselves closely to some form of revenue stream. Those ties mean that they need to react quickly to changes. That sets up a culture of just focusing heavily on the immediate short-term, trying to be as malleable as possible. That works for business, but a large software development project is actually a very slow moving ‘ship’. It slowly goes from version to version, plodding along for years. While the business may need to wobble back and forth as its demands change, the only efficient path for development is to steer as straight a course as possible. There are a lot of moving parts in software and to get them all working properly they need to be organized; put into their proper places. A constantly shifting direction undoes any of this organization, which wastes time and causes severe problems.

A large system full of useful domain functionality isn’t actually very useful if it is technologically unstable. If it crashes frequently or is prone to serious bugs because of mass redundancies or if its performance is dismal, all of that existing functionality is unaccessible. A smaller more stable system would be far more effective at helping the users. The features available should work properly, be complete and be organized.

A very clear symptom that a project is trapped in an Ebony Dungeon is that most of the decisions keep getting punted back to the domain experts. “We’ll have to ask the business what they want”. If the project is balanced then at least 50% of the effort is related to the technical side. That includes using industrial strength components and algorithms, keeping the code base clean and insuring that the operations and installations side are built up to an automated level as well. Technical debt is unavoidable, but it needs to be controlled and managed with the same importance as any other aspect of the project.

In areas like GUIs, industry conventions should trump individual's preferences, so that the screens don’t become a sea of eclectic behaviors and that the functionality isn’t randomly distributed throughout the screens. Failure to properly organize the functionality at the interface level will cause a failure to use the functionality at the user level. Proper organization of a huge number of features is an extremely difficult problem that takes decades to master. A domain expert may understand the functionality requirements, but organization is just as, if not more, critical.

Being trapped in a dungeon exerts a lot of pressure on the programmers to create code as fast as possible. This usually manifests itself as a significant amount of “copy and paste” programming. Old code is copied over and then hastily modified with a small number of differences. We’ve known that this style of development is extremely bad for decades, but it is still commonly used to satisfy time pressure. Programmers need time to understand and refactor their work if its going to be extended properly. A rushed job is a sloppy job.

An Ivory Tower system misleads its backers because the real problems don’t become apparent until the users start working with the system. An Ebony Dungeon system also misleads its backers because it starts off fast and agile but slams into a wall when the work is to be extended. What appeared to be a big success in the early days ends up dying a premature death, usually costing way more than it should.

The software industry has swung rather heavily towards Ebony Dungeons lately. It’s easy to do because domain experts over-simplify the real work involved to build a system then they get mislead by the rapid progress. Without an experienced development crew it becomes easy to miss all of the symptoms of a project in deep trouble. Most that are dying get caught up in tunnel vision, thinking that just a few more features will turn the direction around and save the project. But “just a few more...” is actually the problem.

The best way out of a dungeon is to properly partition the system requirements. Everyone talks about ‘user requirements’ but they are only a small subset of what’s needed for a successful system:

• user requirements
• management requirements
• technical operations/release requirements
• support/debugging requirements
• development/programmer requirements

If you factor in all of the different requirements necessary to build and run the system, you see that ‘features’ are only one aspect of the work. If you ignore the other four (or more) categories than obviously there will be serious problems with the system. If all of the requirements come from the domain experts, since they don’t know about or understand the other issues they won’t place any importance on getting them done.

The real art in building systems is to not go too high or too low, but rather to build on solid ground that is accessible to everyone. Towers and dungeons are equally bad.