Some Rules

Big projects can be confusing and people rarely have little time or energy to think deeply about their full intertwined complexity. Not thinking enough is often the start of serious problems.

Programmers would prefer to concentrate on one area at a time, blindly following established rules for the other areas. Most of our existing rules for software are unfortunately detached from each other or are far too inflexible, so I created yet another set:

1. The work needs to get used
2. There is never enough time to do everything properly
3. Never lie to any users
4. The code needs to be as readable as possible
5. The system needs to be fully organized and encapsulated
6. Don’t be clever, be smart
7. Avoid redundancy, it is a waste of time

The work needs to get used

At the end of the day, stuff needs to get done. No software project will survive unless it actually produces usable software. Software is expensive, someone is paying for it. It needs backers and they need constant reassurance that the project is progressing. Fail at that, nothing else matters.

There is never enough time to do everything properly

Time is the #1 enemy of software projects. It’s lack of time that forces shortcuts which build up and spiral the technical debt out of control. Thus, using the extremely limited time wisely is crucial to surviving. All obvious make-work is a waste of time, so each bit of work that gets done needs to have a well-understood use; it can’t just be “because”. Each bit of code needs to do what it needs to do. Each piece of documentation needs to be read. All analysis and design needs to flow into helping to create actual code. The tests need to have the potential to find bugs.

Still, even without enough time, some parts of the system require more intensive focus, particularly if they are deeply embedded. Corners can be cut in some places, but not everywhere. Time lost for non-essential issues is time that is not available for this type of fundamental work. Building on a bad foundation will fail.

Never lie to any users

Lying causes misinformation, misinformation causes confusion, confusion causes a waste of time and resources, lack of resources causes panic and short-cuts which results in a disaster. It all starts with deviating from the truth. All of the code, data, interfaces, discussions, documentation, etc. should be honest with all users (including other programmers and operations people); get this wrong and a sea of chaos erupts. It is that simple. If someone is going to rely on the work, they need it to be truthful.

The code needs to be as readable as possible

The code in most systems is the only documentation that is actually trustworthy. In a huge system, most knowledge is long gone from people's memory, the documentation has gone stale, so if the code isn't readable that knowledge is basically forgotten and it has become dangerous. If you depend on the unknown and blind luck, expect severe problems. If, however, you can actually read the code quickly, life is a whole lot easier.

The system needs to be fully organized and encapsulated

If the organization of the code, configuration, documentation or data is a mess, you can never find the right parts to read or change when you need to. So it all needs organization, and as the project grows, the organizational schemes need to grow with it and they need organization too, which means that one must keep stacking more and higher levels of organization on top. It is never ending and it is part of the ongoing work that cannot be ignored. In huge systems, there is a lot of stuff that needs organization.

Some of the intrinsic complexity can be mitigated by creating black boxes; encapsulating sub-parts of the system. If these boxes are clean and well-thought out, they can be used easily at whatever level is necessary and the underlying details can be temporarily ignored. That makes development faster and builds up ‘reusable’ pieces which leverages the previous work, which saves time and of course, not having time is the big problem. It takes plenty of thought and hard work to encapsulate, but it isn’t optional once the system gets large enough. Ignoring or procrastinating on this issue only makes the fundamental problems worse.

Don’t be clever, be smart

Languages, technologies and tools often allow for cute or clever hacks. That is fine for a quick proof of concept, but it is never industrial strength. Clever is the death of readability, organization and it is indirectly a subtle form of lying, so it causes lots of grief and eats lots of time. If something clever is completely and totally unavoidable then it must be well-documented and totally encapsulated. But it should be initially viewed as stupendously dangerous and wrong. Every other option should be tried first.

Avoid redundancy, it is a huge waste of time

It is easy to stop thinking and just type the same stuff in over and over again. Redundancy is convenient. But it is ultimately a massive waste of time. Most projects have far more time than they realize, but they waste it foolishly. Redundancy makes the system fragile and it reduces its quality. It generates more testing. It may sometimes seem like the easiest approach, but it isn’t. If you apply brute force to just pound out the ugliest code possible, while it is initially faster it is ultimately destructive.

Overall

There really is no finite, well-ordered set of one-size-fits-all rules for software development but this set, in this order, will likely bypass most of the common problems we keep seeing over the decades. Still, software projects cannot be run without having to think deeply about the interrelated issues and to constantly correct for the ever-changing context. To keep a big project moving forward efficiently requires a lot of special skills and a large knowledge base of why projects fail. The only real existing way to gain these skills is to get mentored on a really successful project. At depth, many aspects of developing big systems are counter-intuitive to outsiders with oversimplifications. That is, with no strong, prior industry experience, most people will misunderstand where they need to focus their attention. If you are working hard on the wrong area of the project, the neglected areas are likely to fail.

Often times, given an unexpectedly crazy context, many of the rules have to be temporarily dropped, but they should never be forgotten and all attempts should be made to return the project to a well-controlled, proactive state. Just accepting the madness and assuming that it should be that way is the road to failure.