Process

A little process goes a long way. Process is, after all, just a manifestation of organization. It lays out an approach to some accomplishment as a breakdown of its parts. For simple goals the path may be obvious, but for highly complex things the process guides people through the confusion and keeps them from missing important aspects.

Without any process there is just disorganization. Things get done, but much is ignored or forgotten. This anti-work usually causes big problems and these feed back into the mix preventing more work from getting accomplished. A cycle ensues, which among other problems generally affects morale, since many people start sensing how historic problems are continuously repeating themselves. Things either swing entirely out of control, or wise leadership steps in with some "process" to restore the balance.

Experience with the chaotic none-process can often lead people to believe that any and all processes are a good thing. But the effectiveness of process is essentially a bell curve. On the left, with no process, the resulting work accomplished is low. As more process is added, the results get better. But there is a maximal point. A point at which the process has done all that it can, after which the results start falling again. A huge over-the-top process can easily send the results right back to where they started. So too much process is a bad thing. Often a very bad thing.

Since the intent of having a process is to apply organization to an effort, a badly thought out process defeats this goal. At its extreme, a random process for example, it is just formalized disorganization. Most bad processes are not truly random but they can be overlapping, contradictory or even have huge gaps in what they cover. These problems all help reduce the effectiveness. Enough of them can drive the results closer to being random.

Since a process is keyed to a particular set of activities or inquires, it needs to take the underlying reality into account. To do this it should be drafted from a 'bottom-up' perspective. Top-down process rules are highly unlikely to be effective primarily because they are drafted from an over-simplification of the details. This causes a mismatch between the rules and the work, enhancing the disorganization rather than fixing it.

Often bad process survives, even thrives, because its originators incorrectly claim success. A defective software development process, for instance, may appear to be reducing the overall number of bugs reaching the users, but the driving cause of the decreases might just be the throttling of the development effort. Less work gets done, thus there are less bugs created, but there is also a greater chance for upper management to claim a false victory.

It's very easy to add complexity to an existing process. It can be impossible to remove it later. As such, an overly complex process is unlikely to improve. It just gets stuck into place becoming an incentive for any good employees to leave, and then continues to stagnate over time. This can go on for decades. Thus arguing for the suitability of a process based on the fact that its been around for a long time is invalid. All it shows is that it is somewhat better than random, not that is is good or particularly useful in any way.

Bad process leaves around a lot of evidence lying around that it is bad. Often the amount of work getting accomplished is pitifully low, while the amount of useless make-work is huge. Sometimes the people stuck in the process are forced to bend the truth just to get anything done. They get caught between getting fired for getting nothing done or lying to get beyond the artificial obstacles. The division between the real work and its phantom variant required by the process manifest into a negative conflict-based culture.

For software, picking a good process is crucial. Unfortunately the currently available choices out there in the industry are all seriously lacking in their design. From experience the great processes have all been carefully homegrown and driven directly by the people most affected by them. The key has been promoting a good engineering culture that has essentially self-organized. This type of evolution has been orders of magnitude more successful than going out and hiring a bunch of management consults who slap on a pre-canned methodology and start tweaking it.

That being said, there have also been some horrific homegrown processes constructed that revel in stupid make-work and creatively kill off the ability to get anything done. Pretty much any process created by someone unqualified to do so is going to work badly. It takes a massive amount of direct experience with doing something over and over again before one can correctly take a step back and abstract out the qualities that make it successful. And abstraction itself is a difficult and rare skill, so just putting in the 10,000+ hours doesn't mean someone is qualified to organize the effort.

Picking a bad process and sticking to is nearly the same as having no process. They converge on the same level of ineffectiveness.

Death by Code

A mistake I've commonly seen in software development is for many programmers to believe that things would improve on a project, if they only had more code.

It's natural I guess, as we initially start by learning how to write loops and functions. From there we move onto to being able to structure larger pieces like objects. This gradual broadening of our perspective continues, as we take on modules, architectures and eventually whole products. The scope of our understanding is growing, but so far its all been contained within a technical perspective. So, why not see the code as the most important aspect?

But not all code is the same. Not all code is useful. Just because it works on a 'good' day doesn't mean that it should be used. Code can be fragile and unstable, requiring significant intervention by humans on a regular basis. Good code not only does the right thing when all is in order, but it also anticipates the infrequent problems and handles them gracefully. The design of the error handling is as critical (if not more) than the primary algorithms themselves. A well-built system should require almost no intervention.

Some code is written to purposely rely on humans. Sometimes it is necessary -- computers aren't intelligent -- but often it is either ignorance, laziness or a sly form of job security. A crude form of some well-known algorithms or best practices can take far less time to develop, but it's not something you want to rely on. After decades we have a great deal of knowledge about how to do things properly, utilizing this experience is necessary to build in reliability.

Some problems are just too complex to be built correctly. Mapping the real world back to a rigid set of formal mechanics is bound to involve many unpleasant trade-offs. Solving these types of problems is definitely state-of-the-art, but there are fewer of these out there than most programmers realize. Too often coders assume that their lack of knowledge equates to exploring new challenges, but that's actually quite rare these days. Most of what is being written right now has been written multiple times in the past in a wide variety of different technologies, It's actually very hard to find real untouched ground. Building on past knowledge hugely improves the quality of the system and it takes far less time, since the simple mistakes are quickly avoided.

So not all code is good code. Just because someone spent the time to write it doesn't mean that it should be deployed. What a complex system needs isn't more code, but usually less code that is actually industrial strength. Readable code that is well-thought out and written with an strong understanding of how it will interact with the world around it. Code that runs fast, but also is defensive enough to make problems easier to diagnose. Code that fits nicely together into a coherent system, with some consistent form of organization. Projects can always use more industrial strength code -- few have enough -- but that code is rare and takes time to develop properly. Anything else is just more "code".

Monitoring

The primary usage of software is collecting data. As it is collected, it gets used to automate activities directly for the users. A secondary effect from this collection is the ability to monitor how these activities are progressing. That is, if you've build a central system for document creation and dissemination, you also get the ability to find out who's creating these documents and more importantly how much time they are spending on this effort.

Monitoring the effectiveness of some ongoing work allows for it to be analyzed and improved, but it is a nasty double-edged sword. The same information can be used incorrectly to pressure the users into performing artificial speedups, forcing them to do unpaid work or to degenerate their quality of effort. In this modern age it isn't unusual for some overly ambitious upper-management to demand outrageous numbers like 150% effort from their staff. In the hands of someone dangerous, monitoring information is a strong tool for abuse.They do this to get significant short-term gains but these come at the expense of inflicting long-term damage. They don’t care, they're usually savvy enough to move on to their next gig long before that debt actually becomes a crisis.

Because of its dual nature, monitoring the flow of work through a big system is both useful but also difficult. It is done well when it gets collected, but is limited in its availability. Software that rats out its users is not appreciated, but feedback to help improve the working effectiveness is. One way of achieving this latter goal is to collect fine-grained information about all of the activities, but only make it available as generalized anonymous statistics. That is, you might know the minimum and maximum times people spend on particular activities, but all management can see is the average and perhaps the standard deviations. No interface exists for them to pull up the info on a specific user, so they can’t pressure or punish them.

Interestly enough, when collecting requirements for systems, fine-grained monitoring often shows up. Not only that, but there is usually some 'nice sounding' justification for having it. Most of software development these days is oriented to giving the ‘stakeholders’ exactly what they want, or even what they ask for, but this is one of those areas where professional software developers shouldn't bow directly to the pressure. It takes some contemplation, but a good developer should always empathize with their users -- all of them -- and not build anything that they wouldn't like applied to themselves. After all, would you really be happy at work if you had to do something demeaning like punch a timecard in and out? If you don't like it why would anyone else?

Organization

“A place for everything, everything in its place.”

Benjamin Franklin

As Benjamin Franklin pointed out there are two parts to organization. The first is that absolute everything needs to fit somewhere. With software this really translates to having a solid 'reason' for every little bit in the system, be it config files, methods, data etc. and a reason for its location. It all needs its place, and often for that it also needs some level of categorization. "It doesn't matter" is synonymous with "it's not organized".

This includes names, conventions, even coding styles. Everything. Each tiny piece of work, each little change should all have its place. It should all have a reason for being where it is, as it is.

The second half of the quote is just as important. What's the point of having an organizational scheme if its not being used. As new things are added, they need to be added into their place. Sometimes it’s clear where they belong, but often times it hasn't been explicitly documented. Assuming that lack of documentation equates to lack of organization (and as such it is a free for all) is a common failing amongst inexperienced coders. Things can be organized with having an accompanying manual.

Failing to keep a development project organized is the beginning of the end. Disorganization is probably the worst aspect of technical debt because it is a direct path into various forms of spaghetti. And spaghetti is a quagmire for time, either to fix bugs or to extend out the functionality.

Part of software development culture over the last couple of decades has been a strong desire for 'freedom'. Freedom is great, but used improperly it just becomes an excuse for creating disorganization. For instance, utilizing the full freedom to create a new screen in a much fancier way than the existing ones is really just breaking the organization of what is already there. It's like helping someone dry the dishes in their kitchen, but then insisting on placing the clean stuff in any other location than where it actually belongs. It isn't really helpful is it?

It's true that in some regards a well-organized existing development project is a boring one. If the work isn't exceeding the existing organizational bounds then it can really just fit in like all the other pieces. But success in development doesn't come from making any one small piece of the system great, it's an all or nothing deal. The system is only as good as its crappiest screen or stupidest functionality. Thus changes to enhance it or its organization can't be selective. The whole thing needs to be fixed or it’s actually just making it worse.

If you're evaluating an existing development project, disorganization is easily the best indicator of the future. A small project can get away with disorganization, but anything larger absolutely needs organization to survive. If it is lacking or failing, then the whole effort is in deep trouble.

Deep Underground

There are two distinct sides to software development. Since software is a solution to someone’s problem -- a tool to help them in some way -- it lies in a specific business or ‘domain’. Its purpose is tied directly to helping automate, track and organize very specific information. But the software itself is constructed using parts like computers, networks, libraries, programming languages, etc. so it is composed of and limited by ‘technology’. Both the domain and the technology are necessary ingredients in being able to help users, rather than make their issues worse.

If a software development project goes wrong by mostly ignoring the domain issues we usually say that it was built in an ‘Ivory Tower’. It’s a nice metaphor to describe how the developers may have completed the technical side, but they were too far up and away from the domain problems. The software is useless because the creators didn’t bother to dig into the details.

But what about the opposite problem? What if the developers pay close attention to the domain, but fail entirely to properly handle the technical issues?

Since that is fundamentally the opposite of an Ivory Tower we can flip the terminology. An ‘Ebony Dungeon’ project then is one that delves deep into the heart of the domain, but so deeply that it ignores the technology side. We often see this in in-house projects where the business or domain experts exert too much influence over the process, techniques and construction of the software.

Most domains have to tie themselves closely to some form of revenue stream. Those ties mean that they need to react quickly to changes. That sets up a culture of just focusing heavily on the immediate short-term, trying to be as malleable as possible. That works for business, but a large software development project is actually a very slow moving ‘ship’. It slowly goes from version to version, plodding along for years. While the business may need to wobble back and forth as its demands change, the only efficient path for development is to steer as straight a course as possible. There are a lot of moving parts in software and to get them all working properly they need to be organized; put into their proper places. A constantly shifting direction undoes any of this organization, which wastes time and causes severe problems.

A large system full of useful domain functionality isn’t actually very useful if it is technologically unstable. If it crashes frequently or is prone to serious bugs because of mass redundancies or if its performance is dismal, all of that existing functionality is unaccessible. A smaller more stable system would be far more effective at helping the users. The features available should work properly, be complete and be organized.

A very clear symptom that a project is trapped in an Ebony Dungeon is that most of the decisions keep getting punted back to the domain experts. “We’ll have to ask the business what they want”. If the project is balanced then at least 50% of the effort is related to the technical side. That includes using industrial strength components and algorithms, keeping the code base clean and insuring that the operations and installations side are built up to an automated level as well. Technical debt is unavoidable, but it needs to be controlled and managed with the same importance as any other aspect of the project.

In areas like GUIs, industry conventions should trump individual's preferences, so that the screens don’t become a sea of eclectic behaviors and that the functionality isn’t randomly distributed throughout the screens. Failure to properly organize the functionality at the interface level will cause a failure to use the functionality at the user level. Proper organization of a huge number of features is an extremely difficult problem that takes decades to master. A domain expert may understand the functionality requirements, but organization is just as, if not more, critical.

Being trapped in a dungeon exerts a lot of pressure on the programmers to create code as fast as possible. This usually manifests itself as a significant amount of “copy and paste” programming. Old code is copied over and then hastily modified with a small number of differences. We’ve known that this style of development is extremely bad for decades, but it is still commonly used to satisfy time pressure. Programmers need time to understand and refactor their work if its going to be extended properly. A rushed job is a sloppy job.

An Ivory Tower system misleads its backers because the real problems don’t become apparent until the users start working with the system. An Ebony Dungeon system also misleads its backers because it starts off fast and agile but slams into a wall when the work is to be extended. What appeared to be a big success in the early days ends up dying a premature death, usually costing way more than it should.

The software industry has swung rather heavily towards Ebony Dungeons lately. It’s easy to do because domain experts over-simplify the real work involved to build a system then they get mislead by the rapid progress. Without an experienced development crew it becomes easy to miss all of the symptoms of a project in deep trouble. Most that are dying get caught up in tunnel vision, thinking that just a few more features will turn the direction around and save the project. But “just a few more...” is actually the problem.

The best way out of a dungeon is to properly partition the system requirements. Everyone talks about ‘user requirements’ but they are only a small subset of what’s needed for a successful system:

• user requirements
• management requirements
• technical operations/release requirements
• support/debugging requirements
• development/programmer requirements

If you factor in all of the different requirements necessary to build and run the system, you see that ‘features’ are only one aspect of the work. If you ignore the other four (or more) categories than obviously there will be serious problems with the system. If all of the requirements come from the domain experts, since they don’t know about or understand the other issues they won’t place any importance on getting them done.

The real art in building systems is to not go too high or too low, but rather to build on solid ground that is accessible to everyone. Towers and dungeons are equally bad.