Software is counter-intuitive. Sometimes to speed up development, you have to slow it down.
Let me explain.
It’s easy to get a small piece of software together. You belt out the basic code; you don’t need to worry about high falutin ideas like architecture, organization, etc. You just need code.
That works, only as long as the software stays small. But the technique itself is predicated on pounding out rather frail, hard-coded instructions that are very specific to the task at hand. That is fine, but it doesn’t scale, Not even a little bit.
Once the project has accumulated enough lines of code or other forms of complexity, making any changes to it involves finding the right compromises that are fragmented across all of the source. Once that source exceeds the ability for someone to easily remember it, and move around it seamlessly, then attempts to fix or improve it will slow it down. They don’t just gradually slow down, they basically walk off a cliff. Changes that might have required a few hours suddenly balloon to taking days or weeks. As well, the quality of the changes declines, it then feeds back into a cycle making all future changes harder as well. Parts of the code hit this vicious cycle at different times in the descent, but the chaos in one part of the system spreads.
If a project has started swirling downwards this way, the only real way out of this is to admit that the code has become a huge mess. That the work necessary to stop this decay is essentially re-organizing huge swaths of the code base with an eye on defragmenting it and laying down some rather strict organization. That kind of work is really slow. You can’t just leap into it, rather it takes time and effort to diligently, slowly, rearrange it at multiple levels, back up to a place where it can move forward again.
But just cleaning up the code isn’t the only issue. Over specific code and redundant code, both occur in projects that are falling apart. Both of these problems need addressing.
Redundant code is easier since you can just look around the codebase and find a lot of similar functions or config data. If they are close enough to each other, it is an easy change to merge them together and only use one copy. Again it is slow, but if done well, it has a huge lift on the quality and tends to make a lot of bugs just disappear, so its payoff is obvious.
Over-specific code is a little harder to tackle.
Most development projects go on for years, if not decades. During the life of the project, the focus generally starts at one specific point in the domain and spreads, much like construction, over the surrounding areas. When the project is started, people are often tunnel-visioned on that initial starting point, but taking a big project one tiny step at a time is painfully slow.
Instead of pounding out code for each specific case, when there are a lot of them in the same neighborhood, the most optimal method over the total lifespan of the development is to batch these cases together and deal with them in one large blow. This is an off-handed way of talking about generalization and abstraction, but it gets the point across that picking the right abstraction that repeats over and over again in the domain, will accelerate a project through that territory at light speed. The cost is often just marginally more effort in the beginning, but the payoff is massive.
Often the counter-argument to the above is that it is impossible to guess at which problems will repeat significantly enough in the domain to make a reasonable guess at what to abstract. However, that’s never been a valid point, in that software starts at a particular spot, and the functionality spreads from there. It is all connected. One doesn’t start writing an accounting system and end up with a video game, it doesn’t work that way. So the possible paths for the code are connected and intertwined, and for the most part, obvious. Given that, some prior experience is enough to lay out the pathways for a very long time, with reasonable accuracy, subject to radical shifts like a startup that pivots. So basically if you get in some experienced people, along with their effort, you get their foresight, which is more than enough to be able to maximize the efficiencies over a long period of time.
On top of all of these other efforts, hard lines need to be drawn through the code to separate it into smaller pieces. Without that separation, code gets fragmented quickly and the problems come right back again.
The test that the separation is good enough and clean enough, is that it can be easily documented with fairly simple diagrams. If the sum of the details is so messy that it is a Herculean job to create an accurate diagram of the parts, then it is disorganized. The two go lockstep with each other. If you can create a simple diagram that only kinda reflects the code, then quite obviously you want to refactor the code to match that diagram.
These lines then form an architecture, which needs to be preserved and enhanced for any and all future extensions. Although it should be noted that as the general size of the code base grows, it is quite common for it to outgrow its current architecture and then need to be entirely reorganized by a different set of parameters. That is, no specific organization of code is scalable. It is all relative to its size and complexity. As it grows, then any organization needs to change with it.
Given the above issues, it is inevitable that during the life of a coded base there are times when it can run quickly without any consequences and times when everyone has to slow down, rearrange the project, and then start working through the consequences.
Avoiding that will bring on disaster faster than just accepting it and rescheduling any plans that are impacted. If a project decays enough, it reaches a point where it can not move forward at all, and is either permanently stuffed in maintenance mode or should be shelved and started over from scratch. The only problem with restarting is that if the factors that forced it into that cycle in the first place are not corrected, then the next iteration of the code will suffer the exact same problem. Basically, if the developers don’t learn from history, then they will absolutely repeat it and get back to the same result.
