Scaling Methodologies

As it usually does, the ultimate quest for a perfect one-size-fits-all methodology has failed. This is a timeless statement, in that it doesn’t matter when or which methodology we are talking about. The quest always fails.

Why?

Because the size of a project dictates the rules and processes you need to keep it running smoothly.

A small project that is chaotic is barely distinguishable from one that is super organized, except it might finish slightly later. It’s small, failures are fine and easily recoverable. So, you want something ultra-lightweight, really easy. The work only goes on for days or weeks. It can be totally reactive. Try stuff, demo it, repeat.

But that changes with a medium project, whose duration is months. You need something a bit heavier.

If you need better quality, you need even more weight. You need to formalize more parts of the work. Mostly, it can still be dynamic, and somewhat reactive, but proactive plans increase efficiency, by avoiding redundancies and batching up stuff.

You might have to worry about staff turnover, some parts of the work should be documented just in case. Sometimes the months turn into years. So long as the codebase remains small, you can stay with the medium process.

Breaking medium projects up into lots of small projects won’t work because they are still dependent on each other. You can’t pretend that half a medium project is a small project, it is not. It is a part of a medium project. If you treat that half incorrectly, it will get out of control. You are often bound by the higher attributes of any dependency unless it's been almost fully encapsulated, like an app in a platform. Still, even in those cases, the larger dependency taints the smaller one.

Once you cross the line and the scale becomes large everything changes again. As well as the codebase organization, the process has to change as well. A lot more stuff needs to be formalized, more people added, and more communication. Way more details flying around.

When this jump occurs it is important to understand that the code needs to be refactored, or even sometimes, fully rewritten. Large projects are a pain. They require very different organization than medium ones. There are at least a dozen people involved in some way, usually a lot more, even if there is still only one coder. The code grows, the impact grows, and the consequences of bugs and failure grow. Projects that evolve eventually cross this threshold and they really need to be revamped. It is easier if it was greenfield, which is usually handled far better.

A wrong turn or dead end is really expensive now, so planning is super necessary, as well as techniques like prototypes, scale models, etc. You can’t just do stuff and see it works later, you need to be far more sure that the changes add value, and solve real problems. Everything needs to be explicit and formalized. Being reactive is pretty bad now, most stuff except production outages has to be planned out in advance; the whole ship turns very slowly.

Large systems tend to integrate with lots of other stuff around them. Letting those connections become spaghetti is a pretty serious mistake. Keeping them organized can push up the scale even higher.

At the next level, huge projects involve hundreds or even thousands of people, they go on for years or decades. Huge projects need an amazing amount of coordination, they are exceptionally slow-moving. No clever ideas will negate that. If there are billions of details, at some point, you have to accept that. Just keeping something huge headed off in a sane direction is a big problem all on its own, it doesn’t matter what you are trying to build.

The size of the project drives the budgeting, the expected lines of code, the methodology, the number of people, the tech stacks; everything. Four categories: small, medium, large, and huge, are usually enough to get across the effects of size. It’s basically an exponential scale, doubling at every new interval, which roughly matches with complexity growth. That is, some projects may feel linear with respect to time, but the dependencies tend towards a worse case of exponential growth, so we should treat it as exponential.

Software development projects are highly multi-dimensional, so it is important to not lose sight of the goal. You are assembling a collection of features that make it possible for users to solve their problems. So getting the correct implementations of those features into the hands of the users is a success. Broken implementations or missing features are not a success. A good project does this smoothly, over and over again.