Agility

The original premise behind ‘agility’ is that the path you choose can change radically, at any time.

For software development, the big problem with older waterfall ideas is that they would ‘lock in’ a direction that often stayed the course for years. It was a long-term approach, which needed significant foresight in order to get it right. Foresight is a rare quality ...

So, rather than set a course and stubbornly stick to it, particularly in the face of evidence that it is invalid, the entire development team would instead be highly reactive and weave left or right as they encountered new obstacles. The manifesto set down four basic priorities for how to do this sanely.

For a startup that is essentially throwing darts at a wall to see what sticks, this makes a whole lot of sense. Why commit to a lot of serious work, if you aren’t sure whether or not the thing that you are building will actually be monetizable? So it’s 1. throw something together, 2. see if it gets any traction, and 3. if not throw it away and start again. Rinse and repeat, until you finally hit something that is sticky.

What they don’t say about startups though, is that that is just the first version. Then you throw that all away and build a second one for the early clients. Then you throw that all away again and build the real version. That is, extreme agility is just an early phase in the company. It doesn’t last long, and later you have to be smart and wipe out all of the technical debt that you accidentally created by acting that way.

While being agile makes a whole lot of sense for early startups, the idea of formalizing it as a methodology and using it to work on all medium, large or huge systems is, well, insane.

The first and most obvious point is that ‘formalizing it’ is also de-agiling it.

For example, if you come together some mornings to quickly discuss stuff as a group, the causal nature of that arrangement is very different from having a formal standup ‘scrum’ every morning. Getting strong communication within a ‘team’ is essential in making it effective, but mandating that with a variant means of communicating to a ‘group’ of people is not the same thing. In one case it is entirely spontaneous, and the participants have control over what they are doing. They choose to meet. But when you take away their ability to choose for themselves, the formality itself is the polar opposite of being agile. It is rigid. That is, they can no longer choose to not meet, for instance. If someone has some long-running work and they need to concentrate on that, instead of making up things to talk about, they might just need to drop out of the discussion for a while. That is okay, and it should be up to them. But it’s not. Now it is formal, now it is a job requirement, now it is a pain. And now it is no longer being agile, it’s just another box that needs to be checked off. Something you ‘have’ to do.

Ironically, a checklist for setting up an agile project is not a very agile way of setting up a project. You should just jump in blindly and deal with all things as they come. It would be helpful when issues occur to be able to see how others have handled the same problem in the past, but if you really are agile, you always have the option to do it in any way you see fit. It occurs, you react with ‘your’ best efforts. It’s not formal, it’s not documented, and the process is that there is no process. You don’t have a methodology, instead, you have a cookbook whose recipes might save you some time and grief, or not.

So, the second point is that if what you are trying to do is be reactive, then you need to leave enough room to actually be able to react at a moment’s notice. If you have scheduled work items that “must” be completed right now, then you can’t just drop them all and do something different.

But that was the original idea. You start working on something, realize that it isn’t going to have enough value, and toss away the work. You toss it ‘early’ so that you minimize the wasted effort, rather than stubbornly digging in until the end, only to have it blow up because it’s all useless now. The key thing to understand is that you did the wrong thing, that you wasted time. The only reason you messed up though, is because you didn’t know what the right thing was when you started. You had no choice but to blunder through it and see what happens. Live and learn.

If you do know what the right thing is, and you are sure you are absolutely correct, getting there by wobbling all over the place and wasting time is not reasonable. You can go straight from A to B. No detours, no reactivity. You have work to do, so just do the work. Not agile, because it isn’t necessary.

The modern Agile movement plays on the fact that most programmers are inexperienced, and that the users don’t want to spend time thinking deeply about what they need to solve their problems. Ironically, the answers are usually there and most of what they need is rather obvious. It’s just a lot of stuff, and it takes a lot of time to collect it, put it together, organize it, and then feed it into a plan to get some long-term work completed. So, oddly, it’s the horizon itself that is vague. Often because the people who understand the problem are too busy and the people building the solution are too rushed, and everyone is impatient.

But the horizon doesn’t change. It itself is often a very slow-moving target. For a large company, the lines of business have been going on for decades, if not centuries in some cases. Even if they are intrinsically super complex domains, the core parts of them are understood well enough to make the line of business profitable. And to clearly highlight the inefficiencies and blockers there as well.

So it’s absolutely and entirely the opposite of a startup trying to guess at what might be viable. It is viable, it has been known to be viable for a long, long time. It’s just that in working in that line of business, there are lots of problems that a computer can help with, and different people who vaguely understand different parts of those problems or parts of how to solve them.

That is, after years of working in a startup, you probably wouldn’t have been able to guess where it all ended up. After years of working within a larger business domain, it’s probably not a surprise where it ended up.

There are, of course, time periods where the business is undergoing some disruption, but the forces causing that are external by definition. That is, some startup gets traction and knocks an older, stable company off its course.

That startup needs to be agile, but the company isn’t going to get back on course by pretending to be agile. It’s not a rational expectation. They just need to play out the old line of business to the end, while simultaneously dipping into the new one. But most likely if the disruption is major, they’ll just purchase the next generation, not try to magically divine it themselves.

Which goes back to saying that for most existing applied software development there is usually no real rational reason for it being reactive. The market moves, there are changes needed, but the shifts are fundamentally predictable or the line of business wouldn’t be profitable. So the reason for an existing company adopting or trying to be agile isn’t because of the business, it is because of the people involved and how they don’t want to accept what they are doing; either because of inexperience or lack of focus. Everything is less intimidating if you don’t have to plan it out in advance.

But, instead of trying to be agile, it would be far better to focus on building up the experience, knowledge, foresight, and skills around the technologies and the domain in order to reduce that long-term vagueness.

If what you do is entirely unknown then being reactive is really your only option. But it is also inefficient, stressful, and certainly is not guaranteed to be successful. It’s just what you have to do now, that is all. But if you have a choice, and you are doing the same things over and over again, then organization and planning can help you get them done in the most efficient and smooth manner possible. If your real problem is that people don’t know what they are doing and are too busy to try and figure it out, then it would be far better to try and solve that issue, instead of adopting a philosophy of just trying to ignore it. It’s not going to go away, it’s costly, and it will always get worse, not better.

Building a big software system takes a long, long time; it is a huge commitment. It is a pain and it is also extremely risky. There is no way to ensure success, but pretending like a short-term reactive approach will increase the odds of winning isn’t a viable strategy either.

Pretty much we need to grow large systems, so that part of agile makes a lot of sense, but ignoring that there is an obvious set of fixed directions to grow and instead, trying to pretend to be super reactive is just going to let the thing spin out of control. There are points in a large development that need to be made flexible, for instance, you can’t solve all of the problems at the same time, it would spread the effort too thin. Things do change, but if they are changing unpredictably that is an analysis problem, not a process one. So there is a necessity for a long-term strategy to help cover as much ground efficiently as possible, while also adjusting the tactics along the way. You have to find a balance between these two extremes.

The really big underlying problem in software is the desire for everyone to try not to think about things by making them static and unwavering. That is, everyone is constantly looking for a way to cut down on the cognitive effort necessary to build complex stuff. So, they latch onto strict, limited, inflexible rules and one-size-fits-all approaches. These don’t work very often.

But the real answer is that you need to leverage any earlier cognitive effort as much as possible. That is, you can’t ever actually avoid deep thought, but you can get more out of having spent time doing it. What you believe you’ve avoided just ends up biting you later.

Getting back to Agile, it’s become a lightweight, rather static methodology that claims to be flexible enough, but its own formalization negates that. It claims that if you follow it, you don’t have to think about stuff, you should just blindly go wherever the stakeholders lead you. But ironically, the stakeholders, as I said earlier, can’t necessarily dispel the vagueness of the solution anymore than an army of mindless coders can. So the most likely outcome is that everyone marches around in circles; which in the past is what we liked to refer to as a ‘death march’. It’s just that with waterfall, the death march knew what goal it was failing to accomplish, while with agile it doesn’t even have a goal.