Agility

The original premise behind ‘agility’ is that the path you choose can change radically, at any time.

For software development, the big problem with older waterfall ideas is that they would ‘lock in’ a direction that often stayed the course for years. It was a long-term approach, which needed significant foresight in order to get it right. Foresight is a rare quality ...

So, rather than set a course and stubbornly stick to it, particularly in the face of evidence that it is invalid, the entire development team would instead be highly reactive and weave left or right as they encountered new obstacles. The manifesto set down four basic priorities for how to do this sanely.

For a startup that is essentially throwing darts at a wall to see what sticks, this makes a whole lot of sense. Why commit to a lot of serious work, if you aren’t sure whether or not the thing that you are building will actually be monetizable? So it’s 1. throw something together, 2. see if it gets any traction, and 3. if not throw it away and start again. Rinse and repeat, until you finally hit something that is sticky.

What they don’t say about startups though, is that that is just the first version. Then you throw that all away and build a second one for the early clients. Then you throw that all away again and build the real version. That is, extreme agility is just an early phase in the company. It doesn’t last long, and later you have to be smart and wipe out all of the technical debt that you accidentally created by acting that way.

While being agile makes a whole lot of sense for early startups, the idea of formalizing it as a methodology and using it to work on all medium, large or huge systems is, well, insane.

The first and most obvious point is that ‘formalizing it’ is also de-agiling it.

For example, if you come together some mornings to quickly discuss stuff as a group, the causal nature of that arrangement is very different from having a formal standup ‘scrum’ every morning. Getting strong communication within a ‘team’ is essential in making it effective, but mandating that with a variant means of communicating to a ‘group’ of people is not the same thing. In one case it is entirely spontaneous, and the participants have control over what they are doing. They choose to meet. But when you take away their ability to choose for themselves, the formality itself is the polar opposite of being agile. It is rigid. That is, they can no longer choose to not meet, for instance. If someone has some long-running work and they need to concentrate on that, instead of making up things to talk about, they might just need to drop out of the discussion for a while. That is okay, and it should be up to them. But it’s not. Now it is formal, now it is a job requirement, now it is a pain. And now it is no longer being agile, it’s just another box that needs to be checked off. Something you ‘have’ to do.

Ironically, a checklist for setting up an agile project is not a very agile way of setting up a project. You should just jump in blindly and deal with all things as they come. It would be helpful when issues occur to be able to see how others have handled the same problem in the past, but if you really are agile, you always have the option to do it in any way you see fit. It occurs, you react with ‘your’ best efforts. It’s not formal, it’s not documented, and the process is that there is no process. You don’t have a methodology, instead, you have a cookbook whose recipes might save you some time and grief, or not.

So, the second point is that if what you are trying to do is be reactive, then you need to leave enough room to actually be able to react at a moment’s notice. If you have scheduled work items that “must” be completed right now, then you can’t just drop them all and do something different.

But that was the original idea. You start working on something, realize that it isn’t going to have enough value, and toss away the work. You toss it ‘early’ so that you minimize the wasted effort, rather than stubbornly digging in until the end, only to have it blow up because it’s all useless now. The key thing to understand is that you did the wrong thing, that you wasted time. The only reason you messed up though, is because you didn’t know what the right thing was when you started. You had no choice but to blunder through it and see what happens. Live and learn.

If you do know what the right thing is, and you are sure you are absolutely correct, getting there by wobbling all over the place and wasting time is not reasonable. You can go straight from A to B. No detours, no reactivity. You have work to do, so just do the work. Not agile, because it isn’t necessary.

The modern Agile movement plays on the fact that most programmers are inexperienced, and that the users don’t want to spend time thinking deeply about what they need to solve their problems. Ironically, the answers are usually there and most of what they need is rather obvious. It’s just a lot of stuff, and it takes a lot of time to collect it, put it together, organize it, and then feed it into a plan to get some long-term work completed. So, oddly, it’s the horizon itself that is vague. Often because the people who understand the problem are too busy and the people building the solution are too rushed, and everyone is impatient.

But the horizon doesn’t change. It itself is often a very slow-moving target. For a large company, the lines of business have been going on for decades, if not centuries in some cases. Even if they are intrinsically super complex domains, the core parts of them are understood well enough to make the line of business profitable. And to clearly highlight the inefficiencies and blockers there as well.

So it’s absolutely and entirely the opposite of a startup trying to guess at what might be viable. It is viable, it has been known to be viable for a long, long time. It’s just that in working in that line of business, there are lots of problems that a computer can help with, and different people who vaguely understand different parts of those problems or parts of how to solve them.

That is, after years of working in a startup, you probably wouldn’t have been able to guess where it all ended up. After years of working within a larger business domain, it’s probably not a surprise where it ended up.

There are, of course, time periods where the business is undergoing some disruption, but the forces causing that are external by definition. That is, some startup gets traction and knocks an older, stable company off its course.

That startup needs to be agile, but the company isn’t going to get back on course by pretending to be agile. It’s not a rational expectation. They just need to play out the old line of business to the end, while simultaneously dipping into the new one. But most likely if the disruption is major, they’ll just purchase the next generation, not try to magically divine it themselves.

Which goes back to saying that for most existing applied software development there is usually no real rational reason for it being reactive. The market moves, there are changes needed, but the shifts are fundamentally predictable or the line of business wouldn’t be profitable. So the reason for an existing company adopting or trying to be agile isn’t because of the business, it is because of the people involved and how they don’t want to accept what they are doing; either because of inexperience or lack of focus. Everything is less intimidating if you don’t have to plan it out in advance.

But, instead of trying to be agile, it would be far better to focus on building up the experience, knowledge, foresight, and skills around the technologies and the domain in order to reduce that long-term vagueness.

If what you do is entirely unknown then being reactive is really your only option. But it is also inefficient, stressful, and certainly is not guaranteed to be successful. It’s just what you have to do now, that is all. But if you have a choice, and you are doing the same things over and over again, then organization and planning can help you get them done in the most efficient and smooth manner possible. If your real problem is that people don’t know what they are doing and are too busy to try and figure it out, then it would be far better to try and solve that issue, instead of adopting a philosophy of just trying to ignore it. It’s not going to go away, it’s costly, and it will always get worse, not better.

Building a big software system takes a long, long time; it is a huge commitment. It is a pain and it is also extremely risky. There is no way to ensure success, but pretending like a short-term reactive approach will increase the odds of winning isn’t a viable strategy either.

Pretty much we need to grow large systems, so that part of agile makes a lot of sense, but ignoring that there is an obvious set of fixed directions to grow and instead, trying to pretend to be super reactive is just going to let the thing spin out of control. There are points in a large development that need to be made flexible, for instance, you can’t solve all of the problems at the same time, it would spread the effort too thin. Things do change, but if they are changing unpredictably that is an analysis problem, not a process one. So there is a necessity for a long-term strategy to help cover as much ground efficiently as possible, while also adjusting the tactics along the way. You have to find a balance between these two extremes.

The really big underlying problem in software is the desire for everyone to try not to think about things by making them static and unwavering. That is, everyone is constantly looking for a way to cut down on the cognitive effort necessary to build complex stuff. So, they latch onto strict, limited, inflexible rules and one-size-fits-all approaches. These don’t work very often.

But the real answer is that you need to leverage any earlier cognitive effort as much as possible. That is, you can’t ever actually avoid deep thought, but you can get more out of having spent time doing it. What you believe you’ve avoided just ends up biting you later.

Getting back to Agile, it’s become a lightweight, rather static methodology that claims to be flexible enough, but its own formalization negates that. It claims that if you follow it, you don’t have to think about stuff, you should just blindly go wherever the stakeholders lead you. But ironically, the stakeholders, as I said earlier, can’t necessarily dispel the vagueness of the solution anymore than an army of mindless coders can. So the most likely outcome is that everyone marches around in circles; which in the past is what we liked to refer to as a ‘death march’. It’s just that with waterfall, the death march knew what goal it was failing to accomplish, while with agile it doesn’t even have a goal.

Clarity

If you hire someone to build stuff, then you want to let them do their job properly, otherwise, it is not a surprise when the things they build do not work correctly. It would be expected.

Computer Programmers write code. Before they start this work they should at least put together a low-level design, as it tends to save a lot of time during the coding and testing parts.

A low-level design comes from the mid-level design, which fits into the higher-level designs. Designs set the constraints and keep everything organized. They match the solution to the problem. If you don’t have any designs, what you will get is just a pile of disconnect stuff, that rather obviously won’t work very well.

For small, or even medium-sized projects, some programmers can visualize the design in their heads and go from there. It’s not a super common attribute, but if they can “see” the work, they can build it. This might confuse people, who falsely assume that there was ‘no design’, but its lack of documentation does not indicate that it was never done.

Programmers tend to focus on the workflow from low-level designs, coding, testing, bug fixing, and some involvement in deployment. But that is a subset of what needs to happen. So, the border category of Software Developers spans the full range, all of the way out from conception, analysis, high-level designs, politics, funding, management, process, methodology, to any operational issues and getting feedback on the suitability or stability of the system.

If the project is medium or larger, you need at least one experienced software developer to lead the way. They have to have deep experience (10+ years) with all 5 stages (analysis, design, coding, testing, deployment) and be able to make sure that the end-to-end flow of the work is reasonable.

Even though a Software Developer gets involved in operational issues, they are not an operations person. They are there to pick up requirements, test suitability, figure out the technical competencies, diagnose serious problems, etc. But they are not 24/7 support. They are not system administrators, database administrators, support people, helpdesk, or operators. They build stuff, it is up to someone else to ‘operate it’.

When you ‘run’ a system, you need a bunch of people to monitor it and keep it working properly. Newer technologies like ‘the cloud’ don’t change that, not even a little bit. If you have a bunch of systems that need to be up 24/7, you need a bunch of support specialists who ensure that things are up 24/7. Nothing can get away from that fundamental issue. If the system is low quality, they need to know a lot more in order to keep it running, making them more expensive and harder to train properly. So better quality in the code is generally a good way to reduce operational costs.

Failing to understand what the different roles should be doing will end up blurring the tasks and responsibilities, which ultimately results in very low-quality development work. A programmer that is burnt out from late-night operational issues, is not someone who should be crafting code the next day, or even that week. Burn the ‘builders’ and what they build going forward will be worse, not better. To build good stuff you need to concentrate rather deeply. Noise, stress, rushing, anger, etc. are all counter-productive.

There is a minimum threshold of quality that enables the software to have enough value to be able to pay for itself. If the work falls below that level, it becomes a cost sink. That is, it is wasting more money than it is enabling. It is possible for the initial quality to be good, but over time for it to degrade to a negative status, and eventually, it all turns red, So, you can see ‘profitable’ projects that while still sitting on a lot of initial value, have reserved their direction towards becoming cost sinks. It happens all of the time.

If the project started out successfully but then turned into an epic disaster, it is not an easy choice to shelf it and start over again. The risks of a new project are significantly higher than the risks of reorganizing and refactoring. That is, the least risky way of getting large and huge systems is to grow them. But to grow them you have to have the foresight and keep control over the technical debt. If they become disorganized messes, the work will grind to a halt. These are nebulous concepts that are fundamentally difficult to understand, which is why you need an experienced Software Developer at the helm.

Vertical and Horizontal

 There is an overall solution space. It is the full scope of things that are possible to build that will use software to solve a specific domain (often business) problem. 

These days, in most organizations, the demands for new development work most often come in as top-down requirements from the domain, users, or directly from some change in a line of business. It is vertical, which is why it often ends up in silos.

But the construction of the system should really be driven horizontally. That is, you need to get as much reuse with similar code in all of the systems as possible; you are looking for commonality as it crosses different boundaries. Technical issues like authentications, logging, and users are the same everywhere. The setup, configuration, building, and deployment are all the same too. It doesn’t make sense to keep reinventing these all too common wheels.

Sort of looks like this:

At least 50% of most systems development is base infrastructure. Just a platform on which the functionality will sit and perform properly. Often it can be higher than 80%. It changes somewhat by tech stack, and maybe a little for specific domain issues like highly restricted data, but otherwise, it is always the same. 

Similarities most often occur with the interface and the persistence. But what is also very common is the same types of basic and general domain-specific data. Vertical silos are very rarely independent, they overlap as messy hierarchies naturally form everywhere in large organizations. So, you see lots of similar features or systems sharing the same basic data, with the same computations, at least they should all be the same.

The vertical time horizons are usually short-sighted. They are reactive to the needs of the stakeholders and are often too busy dealing with old existing problems to be able to lay out longer or more effective options.

The horizontal time horizons are more similar to long-term issues like physical office space. They are infrastructural, slow to accumulate, expensive, and need continuing maintenance. They take foresight, so they are far easier to ignore. But the costs of ignoring them are a crazy large amount of make-work and a lot of impedance mismatch problems (functionality that should be identical but isn’t).

It gets a little more confusing when you realize that most methodologies, processes, and their tools ignore the horizontal axis as well. They can’t deal with two sets of orthogonal drivers, so they just default to top-down, and help reinforce all of the wasted effort. The controls that we are using to better ensure project success are the same ones that are guaranteeing that lots of work will be reduplicated. That lack of time will drive extreme technical debt.

It is hard though, to try to go something reasonable on the horizontal axis, while also satisfying the vertical one. Realistically, it would mean dividing up the resources between the two, then trying to ensure that they intersect as often as possible. It can be done, but it is a lot rarer than just blindly pushing everything top-down. However, when it’s been done, it has amplified the value of the work considerably. That is, it may be hard, but it is also worth it.