Realistically, we need methodologies, even if we don't like to be confined by them. Too many loose strings in a project will eventually tangle.
Which is why I liked the emerging lightweight ideas at the turn of the century. But I found that over time their intent turned away from what I thought was reasonable.
So, I like to propose my own methodology. I guess I’ll call it “Adaptive”.
I realize that the word ‘adaptive’ is already taken for a specific type of ‘project management’, but this methodology is somewhat aligned to that, so they are complementary, while still very different. This is about the act of building complex things, not about the show that surrounds it.
The methodology starts as lightweight, but as quality or funding constraints are needed, it gets heavier. So, it’s almost weightless if you need to whip out a demo, but it’s pretty darn heavy if the code will end up in a medical device.
It values quality over quantity. That is, the intent of a software development project is not to create a mass of code; rather, it is to get the best code you can to solve the problem as precisely as you can. If the solution needs 150K to work properly, then the closer you get to that size, with enough quality to keep it useful, the better it is going.
It values organization over speed. No matter what you create, if it ends up in a giant pile of disorganized stuff, it is useless. Organization is, at minimum, having a place for everything and then putting everything in its place. The development and operations of the solution both need to be kept organized. A big part of organization is knowing where to put things, so there needs to be enough effort devoted to that to allow lots of people to know where things go.
It values engineering over evolution. That is, if there is a state of the art for implementing some specific part of the solution, then the people involved will go out and learn more about those prior efforts. They won’t reinvent the wheel just to save a little bit of time. Crude is not good. They know that the knowledge to do the work is out there, and they’ll attempt to find it first. Once they know, they are free to bend it to their will, or extend it, or even go off in a different direction, but first they have to know.
It values fit over ease of effort. If there are two ways to implement something, a simple one and a complex one, the best implementation isn’t driven by the technology or by the developers' preferences; rather, it is driven by which choice makes the solution the best fit for the users. It’s from their perspective, not the people building the solution. The point of software is that it gets used, so degrading its usefulness to save a little time or concentration doesn’t make sense. What is done is always done to make the users' lives better, even if it is hard and painful for the developers.
As for the specifics, since there are two main axes to the effort, there are two main roles for any project.
On the solution side, there is a lead developer. An expert in software development. They are fully responsible for the development of the software. They have a long, extensive background in building stuff that was complicated enough to act as a reasonable solution for similar problems. It’s not about energy or intent, but rather it is about having been through work in the past that is close enough to what needs to be done now, so that they will make the best choices for the project to keep it from derailing.
On the problem side, there is a domain champion. An expert in the domain problems. They are the person who is managing all aspects of the development from the outside and has considerable expertise in the problem itself. In an enterprise, they are at least a PM / BA combination with actual real-world experience in the business. They have spent time in the domain trenches. They have to have fairly deep knowledge in the domain and its practices, since this is where the details get difficult. They handle the external issues.
The domain expert is responsible for and the expert on all aspects of the problem. They know what the related problems are; they understand them. They have some idea of how to correctly deal with aspects of the problem, but they are not experts in solving it. So they know what the problem is, and have some ideas about how they think a computer might help deal with those issues. They control or have raised money to get the problem solved.
The developer is responsible for and in control of all aspects of the solution. The internals of development. If the solution does not correctly solve the problems, it is their fault. They know how to build it, how to deploy it, and how to keep it running over time. They understand how well it actually fits into the problem space. That is, they can’t just build a generic app and claim it ‘could’ solve the problems; they have to build something that ‘does’ solve the problems, so they can’t ignore understanding the problems themselves.
In that sense, the domain expert sets out a series of features they would like to see in the solution, while the software expert implements these as functionality. It’s a partnership. The software expert understands a lot of the nature of the problems that drive the solution, while the domain expert understands the limits and costs of technologies. But the domain expert has the final word on all domain issues, and the software expert has the final word on all technical ones.
The domain expert can say “your proposed solution is not actually solving our problems”, and the software expert can say “at best we can only approximate a solution to that given problem”. The path forward lies between these two points.
As for specific development processes, it is incumbent on the software expert to keep in constant communication with their own development team. How they choose to do that is up to them. Maybe it’s frequent meetings, or one-on-ones, or some combination. These are internal development product conversations; they should be private away from the domain conversations. What’s important is that they are constantly and actively communicating. The development team cannot be left in the dark, and if they are stuck on something, they need to raise it to the lead immediately.
If the internal project communication breaks, the project will derail.
The software expert and the domain expert are always working together. Neither the problem aspects nor the solution aspects have priority over each other. The idea is that they both need to be balanced. Part of the problem is often ‘time’. If a solution arrives too late, that is a big deal. But, unfortunately, since most development work is iterative over years of effort, prioritizing the correct order of the development work is on the software side. This is a contradiction that both sides have to resolve. It is necessary to avoid wasting time doing the wrong work at the wrong time, while still being timely. So, the two experts are in constant negotiation.
Small ambiguities or misunderstood domain and technical issues pop up on a regular basis. It’s normal. The sooner they are pushed over the fence, the better. While this methodology is not reactive, it does prioritize timeliness; but both sides should try to get ahead of the ball wherever possible. That is, the work may suddenly need to change in response to the ground shifting. It is not appropriate to sit and wait on a showstopper, for example, as the effort going on in the meantime may just become makework.
In very small projects, the lead developer might also be the domain expert, but only if they actually have the background to know enough about the domain. That is, they can’t be a faux domain expert; they have to be a real one. They don’t have to know the total full width of the domain, but they do have to know the width of the domain problems they are intending to solve.
The converse is not true. A domain expert with no significant technical background, given the scale of the project, cannot act as the lead developer. Whacking out some Excel spreadsheets or demos does not qualify someone to run a medium-sized development; the consequences will be disastrous.
A project gets started with the intent to solve a problem. Thus, the domain expert is there from day one. Once the project is in motion, however, the technical issues tend to slowly dominate. Most often, it is difficult to prioritize which features get done first, given whatever the limited time constraints.
Once there is an existing codebase, it has a number of deficiencies, and there is also some desire to grow certain parts of the solution. It’s a bit easier. But the effort slows down, as the work itself should get better; more precise.
There is always a long-term plan, as there is a defined problem, and an expectation that a specific collection of features will act as a reasonable solution to solve them. The initial round of development may only be scheduled for six months, but the long-term plan may extend out five or more years. Both should be discussed; both should be on the table. The value in any software solution comes not from its existence, but from its ability to be there, over and over again, as you need it. That is, even if the initial version is crude, if it does have value, it will gradually evolve into something significantly more complex. If it doesn’t have that value, it was probably a waste of time.
That there is both a short-term and a long-term for all projects is important to keeping the effort organized and to ensuring that the work overall is as effective as it can be. Which is why any sort of work compression like ‘reuse’ or ‘abstraction’ is critical. Once some work is completed, you’ll need to leverage it later, not redo it.
Any and all codebases will eventually reach a maturity point where the effort to extend them is so slow and painful that the life of the codebase is effectively frozen. As well, all codebases need constant maintenance, since the underlying technology is always changing at a crazy rate. If the work gets into that ugly ball of mud stage too early, then it will suffer a premature death. It will waste a lot of money; cost way more. Obviously, for many little demos this is not important, but for any medium-sized system or larger that lots of people rely on, it is highly damaging. If a line of business is reliant on the software, it is extraordinarily destructive.
At the core, though, writing software for the sake of writing software may be an interesting performance art, but it is not the intent of any of these larger development efforts. For these, the software is written explicitly as a solution to some real-world problems. So the code that goes into that solution must fully solve the problems and match their complexity. You shouldn’t over-engineer, but you also shouldn’t oversimplify.
Development projects are inherently iterative. That is, it is a never-ending discrete set of development efforts that can go on for years, or even decades. The frequency of demos and releases is driven by an external timetable and is often influenced by issues such as confidence and trust, particularly in the early stages of development. This means that the length of time for each round of development is variable and shifts. On the one hand, if the release periods are too short, there is more work involved because work is done out of sequence and more likely to be disorganized, but if they are too long, the funding or interest may fade. So, there is no perfect immutable length of time for each iteration. For big work, longer is better; for maintaining trust, shorter is better. On top of this, most funding comes with significant strings, so there is some minimal amount of tracking that is also necessary to maintain confidence. This is not extra work; both the technical and domain sides need to be confident that the big expense of development is worth the risk.
But it’s worth noting that excessive tracking will gum up the effort and the morale. Nobody likes being micromanaged, and any attempt to track the time for each and every little task is a classically bad form. The technology lead needs to keep presenting evidence that the work is progressing as expected, and they need to be diligent about raising any potential show stoppers early and often. The domain expert needs significant patience and to understand that even with the best of specifications, development teams, and technologies, there are still a great deal of uncertainties that may cause significant disruptions, particularly if the pace of development has been rushed.
Overly frequent tiny releases are both expensive and disruptive to the users, so they are not a good thing. People don’t need a new version of an app with just 200 lines of new code to fix minor, mostly unencountered problems. Unless it is an emergency, new releases should at minimum provide revised functionality or infrastructure shifts. That is, most of the time, a user should see some obvious changes in each release. If it’s not visible to them, and they have not been affected, it is just a disruption, not a positive.
Some iterations should always be pure cleanup. The bane of software projects is disorganization. As the project grows, its size invalidates earlier organization efforts. So, even regular consistent cleanup attempts will not be enough to keep the code from reaching the end of its life. Thus, organization itself is both the primary feature and main functionality of every software project. If it’s not there, then the value of the effort is extraordinarily low.
Even for the best lead developers and domain experts, there will be plenty of unknown unknowns. Which means that throughout the entire life of the project, there will be new and better information coming in which needs to be acted on. That is, while we know the technology itself is always changing, the understanding of the domain problems is often in flux as well. Managing change is essentially part of managing the development complexity of any software. Expect change, embrace change, and try to spend as little time as possible fighting about when those changes get reflected properly in the software.
As well, everyone involved in any software development project will grow. They will learn more, they will understand more, and many of the details will emerge. Even for late-stage development where the domain is nearly fully covered, the depth of what is understood is bottomless. Working on software is an exercise in constant learning; it never ends.
In some development efforts, there are significant trust issues. When this is the case, then all major decisions need to be papered, and they need sign-offs. If there is trust, particularly between the lead developer and the domain expert, then it can be bypassed. But if neither side insists on sign-offs, then both are equally guilty later when there is a disagreement. If the sign-off isn’t papered, it can be verbal. If it didn’t happen, then one side moved forward without informing the other. So, you can work on a verbal set of features, functionality, specifications, design, etc. only if both sides are in full trust mode. But if it results in a disruptive disagreement, the weight has to change going forward, and the decisions have to be written out in an email, a document, or any other medium that allows for one party to sign off on the terms of the effort.
It is trust that allows a methodology to be lightweight. If that is broken, it can no longer be lightweight and needs to be as heavy as the conflicts that stand in between the communications. That is obviously costly in time, so trying to stay as lightweight as possible is more effective.
It should also be noted that quality comes from not rushing through the work. So, if enhanced quality is a part of the requirements for solving the problem, then it necessarily slows down the effort. Basic quality for a rushed software project is a pretty low bar. Somewhere around expecting monthly big failures, if not weekly. If that is too much, the development effort needs to be slower; more tracking or oversight will only make the issues worse.
Quality control is an essential part of all software development. Since coding is highly mistake-prone, it is critical to ensure that all code is always properly tested before getting into releases. No untested code should ever end up in the operations environment; that is an avoidable and a seriously bad mistake.
The degree and type of testing depend heavily on the required quality of the effort and the time available for building it. Some code needs incredibly intense testing; some just needs a light amount of smoke testing. That is driven more by the problem domain, not the technical one. That is, writing an app for entertainment may not need much testing, but writing one to be included in a medical device, airplane, or any other strict domain needs a lot and many different types of testing.
So, as well as tracking and sign-offs, heavyweight processes often involve adding strong one-to-one relationships between the specified functionality and its testing in order to verify that it was implemented as described. That is, it ensures that the development process did not accidentally or intentionally deviate from what was specified and is known to be needed for the problem. It also ensures that there is no poorly tested functionality stuffed into the codebase.
In many cases, if the software can glitch with low impact, and the timeline is super tight, the testing can be minimal. But if that results in significant runtime problems, then the testing requirements need to increase. As long as there is any sort of upset from bugs escaping, the testing needs to be enhanced until finally the right level is found. That obviously means that putting effort into automating the testing itself is important, since it is more reliable. But it also means that the type of testing chosen is critical too. If partially testing a few of the components isn’t sufficient, and too many problems are occurring, then it needs more system, regression, or even reference testing.
Testing is not an afterthought; it is tied directly to the problem. That is, some problems need better quality, some are more lenient. It is influenced by the development effort as well. Some teams are better at double-checking their work; some are more reckless. The lead developer should know what the development needs and what the team is capable of. The domain expert should set a minimal level of quality that doesn’t contradict their time expectations.
As a big part of this methodology is in providing solutions that satisfy people’s ability to deal with their problems, part of the trust issues between developers and them is the need to act professionally. That is not just in writing code of reasonable quality or putting in the time to understand the domain perspectives, but also in the words and terminology used to describe the process as well. Because of that, for the parts of the process that developers may or may not need, it is important not to use gamified terminology for them. Some of the earlier lightweight methodologies tried to make the names sound fun or even cute, but that only degrades the way others see the profession. The names and terminology used need to look professional from the outside perspective because that is how developers get people to trust them, which allows them to keep the processes between them and stakeholders lightweight, which we know is at least a little less stressful for developers.
In summary, if a development project is adaptive, you should be able to easily identify the domain expert and the lead developer. Both of them should know what they are responsible for, and be good at avoiding stepping over those lines. As the issues have arisen, more heavyweight processes have been added, but the processes are as lightweight as possible. Both the problem descriptions and the code are organized; you can look through the details and see that they have been cataloged and handled appropriately. The code is always tested enough, and there are few operational issues. Whatever higher-weighted processes like some tracking are there, they are well organized and are not choking out the momentum. The train is on the tracks and moving forward at the best pace it can given the life phase of the project.
So for developers, the idea is that you have to know what the long-term solution is that you are building, while you have to keep adapting to the short-term disruptions that you encounter along the way. It’s trying to be proactive, while not letting the reactive issues knock everything too far off course. It's knowing that there is at least one person who really understands the problem you are trying to solve and another who really understands utilizing technology to really solve those problems, and that they rarely are the same person. It’s not over-engineered solutions that veered way off course, nor overly reactive little partial solutions that are piled together into a mess. That the long-term anchor keeps the short-term responses in line to extend out the lifespan of the effort, and that that is the most cost-effective way to proceed.
