We, the software development community, are often the source of our own really bad software. Sure, you can blame the managers or you can rail against the underlying technologies, but in the end the most frequent, most destructive problems come from the attitudes of the people involved in the development.
Software, as in any endeavor that involves constructing physical things, requires a great deal of discipline. A mess by definition is repetitively doing things differently. All of those inconsistencies add up, and unless it's some type of creative writing or a painting, the result is big and ugly. Picture a house built in ten different styles, or a car made up of several different models. The 'essence' of a mess is inconsistency.
Software development projects are always big. Big projects require big teams. And big teams need to work together. If they don't work together, the result is a big mess.
I have seen enough projects where the teams are maintaining hundreds of thousand lines of repetitive, redundant and inconsistent code when less than a quarter of that should have been enough. It is easy to see the effects of that type of problem, as the team wastes more and more effort on weak testing strategies and chasing inconsistencies throughout the code. It is a black hole for resources. You can't fix the problems if you are too busy slapping on band-aides.
By its very nature, building a software tool is difficult, but not, as many programmers want to believe, because of mapping functionality onto specific computer language code. That part of development is relatively straight-forward, the difficulties come from analyzing the user problems and choosing the appropriate functionality to implement. Well, they also come from dealing with a messy code-base, but that wound is entirely self-inflicted.
The analysis problems are unavoidable, but ultimately they are not related to programming. Once a team has decided on how to 'solve' the problem, the rest of the effort should just be work. Nothing but work. A group of professionals using their collective knowledge to come together and assemble the appropriate instructions to implement some functionality on well-known data.
In the midst of this, there are always changes, but those small unavoidable shifts in understanding can be easily managed across the lifetime of a sophisticated software product. If you were close to begin with, the changes are never really significant.
The really huge problem with our software projects is our own tragic culture of uniqueness.
It is not, in any way, shape or form, that I want to take away from the human individuality. We are all individuals, and we shouldn't be forced to band together as helpless sheep or powerless ants. But, and here is a huge point: it is that very sense of individualism that is destroying so many projects. The blame, often fails on the guy down the hall, or the manager in charge, or someone else, but the 'problem' at the end of the day is the inconsistent mess created by the programmers that is not working correctly.
It is inevitable that if a team of developers flails away at their keyboard, without working in some harmony, the code will become a mess. Developers will leave, managers will change, these things are normal. If every programmer is silo'ed into their own over-protective code-possessive region of the project, the inevitable changes and shifts are going to bring pain. Quite possibly disaster.
A team that is not working together, that does not share all of their code, that does not follow similar standards, etc. is a defective team. One that is just waiting for trouble. And the effort of a defective team, not unsurprisingly, is always a weak and fragile project.
Yet, even as we all know this intrinsically, and so many of us have had this as a direct experience in our professional lives, we still bend towards the dark side. We don't want to be seen as just 'resources', or to be 'interchangeable', or even 'replaceable', but by not being these things we are sabotaging our own projects.
Presumably, if you get any good professional who follows their industry's best practices properly, they should be able to perform the job for which they were commissioned. There is variation to some degree, but ultimately the very definition of 'professional' means that everybody lives up to some recognizable standard; by its nature that means they can be replaced.
Your life shouldn't rest on the skills of your lawyer, your doctor or your accountant. Scientists, mathematicians and engineers shouldn't be able to set their disciplines back hundreds of years because of incorrect theorems or approaches. You should be able to hire a competent professional to accomplish a specific job for which they have knowledge and experience. And yes, these people, are 'resources', that are 'replaceable'.
There is nothing short of insanity, that would lead one to believe that 'programming' as a practice was anything other than a professional activity. Don't confuse vision, analysis or leadership with programming, those are wishy-washy irrational things, not unlike business acumen. Programming, itself is just the act of taking the desired functionality and encoding it. It is a task that can be accomplished by any well-trained professional programmer.
There should only be one good standard practice of doing this. Some variation, of course, but a team of programmers should come together, pick their conventions, and then stick with them. Thus, beyond all of the work and the team itself -- if it is functioning correctly -- should be a team of 'resources' dedicated to getting the job done correctly. Losing one member, aside from the morale or leadership aspects should not change the outcome of the team. That would be bad. That would be unprofessional.
In truth, although the overall direction might change somewhat, a different analyst, product manager or even visionary leader in a well-healed team shouldn't jeopardize the overall efforts either. All professionals bring with them some minimal level of functioning. That should be enough to keep the project moving forward, even if it suffers some change in direction.
Catastrophic problems occur, but they are individual issues. A bad hire is a bad hire, not proof that a discipline is too chaotic to be standardized. Or too radical to be repeatable.
I do believe that a good product needs a strong unique vision, not a committee, to steer its direction. Consistency starts at the top, but even great direction and leadership at the higher levels fall short if there is chaos in the ranks. Once a development path is forged, the implementation team needs to follow it, not wander around aimlessly exploring options.
Filling in a blank is quite different from re-interpreting the design. A product is only unified if its developers are; inconsistent code sucks; inconsistent interfaces suck.
Honestly, I don't want to work with any more 'irreplaceable' programmers. I've been there, and it is bad. You get some programmer that is so far out on the edge, that virtually nobody else can understand or fix their code. That is, by any definition, crappy code.
If your team can't live without a specific person, then your team is dysfunctional. It is that simple.
This is true at most levels. The best managers know how to make themselves a dispensable part of the process, it is the poor ones that are so controlling that they hide the details from everyone else. It is absolutely true for technical people as well. Those people that horde their knowledge, hiding in their cubicles, smug in their own superiority, are not the ones you really want in a well-functioning team. I'd gladly trade the best of them for a good team player any day.
It is intrinsic to the very definition of a well-functioning team, that its members are 'resources'. It is intrinsic to the very definition of good code that it is readable and can be maintained by any competent programmer. It is intrinsic to a software development project that everybody is working towards the same goal. These are not good ideas, they are requirements; things that need to be urgently fixed if not true.
You can blame management all you want for various failures, but if you're not willing to play nicely with the other developers in one big sand box with all of the toys, then you're probably more at fault than you could ever realize. Programmers never work alone.