Developer Management

It’s very difficult to bring a group of software developers together and get a reasonably well-built software system out of the process.

The problems stem from the underlying programming culture. Coders tend to be introverted, their thinking is often extremely black-and-white and they go rogue quite frequently. 

This leads to a number of unfortunate outcomes. 

First, they are not really good at communicating the problems and issues that they are having getting the system built and running. They don’t like to admit problems. Coupled with the modern impatience to get things done too quickly, this often reduces the timeframes to considerably less than is what is necessary to do a good job. If the stakeholders aren’t aware of many of the ongoing problems, they aren’t going to have realistic expectations on the progress of the work.

Coding requires a rather rigorous logical mode of thinking. Stuff either works or it doesn’t. Things are either right or they are wrong. The computer does exactly what the programmer tells it to do, there is no room for ambiguities. If you spend your day writing code with these properties, it tends to leak out into the rest of your thinking, life, and interactions with people. The world, however, is grey and murky, and often requires a soft touch to work through the issues. A big team of people working together generates a lot of different agendas and politics. None of this is nicely black and white, so there is a lot of friction between how the programmers think the world ‘should’ operate and how it actually does operate. This leads to a lot of misunderstandings, anxiety, and confused goals. 

With a different perspective on the priorities, and a desire to not want to talk about it, programmers are infamous for just making a decision on their own and then going off with full confidence to get those things done. The problem is that those attempts are often not in sync with the rest of the project, so they basically go rogue and end up doing something that doesn’t provide value. A sub-group of coders incorrectly heading towards the wrong objectives will conflict with the important ones, so the result is poor or useless work.

It’s hard for management to distinguish between a rogue programmer and one not doing any work at all. In both cases, what gets accomplished is nothing. Usually given that their expectations are off too, this builds up a lot of tension between the developers and management. 

In the past, people like to blame the “waterfall methodology” for going off in the wrong direction and returning with stuff that was not useful. They insisted that it was the methodology that was at fault, that it was a ‘time’ problem, but there is a little more to it than that. 

If it's a big, well-defined project that takes 1.5 years to accomplish, doing it in one long continuous unit of work is a whole lot more efficient than breaking it up into little iterations and trying to stitch them together somehow. Mostly batching together similar work is more effective, is better for consistency, and for focus. 

The big failures before the turn of the century drove the stakeholders to seek out better ways of tightly controlling their software projects. The culture of programming itself helped. Both sides settled on an invasive form of micromanagement. The coders seeded control of their work. So, the prerequisites for deciding on the right work like analysis and design get ignored, while the whole effort is gamified with a rather childish bent on formality. You don’t have “excessive” daily status meetings, you have ‘standups’ instead. There isn’t a long list of pending work, it’s a ‘burndown’ chart. We don’t break down the work into tiny, little, verifiable chunks, it's called a ‘sprint’. Planning is a game, not a necessity, and the stick is called a ‘retro’ which is somehow supposed to good for you. 

Each time management was compelled to reach for the thumbscrews and lockdown inappropriate behavior, consultants came up with cutesy little names and games for implementing it, and pushing it as propaganda to the rest of the industry. It’s unfortunate. 

For me though, the fundamental problem is not upper management controlling how the system is constructed. Rather, it is the role of leading a group of developers that is confused. It’s not a technical role and it's not a business role. 

Ultimately, there are some high-level goals that need to be accomplished. The people setting those goals do not have the ability to break them down and express them as a very, very long list of complicated programming tasks. That’s the communications impedance mismatch. If you hire a bunch of programmers and can’t tell them what to do, and they won’t tell you what problems they are having, then it is pretty obvious that the project is not going to function.

So, you need an intermediary. Someone who has spent a lot of time programming, but also someone who has been around the higher-level objectives enough to understand them. They have to have a foot in both worlds because they have to accurately translate between them. 

They might not be the best programmer, or be able to solve silly little fake coding issues. They just need to have spent time in projects of similar scale. They need to get their priorities straight. They might not fully understand all of the business objectives, or be a domain expert, but they need to have empathy for the users and to have some depth in their specific domain problems. They sit in the middle, and they ensure that the upper goals are progressing, while the lower work isn’t going rogue. 

Over the decades, I’ve heard many an entrepreneur reach the conclusion that all they need is a group of students that can code a little bit in order to get their product to market. That’s kind of the classic delusion. It sees coding as a commodity that just requires enough ‘energy’ to drive it forward. Oddly, that can work for a demo or a proof-of-concept or some other introductory software that just needs to kinda work in order to grab more interest, but it fails miserably once it becomes real, mostly because the necessary skills to keep it all organized and keep it growing are missing. So, it’s a start that can be used to evaluate ideas, but not a product that will work when needed. 

Moving up to that next level means getting serious about keeping the work under control. It’s not a small gap, but actually a rather huge one. It’s not intuitive and the kids that threw together the prototype code won’t be able to magically pull it from the ethos. This is where coding switches from being a game to getting serious. It can’t afford to be ineffective anymore, it can’t afford to be disorganized anymore. Everything changes.

For medium, large and massive projects, even the smallest issues have huge, wide-ranging consequences. You learn to deal with them via experience, and it is these issues that are far more important at this point, than the actual coding itself. Fixing the code is cheap, unrolling a ball of mud is expensive. An intermediary who knows it is important to restrict ongoing dependencies, for example, is a much better asset than a coder who can craft unique algorithms. The wrong algorithm is useless, while the wrong dependencies are often fatal. 

In an industry known for its agism, and for still having a high rate of failure, you’d think it would be obvious by now that we’d know there is a missing critical component in the effort. But oddly, the stakeholders still think programmers are just cogs, and the coders still think that if they just had “more code” their problems would magically disappear. The technologies have changed, the methodologies have gotten crazier, but the underlying problems are still the same. Breaking up a months’ worth of work up into hundreds of artificial 2-week tasks doesn’t ensure that it will go any better or be more appropriate. Instead, it tends to build up a counter-culture of gaming that process. Since it’s all indecipherable from above, it does nothing to ensure that progress is actually moving as best as possible. It just provides a false sense of momentum. And the games that coders play may distract them for a while, but the necessary satisfaction from doing a good job is missing, so they aren’t getting what they want either. 

Part of programming is really boring, routine, software production. It’s just work that needs to be done. Some small parts of getting a big product out to market are creative, but more often than not the creative portions fall into the business, design, and architectural efforts. If the ideas and issues are worked through in advance, and any difficult technological issues are prototyped up front, then the rest of getting out a new release is just the careful assembling of all of the pieces in an organized manner. It’s not a game, it’s not a contest. Having someone who knows what is really important during this phase of the work is going to prevent a lot of predictable quality issues from materializing. Like any other profession, programming isn’t “fun”, but when it is done well it can be quite satisfying. 

Return to the Dreaded M...

You’re being spontaneous. You buy a huge collection of tools. Everything from hammers and saws all the way up to clamps, shovels and drills. You order in tonnes of concrete, wood and drywall. Perhaps you even throw in a bit of re-bar -- those steel rods used to reinforce concrete -- just for extra strength. With all of your energy and a few of your closest friends you hit the dirt and start digging like mad. Dig, dig dig, and then you pour the concrete. Creatively, of course; in various spots around the excavation. A few truck loads at least if not more. A bit here, a bit there. Your not sure yet were it really belongs, but you'll figure that out as you go. You pound in some boards: big and small. Then it is time to whack up the drywall. So you add a wall here, maybe a window or a doorway over there. A bit more concrete; then some more wood; you keep going. It looks like some stairs might help at this point. Towards the end you toss up some "roofing" areas to keep the rain out. Add in a few shingles and "presto", you have finished house. No fuss, no muss and definitely no plan. Sound's a little crazy doesn't it?

So honestly now: how messed up do you think this house is going to be? I've been fixing parts of my house for decades, but I can't imagine building a whole house from scratch. I'd definitely never try it without having a well thought out plan in advance. It would be insane to spend tens of thousands of dollars on supplies, equipment, and all of that time, if you suspected from day one that the house was probably going to fall down. But you'd expect that, wouldn't you, if you started working without a plan. Even if you were a professional house builder with years under your belt, unless you were following a design, detail-by-detail from memory, things will go wrong. Horribly wrong. A house is a complicated thing, you just can't build one without some sort of coherent design. All of the pieces have to come together for it to work. Built randomly, the building will have an inordinately large amount of problems. Any structure built under the same circumstance would, in fact anything at all built under the same circumstance would have lots of problems, unless of course it was trivial.

If you wouldn't do it with thousands of dollars worth of housing supplies, you certainly wouldn't do it with millions of dollars worth of software development, would you? Or would you?

Virtual or not, building software without a design will result in an edifice that won't stand up to the rigors of time. Other than a simple project, there must be a plan and it must contain a design. Arguing otherwise is not rational, and certainly not a position you want to dig in and defend for any length of time. A design is necessary to avoid wasting time and resources, and in the case of a house, in keeping the structure from collapsing on your head. Designing is a good thing. No one should seriously suggest otherwise.

You need a design, but how much of it do you really need? What about going out there and using a Polaroid camera to capture all of the good bits that you see in other houses. Polaroids are useful because you can almost instantly get a physical hard-copy of the image right after taking the picture. Snap, snap, snap, and you capture this corner, that wall, this part of the window frame, and so on. If you visit enough houses, and take enough pictures of all of the cool details, you should be able to put it all together to form an uber-design for the perfect house, right?

Is it that surprising that your stack of Polaroids holds no more value then just winging it?

An incoherent design, one that doesn't present a complete model of the thing to be constructed can't be validated. If it is not small enough, or not abstract enough to be manipulated inside of someone's head, it can't be validated. As such, there will be problems. The only question is how fatal will they be? The problem with a stack of Polaroids as a design is that the details may all be there but the big picture is missing and the organization is awful. The stack might be enough input to produce a design, but it is no substitute for a well-drafted blueprint of a house. Blueprints are a representation that can be worked from. Small, abstract, but highly detailed in a manner that is quickly readable and easily validated. Content is important, but how it is presented is equally as vital.

So why, then do software developers gather buckets of requirements; the little details, and oodles of screen shots; more little details, and then expect to build a coherent system from that mess. Tossing in more details in the form of a bunch of disconnected use cases isn't going to fix the problem.

Capturing the bits may be crucial, but if they aren't integrated into a design they are useless. Or even worse: misleading.

The bulk of most designs is pretty much 'routine engineering' anyways. It is repetitive, and has been done before. Lots of times, actually. It doesn't serve anyone to iterate out all of the same basic details over and over again. The real design quickly gets lost in the mush. You certainly don't see that with blueprints for a house; the actual design is projected onto a 2D surface where the true spacial locations for things are not explicitly specified, yet there is enough there to correctly build it. That is a key point. A critical aspect.

A design needs to encompass most of the details -- it shouldn't get lost in that 80% of redundant stuff -- but the other 20% must be clear. To be useful, the developers need quick access to the right information at the right time. Should you break out each requirement into mind-numbingly trivial detail, it only slows down the process. By being pedantic we often obscure the details with noise. A massive list of all of the possible requirements for a system fails to work as a design. It it too big to validate, too redundant to pay attention to, and too noisy to fully understand.

As well, I think that fully listing out all fifty screens in the system is not a good design either. It makes more sense to design a couple of 'types' of screens such as a details view and a list view, then match them back to the fifty. In that way, there is a concept (screen type), a high-level view (different types of screens) and then the remaining details; a list of fifty screens and their resulting mappings.

Going at the design in this manner serves several useful purposes. First, it is smaller and more likely that a single human can validate it correctness. A feat that should not be underestimated in usefulness.

Second, it helps to provide the much needed consistency to the overall design. The screens flow, because they were designed to flow, from detail to list and back again. Users love that type of consistency, and getting it into the design early means a better likelihood of keeping it there for the implementation.

The most important reason for handling design like this, is that any and all of the problems at the design stage directly influence the implementation; no design for instance causes chaos. While having too many little bits, such as requirements and screen shots blurs the overall picture with too much noise. We are told to avoid compound requirements, as if they are bad somehow. But if the designers define a huge number of independent requirements, that "mass" of detail is exactly what will be coded, and that coding style will be "brute force". The results will be a big ugly mess of code. So, not only will all of the requirements make the design larger and more cumbersome than necessary, but the implementation itself will follow suit; a problem that threatens the design of the system and the ability for the development to finish on time.

Compacting the design compacts the implementation.

Getting back to house building, it is done with considerable care from a working design. Generally one that is thought out in advance. Designing a house is straightforward and well understood, particularly if you gloss over some of the weirder local conventions. Software development needs an equivalent mechanism for design.

We live in our houses, and renovate them over time. They get extended and modified to suit the various different occupants. The designers could not have or would never have conceived of what would happen down the road, it is not possible. My own house is over 100 years old and has been renovated so many times I could not even guess. The original builders would barely recognize it today, and it looks quite different from the other similar houses in our neighborhood that all started life as the same design.

Knowing that about houses, brings up two corollaries for software: one is that the design is going to change over time, as things change around it. People live in their software systems the same way we live in our houses. It changes over time but that should never be an excuse not to design. You might not know exactly where it is headed, but you do know that it is a house you are building.

That leads to the second point: it is not possible to renovate my house, or any other small structure into an apartment building. Won't happen. If someday that's what ends up covering our land, it is because it came from tearing down our house (and several others) and rebuilding it. A house is a house, and you cannot enhance it to what it is not. You need to design and build an apartment building from the ground up to bear the types of stresses and strains that are unique to it. Often that point is missed in software. It is just that we can tear it down in bits and convince ourselves that we haven't made major changes. Or sometimes, we can choose not to tear it down, even though it would have resulted in better code and less work. Virtual monstrosities may not collapse on your head, but they do come tumbling down.

Still there? Good, the M from the title comes rolling back into this blog entry in the form of stating that a methodology defines the steps necessary to design and build software. The aspects of the design -- what goes into its makeup -- are key to the methodology. The way the needs of the users are collected, processed and carefully laid out in a design should be explicit steps in the process. How the design is packed together, should also spring from the process. If you were working with an architect to build or enhance a house, the "thing" that you are working on is the blueprints for the house. That much is defined by the process of building a house.

A good software methodology needs the same. It should tell you how to produce a design in a way that it will be as simple and compact as possible. The blueprints for the system should be precise, but simplified abstractions that contain the overall view embedded with enough detail. What that actually looks like on paper, is a problem for the methodology, not the software developer.

With Cat-like Dignity and Grace...

And I thought I was doing so well. I had found my voice, knew what I wanted to say and was starting to express it an enlightening and hopefully entertaining way. Then why, you ask would I deliberately go out and make my readers cough up a hairball?

"Why not", I say. "Sometimes, you just have to talk about the things that people really don't want to talk about". Sometimes they should listen.

Building software is challenging, but many of our problems spring from the way we are going about it, not from the technology itself. At the very heart of all development is the dreaded m word. Muggle, you say after having read too much Harry Potter? No, the other one: methodology. It is the 'thing' that sets the steps of the process; laying it out for us.

Perhaps it is an infancy thing, but for software the range of methodologies used is unusually vast, ranging from virtually nothing, all of the way up to inches and inches of documented process. The size of the range speaks volumes about where we are in our understandings.

Nothing, is really bad, of course. A complete lack of any recognizable process, when you see it, is a mess. Like a steel ball in a giant pinball machine, flipping from crisis to crisis, the organization never follows the same path twice. The inconsistencies fuel the game. There may be occasional wins but there will be lots of crashes too. Failure is an option. And a common one too, as the ball frequently is missed by the flippers and goes sinking back into the drain. Chaos quickly takes its toll on the organization: low morale, high turnover, and volatility.

Contrast that with one of those long, slow, overwhelming processes. You know the type that act like a giant poisonous gas cloud, sucking the life out of everything they touch. Its presence leaving the coders fearful and huddling in their endless maze of cubicles. A full three letter acronym of dread and despair that is sure to choke off even the most adventurous. The code gets done. Eventually. But it is so damn ugly by the end that most people turn a blind eye towards it, trying instead to develop hobbies outside of work. One last redeeming attempt to get some source of satisfaction in their lives.

Both ends of the spectrum are horrible places to be stuck, but they don't take away from the underlying need to have a good process. Even more important: the process you use absolutely shapes your output. Yep. You may throw all the cute technologies you want into the pile, you may even gather the best and the brightest, but the process always taints the results.

It is that important. It is that influential. It is that elementary.

I've used many processes over the years, none of which I would recommend. They fail more often because of their negative side-effects, not their central focus. It is sad because the process should be there to help me. I need something I can depend upon.

Over my next few blog entries I want to continue this discussion, focusing in on the areas where I think changing the process is crucial. Were it could work. We need new ideas for design, implementation and testing. Not complicated ones, but they need to fix the problems. Our current ones don't live up to our expectations. We either avoid the hairballs, or we rehash the same ideas, dressed differently, but still clinging to the same old flaws.

My ultimate goal is to write another book. A simple reference to a simple methodology. One that fixes the problems, not contributes to them. There is enough knowledge available, it just needs to be consolidated, sifted and repackaged in a way that it is usable. For this task, feedback is important to me.

We probably already know the next great methodology that will transform our industry for the 21st century, we just don't know that we know it, yet.