Scaling Development

Given a software project that would require three people a full year to build, is it possible to get it completed in six months?

The first key point is whether or not it cleanly partitionable? Can it be broken up into two, separate, independent projects? For some types of software development, this is fairly easy. A large website, for instance, can often be split up conveniently by screens. A team works on one set of screens, while another team works on a different set. A complication comes if both sets of screens rely on the same underlying database schema, so that moves the schema definition to the beginning of the project and blocks both teams until completed. It also means that any changes to the schema now have double the impact. Both teams are affected, both need to recode their parts of the system.

If the partition crosses a natural boundary, like one team is working on the user side of the system while the other is working on the administrative side, there are few collisions, so minimal coordination needs to happen. Still, there is at least some effort to bring the two teams into alignment at the management level, so there is some extra effort there.

In most systems, there is are significant opportunities for reuse of identical code. Gui screens actually have a huge amount of redundancy, although most programmers do not try to capitalize on this. Still, it means there will definitely be some redundancies in the code, but also in the design and testing stages. If these redundancies are not fully independent then there will also be bugs created by the two teams being out of sync with other. That problem can be mitigated early but would require more upfront design, particularly at the boundaries that intersect. Many systems can be built effectively with just a top-down, high-level specification, but intersections would require additional mid-level specifications to be properly coordinated. Not doing that work would result in more testing and integration problems, generally with an exponential growth, so a lot of time is saved by coordinating up front.

The differences in the teams would inevitably lead to differences in styles and conventions, so along with the intersections, this would result in significant technical debt. That might not be noticed in the earlier versions, but would definitely impede any efforts to extend the code base, so would ultimately shorten the effective life of the code.

Now that would be the situation for fairly partitionable, routine software development. It is basically doable, but it would be highly likely that the resulting quality would be lower than if the project wasn’t scaled (although there is no intrinsic guarantee that this would be true since it depends on the skill and knowledge of the original team). Like any concurrency issue, doubling the number of people would not directly half the time. The intersection overhead kicks in to prevent it from being linear.
For a project that was highly interdependent and/or deeply on the cutting edge, the situation would be far worse. Since the ability to craft mid and low-level specifications is far harder, the time either grows exponentially or there are exponentially more integration bugs. As well, either the project adopts an onion-like architecture (impedes growth) or the impact of changes becomes more disruptive (often referred to as ‘merge hell’). If maybe a routine development project has a multiple like 2.2x, the management overhead for a complex project would start to hit ranges like 3x-5x, or possibly worse. 

Extending this out, it’s clear to see that if you need 200 lines of code, having 201 programmers means that at least one programmer has nothing to do. If the project is composed of ten 20 line pieces, then 11 programmers is too many. As such, for any development, you might be able to half the time (scale the development) but there is some physical fixed limitation that directly applied that means you cannot recursively apply this halving forever. That is, as always, scaling has fixed limitations. There could never be such a thing as ‘infinite scaling’.

This is important to understand, particularly if we look at a project from the perspective of ‘density’. That is, a routine website development can be considered to not be dense. The code is fairly obvious and straightforward. As the underlying complexity and interdependencies grow, we consider that to be more dense. If we partition a routine project cleanly, we can do so because it has low density, but the two new resulting sub-projects are now denser than the original. If we cannot partition forever, the fundamental reason is that the density increases until it passes a threshold where partitioning is ineffective.

Getting back to already dense code, it by definition is far less partitionable just because of its underlying nature. 

What this says overall is that, yes, there are some projects that you can do twice as quickly. There is a real cost to doing this, but if you can accept lower quality it will work. But there are definitely projects where this is absolutely impossible. That adding more bodies would just prevent or significantly slow down the project. 

Realistically, as an industry, we have known this for a very long time, but there have been some major changes to the way we develop code that has obscured this knowledge. 

One huge change is that more of software development has often become just writing light ‘glue’ code to go between frameworks and libraries. While this degrades the quality of the final system (since the different components don’t fit together nicely) the glue code itself is far more partitionable. In this type of development, scaling is easier and any reductions in quality are intrinsic to the underlying components themselves and not because of the scaling effort. This type of shift has really only enhanced the myth that software development is always linearly scalable (with a minor overhead). Unfortunately, most, if not all, of the software development projects with this attribute are straight up routine development, and we’ve been redundantly applying a massive amount of effort to repeat this work over and over again, with only slight variations in design. Given its nature, it is fairly obvious that we could use software itself to automate significant aspects of this type of work, but as an industry, we have chosen to ignore this and continue forward with the manual approach of just gluing together misfitting components. For software that doesn’t fit into this category, it is obviously more dense and as shown earlier it has real constraints on its scalability, and obviously automation.

So the answer is ‘it depends’, but the corollary is that if it does work, then it is highly likely that there was a more automated approach that could have made it unnecessary anyways. If we wanted to improve our industry, it would make a lot of sense to really clarify density and also build more tools to eliminate the huge amount of manual effort required in continuously rebuilding the same routine systems over and over again.

The Value of Software

A huge whack of code, on its own, is pretty useless. The code needs data to process.

The user’s real-world issues only get resolved by enabling better decisions made from the persistent data. Data is the heart and soul of any software system.

What’s really changed over the decades is not that the code we write got any better, or that the frameworks and libraries became easier to use, but rather that data became hugely abundant. The World Wide Web is a great example. From a coding perspective, it is a dog’s breakfast of good and bad ideas munged together in rapidly evolving chaos that often barely works and has had exponential explosions of obscene complexity. Technically, taken all together, it is pretty crappy. But the Web itself, at least until recently, was a monumental achievement. With a bit of patience and perseverance, you could find a tremendous amount of knowledge. It opened up the world, allowing for people to learn about things that were previously off limits. It was the data, the sheer scale of it, that made the Web wonderful.

That explosion, of course, is diminishing now, as the quality of available data is watered down by noise. The tools we built were predicated on not having enough data, they cannot deal with having too much low-quality stuff.

Programmers still, however, hyperfocus on code. It’s as if an ‘algorithm’ really has the power to save the day. All they think we need is just something better, that can magically separate the good stuff from the bad. Ironically, for our industry, we have known for at least half a century that shoving garbage into code produces garbage as output. And, at least algorithmically, nothing short of intelligence, can reliably distinguish data quality. If we want better data then we have to train an army of people to sift through the garbage to find it. The best we can do is to craft tools that would make this work less painful.

The promise of computers was that they could remove some of the drudgery from our lives, that they could help keep us better organized and better informed. The reality is they now waste our time, while constantly distracting us from the stuff that really matters. The flaky code we rely on is a big issue, but the real problems come from streams and streams of bad data. If we let everyone type in whatever they want, then not only does the conversation degrade, but it also becomes increasingly impossible to pick out value. A massive network of digital noise isn’t going to drive positive changes. Code is a multiplier, the real underlying value of software comes from the quality of the data that it captures. We won’t get better software systems until we learn how to get better data.

Visualizing Code

In its simplest sense, programming is about assembling a list of instructions for a computer to follow.

It has become somewhat more complicated lately because some of these instructions are quite high-level and the underlying details are somewhat convoluted. We like to use badly designed libraries and erratic frameworks to accomplish most of the tasks that we no longer feel obligated to work on ourselves.

That might seem to define programming as just utilizing the knowledge embedded into both the programming language and underlying dependencies. Learn enough of the details and off you go.

But there seems to be another dimension to programming.

In order to fully comprehend the behavior of the code, it helps a great deal if we can visualize the running code. To be able to ‘see’ it in our heads, somehow.

There are plenty of debuggers and techniques for embedding print statements, but these alone, for debugging, are often not sufficient for understanding really complex bugs. And they don’t help at all with design issues. They only show us what is there and step by step, how it runs.

The ability to ‘see’ the running code in our heads, however, allows us to be able to work through the possible corner-cases, instead of having to acquire all of our knowledge from the runtime. Programmers that can visualize their code, can quickly diff their understandings against the actual behavior. It’s this skill that amplifies some programmers to new levels.

The basic underlying act of programming is not particularly creative since generally the problems should be analyzed before the coding began. Most design aspects are fairly routine if the underlying technologies are understood.

But even on an organized and well-thought-out development project, being able to visualize the code before it is being built is a huge advantage. That is, to be able to see the code running in our heads, we need to imagine it in some quasi-tangible way, which needs to be simpler than the actual operation, but not so simple that it doesn’t reflect the reality of the code running. That, because we are human, gives us some internal representations that allow us to understand how we see the code operating versus how it is actually running.

That ability brings creativity back into the mix since it requires imagination, and as it is a slowly growing skill over time it explains why programmers can sometimes accidentally exceed their abilities. That is, someone who is quite comfortable debugging a problem in 5K code might suddenly flounder with a problem that spans 20K. It also explains why a software developer that can design a 50K system might completely fail to design one at 150K or larger. What happens is that they become unable to see these increases in complexity, and if the issues span across the full width of the problem then they are unlikely to make ‘reasonable’ decisions.

What seems true is that visualization size is a very slowly accumulating skill set. It takes plenty of time to keep increasing this ability, little by little. It doesn’t come right away, most people aren’t even aware of its existence and there is no magical way to just hyper-jump it into a broader ability. In that sense, although occasionally an interview question almost taps into it, we generally focus on the exact opposite during interviews. We ask specific ‘definition’ questions, not general visualization questions (a rare exception is Fermi estimations).

There have been plenty of anti 10x programmer movements, where we insist that no such beings exist, but that doesn’t seem to match experience.

If 10x programmers exist, since their ability to type in code is effectively bound then it would be exactly this ability to visualize 10x larger contexts that would separate them from the pack. That makes sense, in that they would be able to see where more of the code should be before they start coding and thus save a huge amount of time by not going down wrong paths. They would also be able to diagnose bugs a whole lot faster and make fewer nasty ones. Both of these account for more of our time than actually typing in the code, which while it comes in bursts is not what usually consumes most programming effort. Someone with stronger visualization skills would obviously be able to work through complicated programs a lot faster than someone with lesser abilities. And it wouldn’t be surprising that this ability rests on an exponential scale, given that increases in code size do as well. Small increases in visualization would give large advantages.

So, it does seem that there is a significant creativity aspect to programming, but it isn’t revolving around finding “creative” solutions when unneeded, rather it is about being able to visualize larger and larger solutions and bind them better to reality.

If we accept this, then it is clear that we need to respond by doing two things. The first is that we need a way of identifying this ability. This helps with interviewing, but it also helps in assigning development work to people. Problems that exceed programmers abilities will not likely get solved well. If we are realistic about the size of our problems, then we can do a better job of matching up the coders to them. We can also assign problems that help push a programmers ability a little bit farther each time. Just a bit beyond their current level, but not far enough that it is demoralizing.

The second thing we should do is find some training methods that help increase this ability. We’ve always focused on simple decomposition methods like data structures or objects, but we’ve never reoriented this approach with respect to how it might help a programmer actually grow their abilities. It’s just considered base useful knowledge. We just drown them with definitions. It also makes a huge amount of sense to learn more alternative means of decomposition and different programming paradigms. These help to strengthen our visualizations by giving us different ways of seeing the same things. Thus we need to revisit the way we are teaching programming with an intent to find new ways to expand this ability.

This issue with visualization may actually help explain both the software crisis and modern bloat. Older programmers always lament that the kids don’t pay attention to the details anymore, and this oddly has been a long-running sentiment for over half a century. That makes sense in that each new generation of tools took some of that cognitive effort away from the coders while exchanging it for the ability to work on larger systems. In this exchange, the underlying visualization requirements are similar but are transferred to different areas. Thus the younger programmers work at a higher level, but what they do spans way more underlying code. So the basic effort stays relatively constant, but for each older generation, it looks like the kids are focussing on the wrong issues.

Indirectly this also aligns with many of Bret Victor’s ideas about reducing the lag between working and results. It is a more explicit external approach to achieving the same goal. It is just that even with his advances in technology, our need to visualize will still supersede them because there will always be a need for some larger visualizations that are beyond the current abilities of hardware. It’s a never-ending problem of growing complexity and context.

The underlying mechanics of programming seem to be fairly straightforward, but we are too focused on the instructions themselves. As we try to build bigger systems, this complexity quickly builds up. In order to cope with it, we develop the skills to visualize the behavior. As we grow this secondary skill, we are able to cope with larger and larger programming issues. This is the ability that gives some programmers an edge over the others. People who can see more of the full solution are far more likely to actually solve a lot more of the problem with fewer resources and better quality.