Estimation

One of the most persistent myths in software development is that it is impossible to estimate the amount of programming work required for a project. No doubt this is driven by people who have never been put in a position where estimates mattered. It is indeed a very difficult skill to master but a worthwhile one to have, since it routinely helps to make better decisions about what to write and when to write it.

In my younger days I was firmly in the camp that estimates were impossible. That changed when I got a job working with a very experienced, veteran team of coders. I whined that it was "hopeless", but they said to "just try it", so eventually I did. We settled on lines-of-code as the metric, and over the years we tracked how much we wrote and how large the system was. Surprisingly, after years of tracking our progress, we found that we could make quite strong estimates about our future work output.

After I left that job, I found myself in a startup that was particularly dependent on selling custom code to the clients. It accounted for a significant part of our revenue, and it all relied on being able to produce quick, accurate estimates during client meetings. Estimate too high and they wouldn't buy in. Estimate too low and we'd lose money on the work. Since it was just after the dotcom bomb, things were really tricky so we needed every penny we could get to avoid bankruptcy. My earlier experience really paid off. I would sit in meetings, noting down all of the pieces in discussion, then be able to quickly turn these features into dollars so that we could hammer out an appropriate set of functionality that matched back to the client's budget. That on-the-fly type of price negotiation could have been really dangerous except that I was quite accurate with calculating the numbers. When you have no alternatives, you get forced to sharpen your skills.

For the rest of this post I'll do my best to describe how I estimate work. There isn't a formal process and of course there are some tricky tradeoffs to be made, but with plenty of practice anyone can keep track of their past experiences and use that to project future events. It's a skill, a very useful one and one that really helps with ensuring that any software project does not get derailed by foreseeable issues.

The first thing that is necessary is to choose a reasonable metric for work. Although it is imperfect, I really like lines-of-code (LOC) since it's trivial to compute and when used carefully it does reasonably match the output of a development project. It's also easy for programmers to visualize.

Most of the systems I've worked on have been at least tens of thousands of lines, often hundreds of thousands, so to simplify this discussion I'll use the notation 10k to mean 10,000 LOC, it will just make it easier to read.

For me, I count most of the lines including spaces and comments, since they do require effort. I generally avoid counting configuration files, and for web apps I usually separate out the GUI. Coding in languages like HTML and CSS is often slower than the more generalized languages like Java or C#. This causes a multiplier effect between different programming languages. That is, one line in Java is worth two lines in straight C, since the underlying primitives are larger. Tracking the differences in multipliers due to language or even the type of code being written is important to being able to assess the size of the work properly.

I generally like to watch two numbers, the first is the current running total of lines for the system. So we might talk about a 150k system or perhaps a rather smaller 40k one. If along the way someone replaces 5k with a better 10k piece, the total has only increased 5k. That is, modified lines count for nothing, deleted lines are subtracted and new lines are added. It's only the final total in the repo that really matters. It's easy to calculate this using a command line tool like word count, i.e. "wc -l *.java". One of the first scripts I write for any new job is to traverse all of the source and get various counts for the existing code. The counts are always tied to the repo. If it isn't checked in, it hasn't been completed.

The second number I like to watch is any single developer's production. That is the total number of lines added to the repo per year (although sometimes I talk about weekly numbers). So I think of programmers as being able to contribute say 20k or 50k. Sometimes I think in terms of new lines, sometimes in terms of net lines, depending on the level of reuse in the code. Again, language and code type are also significant, you can't compare a 15k JavaScript programmer to a 25k C# coder, but knowing the capacity of the coders on a given team really helps in determining if any particular project is actually viable.

One big issue to watch out for are the copy-and-pasters. They might have awesome numbers like 75k, but the messiness of their output and the dangers of their redundant code really reduces their work to 1/3 or worse of their actual counts. That, and they endanger any ability to safely extend their code. If you were to try and capture the counts, the copied code would count for 0 lines, modifications and additions would add, but deletes would be ignored.

Now generally a 50k programmer might have lots of 1k weeks, but very few programmers are actually consistent in their output and modern software development is highly disruptive. It's a lot of 'hurry up and wait". Back in the waterfall days, when the development stage lasted months or years it was easier to be consistent for long periods, and it also meant higher yearly counts. But even then some weeks will have very low counts, while others might be over 2k. Because of that variability it is more appropriate to look at the yearly numbers. As the project matures, the counts should get lower as well. That happens, particularly if the software is well-written, because the size of the underlying primitives grow. Instead of having to start from scratch each time, most new work can leverage the existing code (if the counts aren't going down, then possibly the cut & pasting is increasing).

Code that is algorithmically complex or is particularly abstract is obviously a lot slower to write and debug. In that sense, the backend guy from a project will progress much slower than the front end guy. So you might have a domain expert coder at 15k, with a GUI front guy at 35k. A decent generalized piece might have really low counts, but when it's reused each time it saves the overall system size from bloating, although it doesn’t reflect directly in the numbers.

Thinking in terms of totals and capacity has the nice secondary attribute that it makes it easy to translate into man-years. For example, a small but functional enterprise web-app product needs at least 100k to round out it's functionality (including administration screens). For a 50k coder that's at least 2 man-years. If you see a 350k system, you can get a sense of how long it might take to rewrite it if you had a team of 25k coders, it's sitting at about 14 man-years to replace unless you can get away with a lot less code.

My suggestion to most programmers is to learn to track both numbers for themselves. They don't have to share them with management, but understanding their own abilities to produce code and the size of the projects that they've experienced really helps in knowing their current skill level. That keeps programmers from over-promising on their assignments. If you are a 30k programmer and someone wants you to hammer out a 40k web app in half a year, you can be pretty sure that you're in trouble coming right out of the gate. If you know that in advance -- as opposed to the last month or so -- you can start taking action to address the issue, such as asking for more resources or reducing expectations (or reducing the quality).

It's also worth noting that speed isn't what counts, it's organization and quality that really matter. I would be far happier with a detail-oriented 15k programmer for which we never have to debug any of their work or rewrite it, than a reckless 75k programmer that is essentially just contributing more problems. Projects last forever, so it's important to get the work done well, encapsulate it and then move on to other more interesting code. A well-rounded team usually has a mixture of fast and slow programmers, with hopefully a wide range of different skills.

My first code tracking experiences came when building a massive 150k distributed cache. In the early years I was cranking out about 50k per year as my primary project (I've always had at least one secondary project that was not intended for production, but just to sharpen my skillset). Years later, I was working on a 120k enterprise product in Perl at about 30k per year (with lots of sales and management duties). These days I'm somewhat less than 15k in C# on a 350k behemoth (which if it wasn't for cut & paste should have been closer to 150k), but again my role isn't just pure coding anymore.

Size matters. I've worked on a few >1M systems, but at that level there are usually teams that are responsible for 50k - 200k chunks of the system. From experience I've found that a code base over 50k - 100k starts to get unwieldy for a single individual to handle by themselves. Most single developer in-house systems tend to range from 25k - 50k. Commercial systems have to be larger (to justify the sales) and tend to be higher quality so there are more lines; they usually start around 100k and go into the millions (except possibly for apps). Quality requires more code, but reuse requires quite a bit less, and there can be big multipliers such as 2x or even 10x for brute force and cut & paste. It's those latter multipliers that always makes me push reuse so hard. Time might be saved from pasting in code, but it gets lost again in testing, bugs and obfuscating any attempt to extend the work. In a big project, bad multipliers can result in the work falling "years" behind schedule particularly if the work is compared to a competitor with an elegant code base. If you have to do three times the work, you'll have 1/3 of the features.

Once you've mastered the skill of tracking, the next thing to learn is estimating. I always use ranges, so a new bit of work might be 10k - 20k in size. The best and worst cases are drawn directly from experience with having written similar code. The ranges get wider if there is a lot of uncertainty. They are always relative to the system in question, so that a new screen for one rather brutish system might require 20k in code, but for an elegant one with lots of reuse that might be 5k or less (getting below 1k is awesome).

Of course estimating in an ongoing project is easier than a green field one, but in some sense if a programmer isn't experienced enough to make a reasonable estimate he or she might take that as a sign that they don't currently have enough experience to get the work completed properly. Most system extensions are essentially horizontal, that is they are just adding in features that have the same design patterns as existing features. In that case, if the new work leverages existing work, then a huge amount of time can be saved per feature and the code will automatically maintain good consistency (and have less testing).

I pretty much expect that something similar to the Pareto rule works for overall complexity. That is between 10 and 20 percent of the system is difficult, slow code while the rest is fairly routine. The 80% still requires work, but it should be quite estimable with practice. The difficult stuff is nearly impossible to estimate since unexpected problems creep up, which is why each key aspect of it should be prototyped first during the analysis and/or design phase (for startups this is essentially version 1.0). If you separate out the code into independent problems then it makes it easier to plan accordingly. Often under-estimates in some parts balance against over-estimates in others.

Another common issue is uncontrolled scope creep, where the underlying requirements are constantly shifting. Sometimes that is just a lack of reasonable analysis. Since analysis is way cheaper than code the development should be put on hold until the details are explored more thoroughly. It's always faster than just flailing at the code. Sometimes scope creep is caused by natural shifts in the underlying domain, in which case the design needs to be more dynamic to properly solve the real problem.

It's worth keeping in mind that depending on the testing requirements and the release procedures, a coding estimate is only a fraction of the work. Testing, in reasonable systems tends to stay in lock step as a fraction of the coding work. Releases are more often a fixed amount of effort.

Once you can size incoming work, the next trick is to be able to shift the team around to get it done effectively. Programmers are like chess pieces, they all differ in strengths. It doesn't work to give a backend guy some GUI piece if you have a faster front end guy available for the job. This is spoiled of course since some programmers don't want to focus on their strengths, they want assignments that stretch their abilities. With good estimations, it actually makes it easier to let them try out different parts, since tracking lets you know how much work is remaining and whether there is actually some slack time available. High efficiency teams are generally ego-less such that the programmers aren't siloed, they can work on any section of the code at any time. Of course having at least two coders familiar with every part of the system reduces disruptions with emergencies, holidays and people leaving. It also helps with communication and overall quality. But it has to be managed in a way that doesn't just blindly assume that all programmers are interchangeable cogs. Each one is unique, with their own skills and suitability, thus it takes plenty of observations and some deep management skill to deploy them well.

For me, with deadlines, I'll either accept a variable number of features with a hard date, or the full set of features with a variable date. That is, we can deliver 4 - 10 features on date X, or we can deliver all 10 features at some floating time in the future. Sometimes the stakeholders argue, but generally there are one or two really key things they are after anyways. With reasonable estimations, fixed dates are manageable. You can choose to do a couple of priority items early, then go after some low hanging fruit (easy items), then just aggressively cut what isn't going to make it. A key trick to this approach is to get the developers working on just one or possibly two items at a time, and don't let them go on until the items are done and dusted. Estimations really help here because you can quickly see if any of the items exceeds the length of the next iteration, and thus should be branched off from the main development (and then not tied up in cross dependencies). Obviously the more accurate the estimates, the more options you have in planning the development and managing the expectations of the stakeholders. Accurate estimates reduce a lot of the anxieties.

Once a project gets established, the incoming workload tends to fluctuate. That is, it's quiet for a while, then a whole load of new work comes in all at once followed by a rush to get it done. Software development is best when it is a smooth and steady process, so being able to scale the work loads and re-arrange the teams really helps in smoothing out the development to a nice consistent pace. This avoids destructive practices like 'crunch mode' and allows for intermediate periods to reduce technical debt. It also makes it easier to decide when to just hack up some throw-away code and when to take the time to generalize the code so that you can leverage it later. Foresight is an incredibly useful skill for big development projects.

What gets a lot of projects into trouble is that become reactive. They focus on just trying to keep up with the incoming work, which ultimately results in very high technical debt and further complicates the future workload. Learning to estimate is tricky, but valuable as it helps to make better decisions about what to write and when to write it. This helps in getting ahead of the ball in the development, which leads to getting enough 'space' to be able to build clean, efficient and high quality systems.

Once a programmer has mastered the basics of coding, they start getting the itch of wanting to build larger and larger systems. For medium sized systems and above the technical issues quickly give way to the architectural, process and management issues. Within that level of software development, life is much easier if you have the skills to size both the work and the final system. It allows significant planning and it provides early indicators of potential trouble.

Estimating is a tricky skill and one that can not be trivially formalized, but it's not impossible and once you have enough mastery you really wonder how you ever lived without it. Nothing is perfect, estimates less so, but with practice and experience software developers can produce very usable numbers for most of their development efforts, and the technical leads can understand and utilize the underlying capacity of their team members. Estimations are not feasible from non-programmers and they should not be used to rank programmers against each other. There are very real limits to how and why they can be practical, but when used correctly they can at least eliminate many of the foreseeable problems that plague modern development efforts. A little less chaos is a good thing.

Tuesday

I'm feeling out of sync. When I started programming several decades ago, we basically followed an engineering ethic of always trying to build the "right" thing. These days though, it seems that the software development culture has drifted away from that mindset, leaving me stranded in the past. 

By way of an analogy let us consider a simple program. For its requirements, lets say that the users specified that they only need this program on Tuesdays. Now just for argument sake, lets say that it is considerably easier to write this program with the day of the week hardcoded to Tuesday, so that only on Tuesdays will the program work, but for all other days of the week it will have errors.

Personally, I would never say that the Tuesday program 'works'. Sort of works. Partially works, sure but I'd never describe it as a working program. However, lately in several different variations I have had people absolutely declare it working properly. After all, the requirements said 'Tuesday' and the program functions according to the requirements. 

"It works on Tuesday, that's what is supposed to do. It doesn't need to do anything else," they say. "All we need to do now is to document its behavior."

For me, it seems negligent for programmers to know that there is more than one day in a week, but choose to ignore it. What if someone wants to use the program on a Thursday? Their answer is "All we have to do is make a new copy of the Tuesday program, then change the hardcoded Tuesday value to Thursday," as if that was somehow sane or obvious. 

In a way I do get this new mindset. It's a form of minimalism, and damn the consequences. Is it really so bad to head to a place where there is a special hard coded version of the Tuesday program for every day of the week? If people really need the program for all seven days, then there will only be 'seven' almost identical copies of the program. "It's not so bad," they claim.

The problem, as I see it, is that this type of strict interpretation of the requirements overlooks the first unwritten requirement that most users have: they don't want crap. It's sometimes not the case, and it is never written down, but for most projects, most of the time, especially if the users are in a rush, they're not going to be pleased on the first Thursday that they need to use the program to sit around waiting for the Thursday version to be created. And their not going to be impressed if this happens three to five more times. They'll forgive the programmers after that, but only until the next code modification comes along and now they have to wait either seven times longer or live with different inconsistencies on different days. All of the early gains, by quickly releasing the Tuesday program, will be swept away by the slowness or aggravation they get later. They'll realize that their very first, unwritten, unspoken and believed to be obvious requirement of "don't do this badly" will have been broken. If they eventually get angry, can you blame them? If they don't trust the developers later, would you?

In a much more abstract sense, particularly if you think my analogy is a straw man, what I've been seeing is that programmers have narrowed down the problem context well past a reasonable minimum so that they can deliver quickly. It's not complex at all to deal with all 7 days of the week if you can deal with dates, but if you squeeze that context down to only Tuesday you can avoid some of the drudgery of doing it properly. Saves a bit of time... today. For lack of a better name I'll call this 'context hacking'. The real experts in this art can even take a useless piece of code and claim it "works" too, just by creatively hacking off enough context. It is similar to the all too common "it works on my machine" excuse. There are lots of different variations out there.

Now this should not ever be confused with over-engineering, in that the difference is one of direction. Over-engineering inflates the context well beyond reasonable maximums, while context hacking seeks to go below the minimum. Sometimes that can be a thin line, if during the entire lifetime of the Tuesday program nobody ever uses it on any other day but Tuesday, one might argue that just working on Tuesdays was the real minimum. But the old ethic was that if we have to deal with one day of the week, and we know there are seven in total, then it would be wrong not to make it work for all of them regardless of whether or not the users would actually use it on another day. That is, any day of the week support, of any kind, means that we have to implement the code to do the “right thing” for all of the days. Or basically that the Tuesday requirement is moot if we need any day of the week handling, which takes precedence. And of course there is the added bonus that for the next project -- because there will always be another one -- if we know how to properly handle week days then we can encapsulate this and use it again, which at bare minimum means less testing.

In fact, not writing the Tuesday-specific code, but rather encapsulating the data handling in a reusable manner in a library gradually constructs a toolkit of reusable stuff that can greatly increase the ability to meet future challenges at higher levels, which when compared to repetitively whacking out copies of the same program over and over again is a much more satisfying programming experience. 

Writing the Tuesday program was considered when I started to be very unprofessional. It is all too common these days, but I really am hoping that we rediscover our sense of ethics again. Fast, high-stress programming at first appears to make it all more glamorous and to deliver quickly, but the mess it leaves behind does not make the work enjoyable. It just becomes a quagmire. If we're finding ourselves surrounded by way too much bad code it's because that is what we are creating.

Mathematics and Software Development

Programming a computer is the act of building up a large number of instructions for a machine to follow based on ‘primitive’ operations and underlying libraries. These instructions or ‘algorithms’ are always computed rigorously, which is occasionally not what we intended. Thus ‘bugs’ may interfere with the user’s objectives, but they will not harm the machine itself (although they can occasionally damage peripherals). The machine is simply following the steps that it was told in a deterministic manner.

Underneath, the computer manipulates ‘data’, which itself has to fit to a predefined format. Data stored in a computer is a symbolic placeholder for things that exist in the real world. It doesn’t really exist, but it can be used to track and analyse the world around us.

As such software can be viewed as a ‘system’, whose set of instructions is ‘formal’. We can’t just create any arbitrary collection of bits and expect it to run, the computer will reject the code or data if it isn’t structured properly.

Mathematics, in its simplest sense, is the study of ‘abstract’ formal systems. Mathematicians create sets of ‘primitives’ that act on abstract mathematical ‘objects’. Collectively, the primitives are used to express relationships between the objects that are rigorous, often these can be combined together to form ‘theorems’ and algorithms. That is, for any mathematics to be valid, it must conform to strict formal rules. Mathematical objects exist only in the abstract sense, although they are often used symbolically to relate back to real things in this world. Doing so allows us to explain or predict the way things are working around us. There are many domains that attempt to model the real world by applying mathematics including statistics, physics, economics and most other sciences.

Both mathematics and computer languages are primarily about formal systems. They both allow us to build up the underlying primitives into larger components such as theorems or libraries. They both exist away from the real world, and their utility comes from mapping them back. The underlying most expressive formal system for computers is the Turing Machine. Within this context we often create other more specific systems like programming languages or applications. Turing Machines also exist within mathematics, however they are not the most expressive formal system that is known. There are larger formal systems that encompass Turing Machines. As such we can see that the formal systems within computers are a subset of those within mathematics.

An interesting difference is that mathematics is completely abstract. It can be written down in a serialized fashion, but it is not otherwise tangible. Software however runs on physical machines, that are derived from the mechanization started by the industrial revolution. Internally software might be abstract, but externally the computers on which it runs are subject to real world issues like electricity, temperature and moisture. In this way software manages to bridge the gap between our abstract thoughts and the real world around us. Software also often interfaces directly with users who are creating new input or trying to analyse what is already known.

Given the relationship, software is a form of applied mathematics. Its formal systems share all of the abstract qualities of mathematical formal systems, and the underlying data is essentially various mathematical objects. Building up software on a computer is similar to building up theorems or algorithms within a branch of mathematics. Both mathematics and computers have issues when getting mapped back to reality, since they are at best ‘approximations’ to the world around them.

Software development has been underway for at least five decades, and as such has built up a large base of existing code, knowledge and libraries. The many partial sub-systems of software, like operating systems or domain specific languages can help to hide its underlying mathematical nature, particularly when crafting graphical user interfaces, but ultimately a strong understanding of mathematics helps considerably in understanding and structuring new parts for systems. There are some aspects of programming that are non-mathematical, such as styling an interface or the content of an image, but for the rest of software development having a good sense of mathematics and logic aid in being able to craft elegant, correct and consistent instructions.

Since software is an applied branch of mathematics, it is clear that not all code is intrinsically mathematics, but realistically what isn’t lies only within the intersection between the machine and the user. That is, whatever irrational or illogical adaptations are needed to make the system more usable for people, is non-mathematical. The rest of the system however, if well-written (and possibly elegant) is a formal system like the many that are known in other branches of mathematics.