Debugging

 So, there is a weird bug going on in your system. You’ve read the code, it all looks fine, yet the results are not what you expected. Something “strange” is happening. What do you do now?

The basic debugging technique is often called divide and conquer, but we can use a slight twist on that concept. It’s not about splitting the code in half, but rather about keeping 2 different positions in the code and then closing the gap between them until you have isolated the issue.

The first step, as always, in debugging is to replicate the bug in a test environment. If you can’t replicate the bug, tracking it down in a live system is similar, but it needs to play out in a slower and more complex fashion which is covered at the end of this post. 

Once you’ve managed to be able to reproduce the bug, the next important step is making sure you can get adequate feedback. 

There are 2 common ways of getting debugging feedback, they are similar. You can start up a debugger running the code, walk through the execution, and type of the current state of data at various points. The other way is you can put in a lot of ‘print’ statements directly in the code that output to the log file. In both cases, you need to know what functions were run, and what the contents of variables were at different points in the execution. Most programmers need to understand how to debug with either approach. Sometimes one works way better than the other, especially if you are chasing down an integration bug that spans a lot of code or time.

Now, we are ready to start, so we need to find 2 things. The first is the ‘last known’ place in the code where it was definitely working. Not “maybe” works, or “probably” works, but definitely working. You need to know this for sure. If you aren’t sure, you need to back up through the code, until you get to a place where you are definitely sure. If you are wrong, you’ll end up wasting a lot of time.

The second thing you need is the ‘first’ place in the code where it was definitely broken. Again, like above, you want to be sure that it is a) really broken and b) it either started there or possibly a little earlier. If you know there is a line of code a couple of steps before that was also broken since that is earlier, it is a better choice. 

So, now, you have a way of replicating the behavior, the last working point, and a first broken point. In between those two points is a bunch of code, includes function calls, loops, conditionals, etc. The core part of debugging is to bring that range down to the smallest chunk of code possible by moving either the last or first points closer together. 

By way of example, you might know that the code executes a few instructions correctly, then calls a complex function with good data, but the results are bad. If you have checked the inputs and they are right, you can go into that function, moving the last position up to the start of its code. We can move the broken position up to each and every return, handler block, or other terminal points in that function. 

Do keep in mind that most programming languages support many different types of exits from functions, including returns, throws, or handlers attached to exiting. So, just because the problem is in a function, doesn’t mean that it is returning bad data out of the final return statement at the bottom. Don’t assume the exit point, confirm it.

At some point, after you have done this correctly, a bunch of times, you have narrowed the problem down probably to a small set of instructions or some type of library call. Now, you kinda have to read the code and figure out what it was meant to do and watch for syntactic or semantic reasons why that didn’t happen. You’ve narrowed it down to just a few lines. Sometimes it helps to write a small example, with the same input and code, so you can fiddle with it.

If you’ve narrowed it down to a chunk of code that completely works or is completely broken, it is usually because you made a bad assumption somewhere. Knowing that the code actually works is different from just guessing that it does. If the code compiles and/or runs then the computer is fine with it, so any confusion is coming from the person debugging.

What if the code is in a framework and the bug spans multiple entry-points in my code?

It’s similar in that you are looking for the last entry-point that works and the first one called that is wrong. It is possible that the bug is in the framework itself, but you should avoid thinking that until you have exhausted every other option. If the data is right, coming out of one entry point, you can check that it is still right going into the latter one but then gets invalidated there. Most bugs of these types are caused by the entry points not syncing up correctly, corruption is highly unlikely. 

What if I don’t know what code is actually executed by the framework?

This is an all too common problem in modern frameworks. You can attach a lot of code into different places, but it isn’t often clear when and why that code is executed. If you can’t find the last working place or the first failing place, then you might have to put in logging statements or breakpoints for ‘anything’ that could have been called in-between. This type of scaffolding (it should be removed after debugging) is a bit annoying and can use up lots of time, but it is actually faster than just blindly guessing at the problem. If while rerunning the bug, you find that some of the calls are good, going in and good coming out, you can drop them. You can also drop the ones that are entirely bad, going in and bad going out (but they may turn out to be useful later for assessing whether a code change is actually fixing the problem or just making it worse). 

What if the underlying cause is asynchronous code?

The code seems to be fine, but then something else running in the background messes it up. In most debugging you can just print out the ‘change’ of state, in concurrent debugging, you have to always print out the before and after. This is one place where log files are really crucial to gaining correct understanding. As well, you have to consider the possibility that while one thread of execution is making its way through the steps, another thread of execution bypasses it (starts later but finishes first). For any variables that ‘could’ be common, you either have to protect them or craft the code so that their values don’t matter between instructions.

What if I can’t replicate the problem? 

There are some issues, often caused by configuration or race conditions, that occur so infrequently and only in production systems, that you basically have to use log files to set the first and last positions, then wait. Each time it triggers, you should be able to decrease the distance between the two. While waiting, you can examine the code and think up scenarios that would explain what you have seen. Thinking up lots of scenarios is best, and not getting too attached to any of them opens up the ability to insert a few extra log entries into the output that will validate or eliminate some of them. 

Configuration problems show up as the programmer assuming that X is set to ‘foo’ when it is actually set to ‘bar’. They are usually fairly easy to fix, but sometimes are just a small side-effect of a larger process or communications problem that needs fixing too.

Race conditions are notoriously hard to diagnose, particular if they are very infrequent. Basically, at least 2 things are happening at once, and most of the time one finishes before the other, but on some rare occasions, it is the other way around. Most fixes for this type of problem involve adding a synchronization primitive that forces one or the other circumstances, so basically not letting them happen randomly. If you suspect there is a problem, you can ‘fix’ it, even if it isn’t wrong, but keep in mind that serializing parallel work does come with a performance cost. Still, if people are agitated by the presence of the bug, and you find 6 potential race conditions, you can fix all 6 at once, then later maybe undo a few of them when you are sure they are valid. 

If the problem is neither configuration nor a race condition, then most likely it is probably just unexpected data. You can fix that in the code, but also use it as motivation to get that data into testing, so similar problems don’t keep reoccurring. It should also be noted that it is symptomatic of a larger analysis problem as well, given that the users needed to do something that the programmers were not told about. 

Defensive Coding: Minimal Code

 Sometimes you come across really beautify code. It’s clear and concise. It’s obvious how it works. If you have to edit it, it is intuitive where the changes should go. It looks super-simple. It’s a great piece of work.

Most people don’t realize that getting code to look super-simple is a lot of effort and a huge challenge. Just splatting out any initial version is ugly. It takes a lot of thought, refinement and editing work to get it looking great.

All code degrades with time and changes. If it starts out good, it will get tarnished but should hold its value. If it is ugly on day one, it will be a pit of despair a year later.

One way of approaching the problem is to equate super-simple code with the act of minimizing some of the variations until we come down to one with reasonable tradeoffs. We can list out most of these variations.

Minimize:

• The number of variables

• The length of a ‘readable’ name

• The number of external jumps needed in order to understand the code

• The effort to understand a conditional

• The number of flow constructs, such as if statements and for loop

• The number of overlapping logic paths

• The number of hardcoded constants

• The number of disjoint topics

• The number of layers

• The number of reader’s questions

• The number of possible different behaviors

We’ll go through each of them accordingly.

Variables

We obviously don’t want to have the code littered with useless variables. But we also don’t want the ‘same data’ stored in multiple places. We don’t want to overload the meaning of a variable either. And a little less obvious, if there are several dependent variables, we want to bind them together as one thing, and move it all around as just one thing.

Readable Names

We want the shortest, longest name possible. That is, for readability we want to spell everything out in its full detail, but when and where there are different options for that, we want to choose the shortest of them. We don’t want to make up acronyms, we don’t want to make up or misused words, and we certainly don’t want to decorate the names with other attributes or just arbitrarily truncate them. The names should be correct, we don’t want to lie. If the names are good, we need less documentation.

External Jumps

If you can just read the code, without having to jump all over the code base, that is really good. It’s self-contained and entirely under control. If you have to bounce all over the place to figure out what is really happening then that is spaghetti code. It doesn’t matter why you have to bounce, just that you have to do it to get an understanding of how that block of code will work.

Conditionals

Sometimes people create negative conditionals that end up getting processed as double negatives. Sometimes people see the parts of the condition getting spread across a number of different variables. This can be confusing. Conditionals should be easy to understand, so when they aren’t they should be offloaded into a function that is. So, if you have to check 3 variables for 7 different values, then you certainly don’t want to do that directly in an ‘if’ statement. If the function you need to call requires all three variables, and a couple of the values passed, you probably have too many variables. The inputs to a conditional check function shouldn’t be that complex.

Flow of Control

There is some minimum structural logic that is necessary for a reasonable computation. This is different than performance optimizations, in that code with unnecessary branches and loops is just wasting effort. So if you loop through an array, find one part of it, then loop through it again to find the other part, that is ‘deoptimized’. By fixing it, you are just getting rid of bad code, but still not optimizing what the code is doing. It’s not uncommon in ugly code to see that a more careful construction could have avoided at least half of all of the flow constructs, if not more. When those useless constructs go, what is left is way more understandable.

Overlapping Logic

A messy part of most programming languages is error handling. It can be easily abused to craft blocks of code that have a large number of different exit points. Some necessary error handling supports multiple different conditions that are handled differently, but most error handling is rather boolean. One can mix the main logic with boolean handling and still have it readable. For more sophisticated approaches, the base code and error handling usually need to be split apart in order to keep it simple.

Hardcoded Constants

Once people grew frustrated by continually hitting arbitrary limits where the programmers made a bad choice, we moved away from sticking constants right into the code. Modern code however has forgotten this and has returned to hardcoding all sorts of bad stuff. On rare occasions, it might be necessary, but it always needs to be justified. Most of the inputs to the code should come through the arguments to the function call whenever possible. 

Disjoint Topics

You can take two very specific functions and jam them into one bigger function declaration. The code for each addresses a different ‘topic’, they should be separated, they shouldn’t be together. Minimizing the number of functions in code is a very bad idea, functions are cheap but they are also the basis of readability. Each function should fully address its topic, the code at any given level should all be localized.

Layers

The code should be layered. But taken to the extreme, some layers are useless and not adding any value. Get rid of them. Over minimization is bad too. If there are no layers, then the code tends towards a suer-large jumbled mess of stuff at cross-purposes. It may seem easier to read to some people since all of the underlying details are smacked together, but really it is not, since there is too much noise included. Coding massive lists of instructions exactly as a massive list is the ‘brute force’ way of building stuff. It works for small programs but goes bad quickly because of collisions as the code base grows.

Reader’s Questions

When someone reads your code, they will have lots of questions, Some things will be obvious, some can be guessed at given by the context, but somethings are just a mystery. Code doesn’t want or need mysteries, so it is quite possible for the programmer to nicely answer these questions. Comments, comment blocks, naming, and packaging all help to resolve questions. If it’s not obvious, it should be.

Different Behaviors

In some systems, there are a lot of interdependent options that can be manipulated by the users. If that optionality scrambles the code, then it was handled badly. If it’s really an indecipherable mess, then it is fundamentally untestable, and as such is not production worthy code. The options can be handled in advance by moving them to a smaller set of more reasonable parameters, or polymorphism can be used so that the major permutations fall down into specific blocks of code. Either way, giving the users lots of choices should also not give them lots of bugs. 

Summary

There are probably a few more variations, but this is a good start. 

If you minimize everything in this list, the code will not only be beautiful, but readable, have way fewer bugs and people can keep extending it easily in the future. 

It’s worth noting that no one on the planet can type in perfect code the first time, directly from their memory. It takes work to minimize this kinda stuff, and some of the constraints conflict with each other. So, everyone’s expectation should be that you type out a rough version of the code first, fix the obvious problems, and then start gradually working that into a beautify version. When asked for an estimate, you include the time necessary to write the initial code but also the time necessary to clean it up and test it properly. If there is some unbreakable time panic, you make sure that people know that what is going into production is ugly and only a ‘partial’ answer and still needs refining.

Defensive Coding: KISS

Nothing gets programmers into more hot water more than the ‘keep it simple, stupid’ (KISS) principle.

There are 2 important ways it causes grief. 

The first is when programmers assume it applies directly to them. They want to keep their work simple, so they want to keep their code simple.

The failure here is that there is always some ‘intrinsic’ complexity that is totally unavoidable as a byproduct of our physical universe and its history. If a programmer ignores some of that complexity and makes their code super simple, the complexity hasn’t gone away, it has just moved elsewhere. More often than not, it has gone directly to the users. 

So, now the code is simple, but using it has become significantly more complicated than necessary. A super simple feature either doesn’t do much or it is horrifically hard for users to get real things accomplished. Either way, its value is limited.

If you want to keep users interested in using a system, then the code in it has to really solve their problems and it has to keep it simple for them. KISS applies to the solution, not to the way it was built. That is not simple code to write since it means ensuring that the users don’t need to lean on any outside resources. The system needs to know anything and everything about the problem, to keep it all up-to-date, and to never forget stuff, which is, of course, large and very complicated. 

KISS can be applied to a specific sub-component of the mechanics, but it only works if you are avoiding some explicit overcomplication. That is, if you are thinking about adding stuff that will definitely never get used, then you can remove that. However, if it might get used someday, then by removing it, you are actually setting up a new problem. Maybe a small one, but it also could be a huge one in a short time from now.

The other way programmers get into trouble with KISS is that they assume that if they have to build a series of components, that making each one of them simple will help simplify the whole thing. The opposite is often true.

Simplifications, almost by definition are things that are not there. Gaps in what could have been. Those things might have been necessary or they could have been completely extra, but they are still absent. Collecting together a bunch of non-related things all together in the same place is another way to describe disorganization, and oddly, collecting together a bunch of missing things, related or not, is almost the same. 

The growing absence of stuff is a form of complexity and if not handled explicitly it will get more complex. So, a small piece missing from one area can be worked around fairly easily, but when there are dozens of things missing all over, the workarounds are harder and more complex. Just remembering what isn’t there becomes a huge burden. So, if there are lots of things that are missing, their combined effect will be multiplicative, not additive, if they are not 100% independent (which they usually aren’t). Simple is not a cumulative property, complexity is. 

“Everything should be made as simple as possible, but no simpler.” -- Albert Einstein

The key part of this quote is that while you can overcomplicate things, you can also do the opposite and oversimplify them, which is bad too. When KISS is misapplied or taken to an extreme not only does it craft a system that users hate, but it will also slowly degrade into a horrific ball of complexity that at some point becomes unfixable. 

It’s always a shock to the programmers, particularly if they have been diligent about trying to simplify everything when it suddenly kicks back up and becomes the problem itself.

Stuff that is missing or not included is always going to be a big problem if you find out later that you need it. It’s that old ‘a stitch in time save nine’ saying. Filling the code with a lot of extra stuff that isn’t used ain’t great, but then it is a little better than waking up one day and realizing that what you didn’t do earlier -- when you had the chance -- is now going to derail everything.

Don’t overcomplicate things, but don’t use that as an excuse to swing way out to the opposite extreme, either. 

Developer Management

It’s very difficult to bring a group of software developers together and get a reasonably well-built software system out of the process.

The problems stem from the underlying programming culture. Coders tend to be introverted, their thinking is often extremely black-and-white and they go rogue quite frequently. 

This leads to a number of unfortunate outcomes. 

First, they are not really good at communicating the problems and issues that they are having getting the system built and running. They don’t like to admit problems. Coupled with the modern impatience to get things done too quickly, this often reduces the timeframes to considerably less than is what is necessary to do a good job. If the stakeholders aren’t aware of many of the ongoing problems, they aren’t going to have realistic expectations on the progress of the work.

Coding requires a rather rigorous logical mode of thinking. Stuff either works or it doesn’t. Things are either right or they are wrong. The computer does exactly what the programmer tells it to do, there is no room for ambiguities. If you spend your day writing code with these properties, it tends to leak out into the rest of your thinking, life, and interactions with people. The world, however, is grey and murky, and often requires a soft touch to work through the issues. A big team of people working together generates a lot of different agendas and politics. None of this is nicely black and white, so there is a lot of friction between how the programmers think the world ‘should’ operate and how it actually does operate. This leads to a lot of misunderstandings, anxiety, and confused goals. 

With a different perspective on the priorities, and a desire to not want to talk about it, programmers are infamous for just making a decision on their own and then going off with full confidence to get those things done. The problem is that those attempts are often not in sync with the rest of the project, so they basically go rogue and end up doing something that doesn’t provide value. A sub-group of coders incorrectly heading towards the wrong objectives will conflict with the important ones, so the result is poor or useless work.

It’s hard for management to distinguish between a rogue programmer and one not doing any work at all. In both cases, what gets accomplished is nothing. Usually given that their expectations are off too, this builds up a lot of tension between the developers and management. 

In the past, people like to blame the “waterfall methodology” for going off in the wrong direction and returning with stuff that was not useful. They insisted that it was the methodology that was at fault, that it was a ‘time’ problem, but there is a little more to it than that. 

If it's a big, well-defined project that takes 1.5 years to accomplish, doing it in one long continuous unit of work is a whole lot more efficient than breaking it up into little iterations and trying to stitch them together somehow. Mostly batching together similar work is more effective, is better for consistency, and for focus. 

The big failures before the turn of the century drove the stakeholders to seek out better ways of tightly controlling their software projects. The culture of programming itself helped. Both sides settled on an invasive form of micromanagement. The coders seeded control of their work. So, the prerequisites for deciding on the right work like analysis and design get ignored, while the whole effort is gamified with a rather childish bent on formality. You don’t have “excessive” daily status meetings, you have ‘standups’ instead. There isn’t a long list of pending work, it’s a ‘burndown’ chart. We don’t break down the work into tiny, little, verifiable chunks, it's called a ‘sprint’. Planning is a game, not a necessity, and the stick is called a ‘retro’ which is somehow supposed to good for you. 

Each time management was compelled to reach for the thumbscrews and lockdown inappropriate behavior, consultants came up with cutesy little names and games for implementing it, and pushing it as propaganda to the rest of the industry. It’s unfortunate. 

For me though, the fundamental problem is not upper management controlling how the system is constructed. Rather, it is the role of leading a group of developers that is confused. It’s not a technical role and it's not a business role. 

Ultimately, there are some high-level goals that need to be accomplished. The people setting those goals do not have the ability to break them down and express them as a very, very long list of complicated programming tasks. That’s the communications impedance mismatch. If you hire a bunch of programmers and can’t tell them what to do, and they won’t tell you what problems they are having, then it is pretty obvious that the project is not going to function.

So, you need an intermediary. Someone who has spent a lot of time programming, but also someone who has been around the higher-level objectives enough to understand them. They have to have a foot in both worlds because they have to accurately translate between them. 

They might not be the best programmer, or be able to solve silly little fake coding issues. They just need to have spent time in projects of similar scale. They need to get their priorities straight. They might not fully understand all of the business objectives, or be a domain expert, but they need to have empathy for the users and to have some depth in their specific domain problems. They sit in the middle, and they ensure that the upper goals are progressing, while the lower work isn’t going rogue. 

Over the decades, I’ve heard many an entrepreneur reach the conclusion that all they need is a group of students that can code a little bit in order to get their product to market. That’s kind of the classic delusion. It sees coding as a commodity that just requires enough ‘energy’ to drive it forward. Oddly, that can work for a demo or a proof-of-concept or some other introductory software that just needs to kinda work in order to grab more interest, but it fails miserably once it becomes real, mostly because the necessary skills to keep it all organized and keep it growing are missing. So, it’s a start that can be used to evaluate ideas, but not a product that will work when needed. 

Moving up to that next level means getting serious about keeping the work under control. It’s not a small gap, but actually a rather huge one. It’s not intuitive and the kids that threw together the prototype code won’t be able to magically pull it from the ethos. This is where coding switches from being a game to getting serious. It can’t afford to be ineffective anymore, it can’t afford to be disorganized anymore. Everything changes.

For medium, large and massive projects, even the smallest issues have huge, wide-ranging consequences. You learn to deal with them via experience, and it is these issues that are far more important at this point, than the actual coding itself. Fixing the code is cheap, unrolling a ball of mud is expensive. An intermediary who knows it is important to restrict ongoing dependencies, for example, is a much better asset than a coder who can craft unique algorithms. The wrong algorithm is useless, while the wrong dependencies are often fatal. 

In an industry known for its agism, and for still having a high rate of failure, you’d think it would be obvious by now that we’d know there is a missing critical component in the effort. But oddly, the stakeholders still think programmers are just cogs, and the coders still think that if they just had “more code” their problems would magically disappear. The technologies have changed, the methodologies have gotten crazier, but the underlying problems are still the same. Breaking up a months’ worth of work up into hundreds of artificial 2-week tasks doesn’t ensure that it will go any better or be more appropriate. Instead, it tends to build up a counter-culture of gaming that process. Since it’s all indecipherable from above, it does nothing to ensure that progress is actually moving as best as possible. It just provides a false sense of momentum. And the games that coders play may distract them for a while, but the necessary satisfaction from doing a good job is missing, so they aren’t getting what they want either. 

Part of programming is really boring, routine, software production. It’s just work that needs to be done. Some small parts of getting a big product out to market are creative, but more often than not the creative portions fall into the business, design, and architectural efforts. If the ideas and issues are worked through in advance, and any difficult technological issues are prototyped up front, then the rest of getting out a new release is just the careful assembling of all of the pieces in an organized manner. It’s not a game, it’s not a contest. Having someone who knows what is really important during this phase of the work is going to prevent a lot of predictable quality issues from materializing. Like any other profession, programming isn’t “fun”, but when it is done well it can be quite satisfying. 

Puzzles

I’ve always enjoyed doing large jigsaw puzzles. You dump out the box of pieces on a big table, turn them all over, find the edges pieces, and then disappear into the problem. For me, it is relaxing and somehow it recharges my batteries. 

Programming has similar qualities but is a little different. It is grabbing a lot of pieces and assembling them into a big picture, but instead of there saying 1000 pieces that we know belong to the final picture, there is almost an endless number of them. In a puzzle each piece has a pre-defined place, in programming, each place has a large number of different options.

In a puzzle, we know what the final picture should be. It’s basically decided in advance, long before it was printed on the cardboard, long before it was cut up. In programming, the big picture is there too, but it has a large degree of flexibility, it’s vague and often people have wildly different images of it. 

In a puzzle, there are many different ways you can tell that you have connected the wrong piece. The knob might not fit tightly, the edges are different sized or the images don’t align. Since the end goal is to assemble the whole puzzle, a misfitting piece is wrong twice. It displaces the piece that should be there, but it also is not placed in its own correct location. If the image is close but subtly-wrong, it can cause quite a delay as it can also interfere with the placement of the pieces around it. 

In programming though, since most people aren’t aware of the big picture, they just put the pieces somewhere that kinda fits. Then they force it in. So, if you could imagine a puzzle with no edges and mostly fungible pieces, people assembling it would be quite fast, but the outcome would be very unpredictable. That’s closer to what happens in coding. Without a big picture, a lot of programmers just pound in the pieces and then stick to the assertion that it was right. When it’s not, then what emerges is a scrambled big picture, that is incredibly difficult to add more pieces too. 

At it’s best, I enjoyed putting together software systems. It has a similar vibe to puzzles. You just focus on finding the right pieces, keep it somewhat organized and eventually the picture or the system emerges from the effort. At its worse though, people are furiously pounding in pieces, arguing about their validity and the whole thing is just a growing mess. 

That kinda gives us a way of thinking about how to fix it. The first point is obviously that the people working on the pieces, need to understand the big picture. You can’t just build a working system by randomly tossing together logic. It’s not going to happen. You need the big picture, in the same way, you need it for a puzzle. It keeps you on track, it prevents mistakes. 

The second point is that forcing a non-fitting piece to connect to other pieces is a bad idea. In most puzzles, you get a sense that the piece you are adding is the ‘right’ piece. It fits, the images line up, it is where it is supposed to be. The same is actually true in programming. There is a very real sense of ‘fit’. The code nicely snaps into place, it is readable and it feels like it is in a place where it belongs. Some people are more sensitive to this than others, but most programmers do get a real sense of misfitting, some are just taught to ignore it for the sake of speed. 

Still, what makes a puzzle fun to me at least, is not that I can do it super fast, but rather that I focus in on it and enjoy the process. The people around me may want it faster, but I have learned to take it at a smooth and reasonable speed.

Defensive Coding: Names and Order

A fundamental rule of programming is that the code should never lie. 

One consequence of that is that a variable name should be correctly descriptive of the data held inside. So, a variable called ‘user’ that holds ‘reporting data’ is lying. Report data is not user data, it is incorrect.

A second consequence is that in the system if most of the variables that are holding data about people interacting with the system are called ‘user’, then pretty much every variable in the system that holds the same data should have the same name. Well, almost. If one part of the system only deals with a subset of the users who have admin privileges, then it’s okay that the variable name is ‘admin_user’. Semantically, they are an admin user, which is a subset of all users. The variable name can be scoped a little more tightly if that increases its readability or intent. Obviously though, if in another part, the user data is passed into a variable called ‘report’ then that code is busted.

In some lower-level parts, we might just take the user data, the reporting data, and the event data, convert them all into a common format, and pass that somewhere else. So, the name of the common format needs to be apropos to the polymorphic converted data that is moving around. It might be the format, like ‘json’, or it could be even more generic like a ‘buffer’. Whatever helps best with the readability and informing other programmers about the content. General names are fine when the code has been generalized.

In most systems, the breadth of data is actually not that wide. There are a few dozen major entities at most. For some people, naming things is considered one of the harder parts about coding, but if variables are named properly for their data, and the system doesn’t have that many varieties of data anyways, and not a lot of new ones coming in, then the naming problem quickly loses its difficulty. If the data already exists and it is named properly, a new variable holding it elsewhere should not have a new name. There is no need for creativity, the name has already been established. So, naming should be a rather infrequent issue, and where readability dictates that the name should be tightened or generalized, those variations are mostly known or easy to derive. 

The other interesting aspect is that there is an intrinsic order to the entities. 

If we were writing a system that delivers reports to users, then the first most important thing we need to know is “who is the user?” That is followed by “what report do they get?” Basically, we are walking down through the context that surrounds the call. A user logs in, then they request a report. 

What that means is that there is a defined order between those two entities. So for any function or method call in the language, if both variables are necessary, they should be specified in that correct order. If there are 3 similar functions, ‘user’ always comes before ‘report’ in their argument lists. 

Otherwise, it is messy if some of the calls are (user, report) and others are (report, user). The backward order is somewhat misleading. Not exactly incorrect, but some pertinent information is lost.

Now if the system has a couple of dozen entities, the context may not be enough to make the order unambiguous, but there is also an order that comes from the domain. Generally, the two, have been enough to cover every permutation.

The corollary to this is that if we are focused on preserving order, but we have code that instead of passing around the entities, is passing around the low-level attributes individually, it becomes clear that we should fix that code. Attributes that are dependent should always travel together, they have no meaning on their own, so there is no good reason to keep them separated. That is, we are missing an object, or a typedef, or whatever other approaches the language has to treat similar variables as a single thing. When that is done, the attributes disappear, and there are fewer entities, and of course less naming issues.

What’s interesting about all of this is that if we set a higher level principle like that ‘code should never lie’, the consequences of that trickled down into a lot of smaller issues. Names and order are related back to keeping the code honest. To be correct about one means being consistent about the other two.

Defensive Coding: Locality

A property of spaghetti code is that it is highly fragmented and distributed all over the place. So, when you have to go in and see what it is doing, you end up getting lost in a ‘maze of twisty passages’.

Code gets that ugly in two different ways. The first is that the original base code was cloudy. That is, the programmer couldn’t get a clear view of how it should work, so they encoded the parts oddly and then mashed them all together and got them to work, somehow. The second is that the original code might have been nice, tight, and organized, but a long series of successive changes over time, each one just adding a little bit, gradually turned it into spaghetti. Each change seemed innocent enough, but the collective variations from different people, different perspectives, added up into a mess.

The problem with spaghetti is that it is too cognitively demanding to fix correctly. That is, because it is spaghetti there is a higher than normal likelihood that it contains a lot of bugs. In a system that hasn’t been well used, many of these bugs exist but haven’t had been noticed yet. 

When a bug does rear its head, someone needs to go into this code and correct it. If the code is clear and readable, they are more likely to make it better. If the code is a mess, and difficult, they are more likely to make it worse but hide those new problems with their bugfix. That is, buggy code begets more bugs, which starts a cycle.

So, obviously, we don’t want that. 

To see how to avoid this, we have to start at the end and work backward. That is, we have to imagine being the person who now has to fix a deficiency in the code. 

What we really want is for them to not make the code worse, so what we need to do is put all similar aspects of what the code does, in as close proximity to the code as possible. That is, when it is all localized together, it is way easier to read and that means the bug fixer is more likely to do the right thing. 

Most computations, in big systems, naturally happen in layers. That is, there is a high-level objective that the code is trying to achieve to satisfy all or part of a feature. That breaks down into a number of different steps, each one of them is often pretty self-contained. Below that, the lower steps generally manipulate data, which can involve getting it from globals, a cache, persistence, the file system, etc. In this simple description, there are at least three different levels, that have three very different focuses. 

When we are fetching data, for example, we need some way to find it, then we need to apply some transformations on it to make it usable. That, in itself, is self-contained. We can figure out everything that is needed to get ‘usable’ data from the infrastructure, we don’t have to be concerned with ‘why’ we need the data. There is a very clear ‘line’ between the two. 

So, what we want from our layers, is clear lines. Very clear lines. An issue is either 100% on one side, or the other, but not blurred. 

We do this because it translates back to stong debugging qualities. That is, in one of the upper layers, all one needs to do is check to see if the data is usable, or not. If it is, the problem is with the higher-level logic. If not, then it is how we have fetched the data that is the real problem. Not only have we triaged the problem, but we’ve also narrowed down to where the fix may need to go.

There have often been discussions amongst programmers about how layers convolute the code. That is true if the layers are arbitrary. There are just a whole bunch of them, that don’t have a reason to exist. But it is also true that at least some of them are necessary. They provide the organizational lines that help localize similar code and encapsulate it from the rest of the system. They are absolutely necessary to avoid spaghetti.

Getting back to the three-level basic structure discussion, it is important that when there is a problem, a programmer can quickly identify that it is located somewhere under that first higher level. So, we like to create some type of ‘attachment’ between what the user has done in the application and that series of steps that will be executed. That’s fundamentally an architectural decision, but again it should be clear and obvious. 

That means that in the best-case scenario, an incoming programmer can quickly read the different steps in the high-level, compare them back to the bug report and narrow down the problem. When the code is really good, this can be done quickly by just reading it, ponding the effect, and then understanding why the code and the intent deviated. If the code is a bit messier, the programmer might have to step through it a few times, to isolate the problem and correct it. The latter is obviously more time-intensive, so making the code super readable is a big win at this point. 

What happens in a massive system is that one of the steps in a higher-level objective drops down into some technical detail that is itself a higher-level objective for some other functionality. So, the user initiates a data dump of some type, the base information is collected, then the dump begins. Dumping a specific set of data involves some initial mechanics, then the next step is doing the bulk work, followed by whatever closing and cleanup is needed. This lower functionality has its own structure. So there is a top three-layer structure and one of the sub-parts of a step is another embedded three-layer structure. Some fear that that might be confusing. 

It’s a valid concern, but really only significant if the lower structure does something that can corrupt the upper one. This is exactly why we discouraged globals, gotos, and other side-effects. If the sub-part is localized, correctly, then again either it works, or it doesn’t. It doesn’t really matter how many levels are there. 

In debugging, if the outputs coming back from a function call are sane, you don’t need to descend into the code to figure out how it works, you can ignore that embedded structure. And if that structure is reused all over the system, then you get a pretty good feeling that it is better debugged than the higher-level code. You can often utilize those assumptions to speed up the diagnosis.  

So we see that good localization doesn’t just help with debugging and readability, it also plays back into reuse as well. We don’t have to explicitly explore every underlying encapsulation if their only real effect is to produce ‘correct’ output. We don’t need to know how they work, just that the results are as expected. And initially, we can assume the output is correct unless there is evidence in the bug report to the contrary. That then is a property that if the code is good, we can rely on to narrow down the scope to a much smaller set of code. Divide and conquer. 

The converse is also true. If the code is not localized, then we can’t make those types of assumptions, and we pretty much have to visit most of the code that could have been triggered. So, its a much longer, and more onerous job to try and narrow down the problem.

A bigger codebase always takes longer to debug. Given that we are human, and that tests can only catch a percentage of bugs that we were expecting, then unless one does a full proof of correctness directly on the final code, there will always be bugs, in every system. And as it grows larger, the bugs will get worse and take longer to find. Because of that, putting in an effort to reduce that problem, pays off heavily. There are many other ways to help, but localization, as an overall property for the code, is one of the most effective ways. If we avoid spaghetti, then our lives get easier.

Normalization Revisited

Over 10 years ago, I wrote 3 rather rambling posts on ideas about normalization:

1. http://theprogrammersparadox.blogspot.com/2008/11/code-normal-form.html
2. http://theprogrammersparadox.blogspot.com/2008/10/structure-of-elegance.html
3. http://theprogrammersparadox.blogspot.com/2008/10/revisiting-structure-of-elegance.html

It’s not a subject that has ever grabbed a lot of attention, but it’s really surprising that it didn’t. 

Most code out there is ‘legacy code’. That is, it was written a while ago, often by someone who is gone now. Most projects aren’t disciplined enough to have consistent standards, and most often any medium-sized or larger system consists of lots of redundant code, a gazillion libraries, endless spaghetti, broken configurations, no real documentation, etc. 

Worse, a lot of code getting written right now depends on this pre-existing older stuff and is already somewhat tainted by it. That is, a crumbling code base always gets worse with time, not better.

These days we use good automated tooling like gofmt (in Golang) or rubocop (in Ruby) for different languages that are capable of enforcing light standards by automatically fixing the syntax, often they are set to reformat during saving in the editor. This lets programmers be a bit sloppy in their coding habits but auto-corrects it before it stays around for any length of time. 

Just putting some of these formatting tools into play in the editors is a big help in enforcing better consistency, getting better readability, and thus better overall quality.

What does that have to do with normalizations? The idea behind normalizing things is that much like in a relational database, there is a small set of rules that control relationship properties. In a database, it is applied to structural issues for the data. In code, it can also be applied to structural issues. 

The execution of code is a serialized list of instructions that each processor in the computer follows, but we often see it as a “tree” of function calls. To do this each function is a node, calling all of the other functions as children. Mostly, it looks like a tree, but since we can have reuse, and there can be infinite loops, it’s really a directed graph. A stack dump then is just one specific ‘path’ through this structure. If we dumped the stack a lot of times and combined it together we’d get a more complete picture.

We can take any of the instructions in this list of execution steps and move them ‘up’ or ‘down’ into other function calls in the graph. We can break up or reassemble different function calls, nodes, so that the overall structure has properties like symmetry, consistent levels, and encapsulated component structure.

It is probably incredibly slow to take a lot of code and rework it into a ‘semantic’ structure, but once that is in place, it can be shifted around based on the required normalizations, then returned to the source language. It might require a lot of disk space and a lot of CPU, but the time and resources it takes are basically irrelevant. Even if it needed a few weeks and 100 Gb, it would still be incredibly useful. 

The idea is that if you take a huge pile of messy code, and set up a reasonable number of ‘non-destructive’ normalized refactors, it would just go off and grind through the work, cleaning up the code. It absolutely needs to be non-destructive so that the final work is trustworthy (or at least as trustworthy as the original code), and it needs to be fully batched, to avoid a lot of user interaction. 

You could literally set it running, go off for a two-week vacation, and come back to a trustworthy, cleaned up codebase, that had exactly 0 operational differences from the mess you had before you left. You could just flip that into production right away, without even having to do regression testing.

A well-set group of rules could clean up variable naming, shift around the layers to unwind spaghetti, put a scope on globals, and even put the basis there for commenting. As well, it could identify redundant code, even if it doesn’t do the merge itself, and it could definitely identify useless data transformations, missing error handling, inconsistent calls, and a huge number of trivial and obvious problems. Basically, it’s a code review on steroids but applied across the entire system, all at once.

It could create menial unit tests, redo the configurations, shift behavior from one type of resource to another (config file -> database, or vice versa). Basically a huge number of super useful cleanup work that programmers hate to do and procrastinate on doing until it causes grief.

To deal with interactivity, suggested changes could be put into comment blocks. So, it generates a better method in the code, comments it out, but with the same name as the replacement method. Much like dealing with SCM merges, you could read both versions, then pick the better one (and add some type of test to vet it properly). 

Once the code is normalized, it takes considerably less cognitive effort to read through it and see the problems. Most of them will be obvious. Starting something, without closing it. Doing weird translations on the data, not checking for errors, and not doing any validations on incoming data. Most bugs fall into these categories. It could also help identify race conditions, locking issues, and general resource usage. For instance, if you normalize the code and find out that it has 4 different objects to cache the same user data, it’s a pretty easy fix to correct that. Bloat is often a direct consequence of obvious redundancies. 

You could look at the resulting code right away, or just leave it to explore on an as-needed basis. Since it’s all consistent and follows the same standards, it will be way easier to extend. 

It would also be useful if it has an idiom-swapping mechanism, for languages where there are too many different ways to accomplish the same thing. If there are 4 ways to doing string fiddling, then swapping the other 3 down to 1 consistent version helps a lot for readability. 

What kills a lot of development projects is that the code eventually becomes so messy that people are afraid to change it. Once that happens, either it goes crazy as an onion architecture (new stuff just wraps around the old stuff with lots of bugs) or it slows down into maintenance mode. Either way, the trajectory of the codebase is headed downwards, often quite rapidly. 

If all it took to get out of that death spiral was to spend a week fiddling with the configuration and then a week of extreme processing, that would make a huge difference. 

Code is encapsulated knowledge about the domain and/or technical problems. That is a lot of knowledge, captured over time, of intermittent quality, and it would be a huge amount of work to go backward, throwing all of it away and starting over. Why? If you can take what was already there, already known, and leverage it, but not as an unreadable opaque box, rather a good codebase that is extendable, then life just got easier. People are somewhat irrational, programmers all have different styles, and as a big ‘stew’ of code, it is overwhelming. If you can just normalize it all into something you want to work on, then most of those historic influences have been mitigated. 

Time and Shortcuts

I haven't posted anything for a while now. This has been the longest gap I've had since I started blogging five years ago. It's not that I don't have any anything to say -- my hard drive is littered with half-finished posts -- but rather that I just haven't had the time or energy to get my thoughts down nicely.

Lately I've been wrapped up in a large complicated system that has a wide array of problems, none of which are new to me but it's unusal to see them all together in the same project. I like this because it is extremely challenging, but I definitely prefer projects to be well-organized and focus on engineering problems, not organizational ones. 

My sense over the last couple of decades is that software has shifted from being an intense search for well-refined answers to just a high-stress fight against allowing all of the earlier shortcuts to swamp the project. I find that unfortunate because the part of the job that I've always loved was that satisfaction from building something good. Just whipping together a barely functoning mess, I find depressing.

 

What I've noticed over the years is that there is a whole lot more progress made when the code is well-thought out and clean at all levels (from syntax to the interface). The increasingly rare elegant solution takes it one step higher. You get so much more out of the work, since there are so many more places to leverage it. But of course spending the time to get it right means coming out of the gate that much slower, and it seems that people are increasingly impaitent these days. They just want a quick bandaid whether or not that will make the probem worse. 

Getting a mess back under control means having to say 'no' often. It's not a popular word and saying it comes with a lot of angst. It does not help that there are so many silver bullet approaches floating about out there. Anyone with little software knowledge is easily fooled by the over abundance of snake charmers in our industry. It's easy to promise that the work will be fast and simple, but experience teaches us that the only way to get it done is hard work and a lot of deep thinking. Writing code isn't that hard, you can teach it to high school students rapidly, but writing industrial strength code is a completely different problem.

I'd love to take off some more time and write another book that just lists out the non-controversial best practices that we've learned over the last few decades -- a software 101 primer -- but given that my last effort sold a massive 56 copies and I'm a wage slave it's not very likely that I'll get a chance to do this anytime soon. The trick I think is to shy away from the pop philosophies and stick to what we know actually works. Software development is easy to talk about, easy to thoerize about, but what often really works in practice is counter-intuitive for people with little experience. That's not unusal for complex systems and a large development projects contains millions of moving parts (people, code, technology, requirements, data, etc) with odd shifting dependencies. You can't understand how to organize such a volitile set of relationships without first devling deep into actual experience and even then it's hard to structure the understanding and communicate it. 

What flows around the production of the code is always more complex than the code itself and highly influenced by its environment. A good process helps the work progress as rapidly as possible with high quality results, most modern methodologies don't do that. That's one of our industries most embarrassing secrets and it seems to be only getting worse.

Hopefully one of these days I'll catch my breath and get some of my half-finished posts completed. There are some good lessons learned buried in those posts, it just takes time and patience to convert them into some shareable.