Herding Cats

The idiom about “herding cats” is frequently used in IT and plays out in many different places in the industry.

It adequately describes programmers. Because the industry keeps expanding far faster than people can follow, the culture tends to overcompensate. Programmers are frequently crazy overconfident in their beliefs, views, and opinions, as it oddly tends to help in their careers. But it also means a high rate of failure and a lot of personality clashes.

I always figured that if we got the 10 greatest programmers on the planet all together on the same project, instead of producing magic, they would just dissolve into bickering with each other. The cats all want to go off in their own direction.

That tendency plays out in technology as well.

Microservices was a super popular idea. Instead of everyone coming together to build something, it offers the dream that everyone could just build little fully independent pieces that would all magically work together. So, you don’t have to compromise for the team, you just do what you know is right, toss it into the mix, and it will be good.

But of course, it doesn’t work that way. If at the binding level, the sum of the services is hugely disorganized and irrational, the whole thing just isn’t going to be useful or stable.

The health of the ‘forest’, dominates the health of the ‘trees’. Each cat tending to their own little tree without having to worry about the other trees only works if all of the trees are contained and arranged nicely. If the “tree” however, is really various branches from a lot of other trees scattered all over the forest, it's nuts. The parts need to be perfectly independent and totally aligned, which is a contradiction.

It shows up in products too.

Most, if not all, big products were evolutionary. A small group of people got it going, and then it was scaled way up over a very long time to reach its current level of maturity. So, it’s not uncommon that hundreds or even thousands of programmers were involved over decades.

That initial small group likely had a strong tight focus, which is why the product didn’t fail. But as people come and go that focus gets watered down, realigned, and bounced all over the place. That shows up in the product. It goes from being a tight nicely arranged set of functionality to being a pile of mistaching features, erratically shoved into strange corners of the interface.

Before maturity, there wasn’t as much functionality, but you could always find it. After maturity, everything including the kitchen sink is there, but now it is tricky or impossible to find, and behaves inconsistently with everything else.

We see it with in-house software as well. A friend of mine used to love to say that companies get the software they deserve. If you look at most big non-software companies, their IT depts are a whacking great mess of chaos, bugs, and defective code.

Since personality issues tend to be the downfall of large internal projects, most of the stuff is little bits of glue haphazardly applied to everything.

It isn’t even a ball of mud, but rather a gaint field of wreckage. And the software industry loves this. They endlessly keep selling magic bullets to fix the ever-growing disaster, but generally just make it worse somehow. Generation after generation of technology that makes bold claims and then disappoints.

In so long as programmers want the freedom to go off and do their own thing, ignoring what they don’t want to do or know, then the results of their work have strictly limited usefulness.

If it is fully eclectic, it fits with nothing. Yes, it is a creative work, but that creativity is what keeps it from being useful.

If it is fully standard, to whatever the larger context is, there is almost no freedom whatsoever. You figure out the standards, implement them correctly, and then put state-of-the-art pieces below. Almost no creativity, but a massive amount of prior knowledge. The thing is great and works with everything else, but it is as it should be, not as you’d like it to be.

The work is figuring out what it should be. This is why the programmers who can successfully work with the greatest programmers on the planet have more value than their peers. They are humble and know that creativity isn’t necessary. Research is a better use of their time. Get as close to standard as you can in the time you are allowed.

The broader theme is that ‘focus’ is a huge part of quality. If the work is large and unfocused, there will be plenty of problems with it and if there are too many of them it undoes any or all of the value of the work itself. If, however, the work is tightly focused then most of the contributors must orient their work in that direction, which takes a lot of the fun out of it. More focus means less freedom, but also better results. There does not seem to be any way around this.

Development Automation

One popular programming fallacy is that it is not worth spending five hours writing a script that runs in a few minutes.

Mostly, this perspective seems to come from people who don’t like scripting. They want to have just one primary coding language; they only want to stay working in that language all of the time. Scripting usually requires at least one other language and lots of little tools. It is different from the other coding tasks. It’s more stuff to learn.

This perspective is wrong for a bunch of reasons.

First, if you spend five hours writing a script and then run it thousands of times, it obviously pays off. It still pays off even if you run it hundreds of times, and probably even if you run it only a dozen.

But if you’ve scripted all sorts of stuff, even if a few of those scripts don’t get run very often, the more you script, the faster you will get at doing it. Practice makes perfect, but it also makes it faster.

If you’ve mostly scripted everything, then creating new variations on other scripts is nearly trivial. You have lots of examples to work off of. You don’t want a billion brute force scripts, but most of the things you are automating are essentially similar. Sometimes you refactor to expand scope (preferred), but other times you just copy and modify to get it going.

And whenever you run the script, the results will be deterministic. A good script will run the same every time. You have at least semi-automated the tasks if not better. That’s what we do for users; why not for ourselves?

If, instead of scripting, you need to click on 10 buttons each time, the chances are that you’ll miss some steps. If you need to type 10 long CLI commands, it is even worse. More chances to forget stuff.

The cost of getting it wrong is always way more expensive than the automation costs. You do a PROD install and forget a tiny piece; it isn’t just the time you have blown, it is also the loss of trust from the people around you. Oddly, that often results in more meetings and more tracking. A history of screwups is very costly.

And, of course, documentation. If someone new starts, you point them to the scripts. If you step away for a while, you point yourself to them. They are a record of the things you are doing every day. You can’t really share them with non-technical people, but that is fine, even better.

Finally, encapsulation. If you bury the ugly grunge into a script, you can forget about it. Pretend it never happened. You’ve stripped off some small annoying piece of complexity and replaced it with a simple command. If you keep that up, there is far less friction and far more accuracy, which gives you far more time to concentrate on the newer, bigger, far more interesting problems.

Scripting everything, or at least as much as you can, is and has always been a super strong habit. Part of mastering coding.

Ironically, while the main languages have changed radically over the decades, scripting has not. You can script well with early 90s techniques, and it will be as useful today as it was then.

Concurrency

For large complex code, particularly when it is highly configurable, polymorphic, or has dependency injection, it is sometimes difficult to get a clear and correct mental model of how it actually runs. The code might bounce around in unexpected ways. Complicated flow control is the source of a lot of bugs.

Now take that, but interlace multiple instances of it, each interacting with the same underlying variables, and know that it can get far worse.

The problem isn’t the instructions; it is any and all of the variables that they touch.

If there is a simple piece of code that just sets two variables, it is possible that one instance sets the first one, but then is leapfrogged by another instance setting both. After the first instance finally sets the last variable, the state is corrupt. The value of the variables is a mix between the two concurrent processes. If the intent of the instructions was to make sure both are consistent, it didn’t work.

If it was just an issue with a single function call that could be managed by adding a critical section on top of the two assignments, but more often than not, a function call is the root of a tree of execution that can include all sorts of other function calls with similar issues.

A simple way around this is to add big global locks around stuff, but that effectively serializes the execution, thus defeating most of the benefits of concurrency. You just added a boatload of extra complexity for nothing.

It doesn’t help if you are just reading variables if they can be changed elsewhere. A second writing thread could be half completed during the read, which is the same problem as before.

You can make it all immutable, but you still have to be very concerned with the lifetime of the data. Where it is loaded or deleted from memory can get corrupted too. You can only ignore thread safety when everything is preloaded first, strictly immutable, and never deleted.

Most days, concurrency is not worth the effort. You need it for multi-use code like a web server, but you also want to ensure the scope of any variable is tightly bound to the thread of execution. That is, at the start of the thread, you create everything you need, and it is never shared. Any global you need is strictly immutable and never changes or gets deleted. Then you can ignore it all. Locking is too easy to get wrong.

Some neat language primitives like async seem to offer simple-to-use features, but if you don’t understand their limits, the cost is Heisenbugs. Strange concurrency corruptions that are so low in frequency people confuse them with random hiccups. They might occur once per year, for example, so they are entirely impossible to replicate; they tend to stay around forever and agitate the users.

If you aren’t sure, then serial execution is best. At least for the start. If a language offers features to catch absolutely every possible concurrent issue, that is good too, but one that only catches ‘most’ of them is useless because ‘some’ of them are still out there.

Most concurrent optimizations are only ever micro optimizations. The performance gains do not justify the risks involved. It is always far better to focus on normalizing the data and being frugal with the execution steps, as they can often net huge macro optimizations.

The hard part about concurrency is not learning all of the primitives but rather having a correct mental model of how they all interoperate so that the code is stable.

The Digital Realm

There is the real world. The physical one has been around for as long as humans can remember.

Then there is the digital world, which is an artificially constructed realm based on top of millions, possibly billions or even trillions, of interconnected computers.

Hardware always forms the sub-structure. The foundation. It is what binds the digital realm to reality.

What’s above that is just data and code. Nothing else.

Any other thing that can be imagined in all ways is either data, code, or a combination of the two.

Data is static. It just exists as it is. You can really only change it by writing some other data on top of it, wiping the original copy out of existence.

Code is active. It is a list of instructions, often crazy long, sometimes broken up in countless pieces spread across all soft of places.

Code ‘runs’. Something marches through it, effectively instruction by instruction, executing it, in more or less a deterministic fashion.

Code is data long before it is code. That is because it is a ‘list’ of instructions; when it is not running it is just a list of things. It is data when inactive.

Data can effectively be code. You can declare a whack load of data that is interpreted as ‘high-level’ code to trigger very broad instruction sets.

Data is not just bits and bytes. It is not just single pieces of information encoded in some fashion. Most data only has value if it is used in conjunction with related data. Those groups have structure, whether it is a collection of individual data points or a list of stuff. There are higher level structure relationships too, like dags, trees, graphs, and hypergraphs. Mostly, but not always, the individual parts and their various structures have some names associated with them. Meta-data really. Information about how all the individual points related back to each other. Data about the structure of the underlying data.

In it’s simplest sense, data corresponds to the way we use nouns in language, code corresponds to verbs. We blur the lines for some sophisticated usage, but most forms of programming tend towards keeping them separate and distinct.

We know we need to secure data. It is the heart and soul of the information we are collecting with our computers. That information could be used by malicious people for bad ends. But we also need to secure code. Not just when it is data but also as it executes. As they are distinct, one means of securing them will never cover both; they are, in effect, two different dimensions. Thus, we need two different and distinct security models, each of which covers its underlying resource. They won’t look similar; they can not be blended into one.

Mastery

Oddly, mastering programming is not about being able to spew out massive sets of instructions quickly. It’s about managing cognitive load. Let me explain.

Essentially, we have a finite amount of cognitive horsepower, most of which we tend to spend on our professions, but life can be quite demanding too.

So that’s the hard limit to our ability to work.

If you burn through that memorizing a whack load of convoluted tidbits while coding, it will utilize most of your energy. I often refer to this as ‘friction’.

So, what you need to make the best progress on your work as possible is to reduce as much of that friction as you can. You don’t want to spend your time thinking about ‘little’ things. Instead, you want to use it diving deeply into the substantial problems. The big ones that will keep you from moving forward directly.

In that sense, less is more. Far more.

For example, I’ve known for a long time that a million little bits of functionality is not all that helpful. It’s not a coherent solution.

It’s better to generalize it a little, then pack it all together and encapsulate it into larger ‘lego’ bricks. Now you have less things that you can call, but they do a far wider range of tasks. You still need to supply some type of configuration knowledge to them, but that too can be nicely packaged.

This fits with what I said initially about cognitive load. You don’t have to remember a million little functions. You don’t have to keep rewriting them. Once you have solved a problem, big or small, you package it up and lean on that solution later. So now you can forget about it. It’s done, it's easily findable. You get a bunch of those at a low level, you can build higher level ones on top.

Since you rarely need to work at that previous level now, you have far less to think about. It is a solved problem. As you go, you get higher and higher, the friction is less, and the work you are doing is far more sophisticated.

It was known long ago that good projects get easier to work on with time, bad ones do not. Easier because there is less friction and you can do larger things quickly with better accuracy. You’ve got this codebase that solves the little problems, so you can now work on bigger ones. That’s why we have system libraries for languages, for instance.

If you’ve got some good pieces and someone asks for something complicated, it is not that hard to extend it. But it is really hard to go right back to ground zero and do it all over again. You brain has to cope with lots of levels now, it will get overwhelmed. But if you can skip over all of the things already solved, then you can focus on the good stuff. The new stuff. The really hard stuff.

In a similar way, disorganization and inconsistencies eat through massive amounts of cognitive load. It is inordinately harder to work in a messy environment than a neat and tidy one. If all your tools are neatly lined up and ready when you need them, then jumping around is fluid. If you have to struggle through a mess just to find something, it bogs you down. Struggling through the mess is what you're doing, not solving the problems you need to solve.

So you learn that the dev shop and its environment needs to be kept as clean as time allows. And tidying up your working environment is almost always worth the time. Not because you are a neat freak, but because of how the friction will tire you out.

If you manage your cognitive load really well then you don’t have to spend it on friction. You can spend it on valuable things like understanding how the tech below you really works. Or what solutions will really help people with their problems.

The less time you spend on things like bizarre names, strange files, and cryptic configurations the more you have to spend on these deeper things, which helps you see straighter and more accurate paths to better solutions. In that sense the ‘works on my machine’ excuse by someone who exhausted themselves drowning in a mess is really just a symptom of their losing control over the tornado of complexity that surrounds them.