Reinventing the Wheel

An all too common phrase in programming is “Don’t reinvent the wheel”. However, I believe that the phrase is often used incorrectly.

First, ‘the wheel’ is a theoretical concept. It is the notion that if something is ‘round’, it will ‘roll’, and you can put that under other stuff in order to move it forward. A bike tire or car tire are wheels, but creating a new type of tire is not ‘reinventing the wheel’; it is actually just reimplementing one. You haven’t reinvented it unless you replace ‘round’ with something else, like a hexagon or tank track.

In software, that means that if you write your own persistence solution so that you can avoid using an RDBMS, you have not reinvented persistence; you just went your own way for the implementation.

On the web, we do occasionally see really neat alternatives for ‘wheels’. Mostly, though our vehicles and furniture stick with the familiar old round things that have been around for millennia.

The reason is simple. The concept of a wheel works really well. It is state-of-the-art for moving things. In a specific case, you may need to implement your own variation of it; maybe air in a rubber layer is inappropriate, but rarely do you need to revisit the theory. It is fine.

It is also good that other people are exploring alternatives, as some of them seem quite useful, but if you are asked to engineer something that moves right now, the theory of a wheel is probably the best place to start. Then, maybe, you can leverage an existing implementation like a castor to get yourself going.

All of this is the same for software. You are asked to put together a big system or product, so you break it down into small parts. If one of those parts is ‘persistence’, you first look at an RDBMS or NoSQL. You evaluate the properties of both and likely choose an implementation that honestly best fits your requirements.

You would not commit to doing your own implementation unless there is a particularly nasty requirement that is absolutely not met by any of the existing implementations.

But if you did decide to go your own way, first you would acquire the full knowledge of all of the theories, the state-of-the-art for persistence. You’d know it all, from the distributed fault tolerance perspectives all of the way over to data normalization.

You can’t skip parts because you’d have to reinvent them, and since most of them are decades old, you’d be falling right back to ground zero. You don’t want some piece to be crude if there is already an implementation out there that is not crude.

Given the requirement to know the state of the art, it is pretty much not an option for most people to write their own persistent solution. They don’t have the knowledge to get their implementation up to the state-of-the-art, and going far lower isn’t going to net them the benefits that they need. This is the core of “don’t reinvent the wheel”, which is that you shouldn’t, unless you really can. And you really can when you know a crazy amount of stuff and also have a crazy amount of time to get it implemented; both are rare circumstances. So, don't do it.

But while persistence is a very deep and broad knowledge base, people frequently make the mistake of applying that logic to much simpler issues.

You could, for instance, reinvent or reimplement your own widget. Reinvention is trickier as the widget will be perceived as eclectic for most users, so they probably won’t like it, but there are times when a brand new type of widget is actually what is needed. Rare, but possible. As for reimplementing, if you fix enough of the annoyances with most modern widget sets in order to lift up their quality, then it is probably very worthwhile. Or maybe you just wrap all of the other widgets to fix their inconsistencies, another variation on reimplementation that often makes sense.

This applies to all sorts of places in the tech stacks where the current implementations are weak or just glue over something else. You probably don’t need to reinvent them, but it may make your life considerably more comfortable to reimplement them, or at least wrap them up nicely. These two types of coding tasks are really good but, unfortunately, are often confused with reinvention. You’re not reinventing the wheel; you are just applying your knowledge to get a better implementation somehow. Of course, to do this successfully, you need that knowledge first. Without that, you are just going backward.

Herding Cats

The idiom about “herding cats” is frequently used in IT and plays out in many different places in the industry.

It adequately describes programmers. Because the industry keeps expanding far faster than people can follow, the culture tends to overcompensate. Programmers are frequently crazy overconfident in their beliefs, views, and opinions, as it oddly tends to help in their careers. But it also means a high rate of failure and a lot of personality clashes.

I always figured that if we got the 10 greatest programmers on the planet all together on the same project, instead of producing magic, they would just dissolve into bickering with each other. The cats all want to go off in their own direction.

That tendency plays out in technology as well.

Microservices was a super popular idea. Instead of everyone coming together to build something, it offers the dream that everyone could just build little fully independent pieces that would all magically work together. So, you don’t have to compromise for the team, you just do what you know is right, toss it into the mix, and it will be good.

But of course, it doesn’t work that way. If at the binding level, the sum of the services is hugely disorganized and irrational, the whole thing just isn’t going to be useful or stable.

The health of the ‘forest’, dominates the health of the ‘trees’. Each cat tending to their own little tree without having to worry about the other trees only works if all of the trees are contained and arranged nicely. If the “tree” however, is really various branches from a lot of other trees scattered all over the forest, it's nuts. The parts need to be perfectly independent and totally aligned, which is a contradiction.

It shows up in products too.

Most, if not all, big products were evolutionary. A small group of people got it going, and then it was scaled way up over a very long time to reach its current level of maturity. So, it’s not uncommon that hundreds or even thousands of programmers were involved over decades.

That initial small group likely had a strong tight focus, which is why the product didn’t fail. But as people come and go that focus gets watered down, realigned, and bounced all over the place. That shows up in the product. It goes from being a tight nicely arranged set of functionality to being a pile of mistaching features, erratically shoved into strange corners of the interface.

Before maturity, there wasn’t as much functionality, but you could always find it. After maturity, everything including the kitchen sink is there, but now it is tricky or impossible to find, and behaves inconsistently with everything else.

We see it with in-house software as well. A friend of mine used to love to say that companies get the software they deserve. If you look at most big non-software companies, their IT depts are a whacking great mess of chaos, bugs, and defective code.

Since personality issues tend to be the downfall of large internal projects, most of the stuff is little bits of glue haphazardly applied to everything.

It isn’t even a ball of mud, but rather a gaint field of wreckage. And the software industry loves this. They endlessly keep selling magic bullets to fix the ever-growing disaster, but generally just make it worse somehow. Generation after generation of technology that makes bold claims and then disappoints.

In so long as programmers want the freedom to go off and do their own thing, ignoring what they don’t want to do or know, then the results of their work have strictly limited usefulness.

If it is fully eclectic, it fits with nothing. Yes, it is a creative work, but that creativity is what keeps it from being useful.

If it is fully standard, to whatever the larger context is, there is almost no freedom whatsoever. You figure out the standards, implement them correctly, and then put state-of-the-art pieces below. Almost no creativity, but a massive amount of prior knowledge. The thing is great and works with everything else, but it is as it should be, not as you’d like it to be.

The work is figuring out what it should be. This is why the programmers who can successfully work with the greatest programmers on the planet have more value than their peers. They are humble and know that creativity isn’t necessary. Research is a better use of their time. Get as close to standard as you can in the time you are allowed.

The broader theme is that ‘focus’ is a huge part of quality. If the work is large and unfocused, there will be plenty of problems with it and if there are too many of them it undoes any or all of the value of the work itself. If, however, the work is tightly focused then most of the contributors must orient their work in that direction, which takes a lot of the fun out of it. More focus means less freedom, but also better results. There does not seem to be any way around this.

Development Automation

One popular programming fallacy is that it is not worth spending five hours writing a script that runs in a few minutes.

Mostly, this perspective seems to come from people who don’t like scripting. They want to have just one primary coding language; they only want to stay working in that language all of the time. Scripting usually requires at least one other language and lots of little tools. It is different from the other coding tasks. It’s more stuff to learn.

This perspective is wrong for a bunch of reasons.

First, if you spend five hours writing a script and then run it thousands of times, it obviously pays off. It still pays off even if you run it hundreds of times, and probably even if you run it only a dozen.

But if you’ve scripted all sorts of stuff, even if a few of those scripts don’t get run very often, the more you script, the faster you will get at doing it. Practice makes perfect, but it also makes it faster.

If you’ve mostly scripted everything, then creating new variations on other scripts is nearly trivial. You have lots of examples to work off of. You don’t want a billion brute force scripts, but most of the things you are automating are essentially similar. Sometimes you refactor to expand scope (preferred), but other times you just copy and modify to get it going.

And whenever you run the script, the results will be deterministic. A good script will run the same every time. You have at least semi-automated the tasks if not better. That’s what we do for users; why not for ourselves?

If, instead of scripting, you need to click on 10 buttons each time, the chances are that you’ll miss some steps. If you need to type 10 long CLI commands, it is even worse. More chances to forget stuff.

The cost of getting it wrong is always way more expensive than the automation costs. You do a PROD install and forget a tiny piece; it isn’t just the time you have blown, it is also the loss of trust from the people around you. Oddly, that often results in more meetings and more tracking. A history of screwups is very costly.

And, of course, documentation. If someone new starts, you point them to the scripts. If you step away for a while, you point yourself to them. They are a record of the things you are doing every day. You can’t really share them with non-technical people, but that is fine, even better.

Finally, encapsulation. If you bury the ugly grunge into a script, you can forget about it. Pretend it never happened. You’ve stripped off some small annoying piece of complexity and replaced it with a simple command. If you keep that up, there is far less friction and far more accuracy, which gives you far more time to concentrate on the newer, bigger, far more interesting problems.

Scripting everything, or at least as much as you can, is and has always been a super strong habit. Part of mastering coding.

Ironically, while the main languages have changed radically over the decades, scripting has not. You can script well with early 90s techniques, and it will be as useful today as it was then.

Concurrency

For large complex code, particularly when it is highly configurable, polymorphic, or has dependency injection, it is sometimes difficult to get a clear and correct mental model of how it actually runs. The code might bounce around in unexpected ways. Complicated flow control is the source of a lot of bugs.

Now take that, but interlace multiple instances of it, each interacting with the same underlying variables, and know that it can get far worse.

The problem isn’t the instructions; it is any and all of the variables that they touch.

If there is a simple piece of code that just sets two variables, it is possible that one instance sets the first one, but then is leapfrogged by another instance setting both. After the first instance finally sets the last variable, the state is corrupt. The value of the variables is a mix between the two concurrent processes. If the intent of the instructions was to make sure both are consistent, it didn’t work.

If it was just an issue with a single function call that could be managed by adding a critical section on top of the two assignments, but more often than not, a function call is the root of a tree of execution that can include all sorts of other function calls with similar issues.

A simple way around this is to add big global locks around stuff, but that effectively serializes the execution, thus defeating most of the benefits of concurrency. You just added a boatload of extra complexity for nothing.

It doesn’t help if you are just reading variables if they can be changed elsewhere. A second writing thread could be half completed during the read, which is the same problem as before.

You can make it all immutable, but you still have to be very concerned with the lifetime of the data. Where it is loaded or deleted from memory can get corrupted too. You can only ignore thread safety when everything is preloaded first, strictly immutable, and never deleted.

Most days, concurrency is not worth the effort. You need it for multi-use code like a web server, but you also want to ensure the scope of any variable is tightly bound to the thread of execution. That is, at the start of the thread, you create everything you need, and it is never shared. Any global you need is strictly immutable and never changes or gets deleted. Then you can ignore it all. Locking is too easy to get wrong.

Some neat language primitives like async seem to offer simple-to-use features, but if you don’t understand their limits, the cost is Heisenbugs. Strange concurrency corruptions that are so low in frequency people confuse them with random hiccups. They might occur once per year, for example, so they are entirely impossible to replicate; they tend to stay around forever and agitate the users.

If you aren’t sure, then serial execution is best. At least for the start. If a language offers features to catch absolutely every possible concurrent issue, that is good too, but one that only catches ‘most’ of them is useless because ‘some’ of them are still out there.

Most concurrent optimizations are only ever micro optimizations. The performance gains do not justify the risks involved. It is always far better to focus on normalizing the data and being frugal with the execution steps, as they can often net huge macro optimizations.

The hard part about concurrency is not learning all of the primitives but rather having a correct mental model of how they all interoperate so that the code is stable.

The Digital Realm

There is the real world. The physical one has been around for as long as humans can remember.

Then there is the digital world, which is an artificially constructed realm based on top of millions, possibly billions or even trillions, of interconnected computers.

Hardware always forms the sub-structure. The foundation. It is what binds the digital realm to reality.

What’s above that is just data and code. Nothing else.

Any other thing that can be imagined in all ways is either data, code, or a combination of the two.

Data is static. It just exists as it is. You can really only change it by writing some other data on top of it, wiping the original copy out of existence.

Code is active. It is a list of instructions, often crazy long, sometimes broken up in countless pieces spread across all soft of places.

Code ‘runs’. Something marches through it, effectively instruction by instruction, executing it, in more or less a deterministic fashion.

Code is data long before it is code. That is because it is a ‘list’ of instructions; when it is not running it is just a list of things. It is data when inactive.

Data can effectively be code. You can declare a whack load of data that is interpreted as ‘high-level’ code to trigger very broad instruction sets.

Data is not just bits and bytes. It is not just single pieces of information encoded in some fashion. Most data only has value if it is used in conjunction with related data. Those groups have structure, whether it is a collection of individual data points or a list of stuff. There are higher level structure relationships too, like dags, trees, graphs, and hypergraphs. Mostly, but not always, the individual parts and their various structures have some names associated with them. Meta-data really. Information about how all the individual points related back to each other. Data about the structure of the underlying data.

In it’s simplest sense, data corresponds to the way we use nouns in language, code corresponds to verbs. We blur the lines for some sophisticated usage, but most forms of programming tend towards keeping them separate and distinct.

We know we need to secure data. It is the heart and soul of the information we are collecting with our computers. That information could be used by malicious people for bad ends. But we also need to secure code. Not just when it is data but also as it executes. As they are distinct, one means of securing them will never cover both; they are, in effect, two different dimensions. Thus, we need two different and distinct security models, each of which covers its underlying resource. They won’t look similar; they can not be blended into one.