Phone vs Cable

There are only two basic paradigms for communicating between things (threads, processes, containers, machines, hardware) in software.

The first is based on explicit requests, basically like a telephone. One computer phones up another computer, they talk, then the line is closed.

The other is broadcast based like cable TV. A source computer sends out a signal over a wire, everyone else just listens to the different channels.

Request-based communication is very specific, but also expensive. That is, you can request a piece of data, but both the server and client have to be actively involved. If there is a small number of clients this works quite well, but it usually fails to scale.

Broadcast-based communication is very general, but also relatively cheap. One server can stream out tonnes of different channels. If it’s capable, it can do it steadily.

Request-based systems degrade under a heavy load. If too many people connect, they tend to either slow down for everyone or drop connections.

Broadcast systems are independent of load. If they are set up to handle N channels, then they can consistently do that going forward. As long as the server can handle sending out M channels you can go from N to M.

If you need to ensure that the two machines actually reliably communicated something to each other then requests are far better. The client can ensure that it actually gets the data, or that it will continually retry until it is retrieved. The server knows exactly what it got and from which client.

For broadcast, the server often doesn’t even know who the clients are, how many of them are out there, or what they have received. For a client, if it misses part of the stream, and there is no historic cache of the older broadcasts, then it will not be aware of the data.

Over the decades, there have been lots of technologies that try to combine both paradigms. They might receive a broadcast, but then be able to initiate a connection as well. Or it is request based but receives a stream of something like events or updates later. While that gives the best of both worlds, it does still open up the servers to performance problems, just that they are less frequent.

There is a slight difference in error handling. For requests, the client needs to keep retrying until it is successful. For broadcasts, the client needs to be aware of its own uptime, and do extra processing in order to fill in any gaps that occurred.

Phrased as “phone vs cable” it is pretty easy to remember which basic properties are tied to which paradigms. Sometimes it is obvious which one you need, but sometimes the trade-offs are difficult. Usually, the difficulties are mixed with issues about integrity, frequency, and correctness.

Helpers or Encapsulation

I was having a discussion with another developer recently. He suggested that we could create a library of ‘helpers’.

I like creating libraries for most things, but this suggestion turned me off. It just sounded wrong.

A while ago I was working on a medium-sized system where the developers basically went crazy with ‘helpers’. They had them everywhere, for everything.

Generally, if you take any ideal in programming and apply it in an over-the-top extreme manner it doesn’t work out very well, and that was no exception.

It basically destroyed the readability, and the code in the helpers was haphazard, all over the place. So without bouncing into lots of different files and directories, you wouldn’t get any sense of what the code was actually trying to do. And far worse, that fragmentation was hiding some pretty significant bugs, so really it was a huge mess.

But breaking things down is one of the core ideas of good software development, so why did the helper-fest go so horribly wrong?

You want to build up a lot of reusable pieces, gradually moving up from low-level operations into higher-level domain primitives. You want this so that when you go to build new stuff, you can most often do it from a trusted set of existing lower stuff. You’ve built a lot of the parts already, you’ve tested them, and they are being heavily used. So, if you leverage your earlier work you’ll save yourself massive amounts of time.

But you also need to encapsulate any of the complexity. If you don’t it will grow out of control.

Basically, you want to stick something complex into a box, hide it away, and then keep reusing it. Once you have solved a specific coding problem, you don’t want to waste time by having to solve it again.

Encapsulation is always more than just code. There are some data, constants, or config parameters of some type that goes inside the box as well. You might even have a bit of state in there too, but you have to be very careful with that. As long as some of the underlying details never leave the box, you’ve at least encapsulated something.

So, a good reusable lower component is encapsulated code, data, and the mechanics needed to do some functionality, that is nicely wrapped and hidden away from the caller. You see this commonly in language libraries, string handling is often a good example. You don’t get caught in messing with an array of individual characters, instead, you do common operations directly on the abstraction notion of a ‘string’.

Basically, you’ve removed a piece of complexity and replaced that with a box that adds only a little bit of extra complexity. Overall, complexity is higher, but at any given level you have lowered it.

As long as you name it correctly, it is reusable. Later someone can find it, use it, and trust it, without having to get lost in those underlying details.

The way most people write helpers though, they are just pure code fragments wrapped with a function. Really just some idiom or a clump of code that people are using often. It’s more like a cut and paste, but without making explicit copies.

So that’s where helpers get into trouble. Basically, they’re often disjoint code fragments, arbitrarily chosen, and both because they do not encapsulate, and because they are not well-defined primitives, you could reuse the code but the overall complexity is still going up. It doesn’t hide things, really it just reduces retyping. You gain a little bit from calling it, but you lose a lot more from its inherent complexity, especially if you need to read it later.

And that’s pretty much what I saw in that earlier project. Readability was shattered because the lines drawn around the helper code fragments were arbitrary. You can’t even guess what the calls will do.

In that earlier case though, a lot of the helpers also modified globals, so the side effects were scary and rather unpredictable. But even if that wasn’t true, the general ideas around helpers may help reduce pasted code, but do not encapsulate any complexity, so they really aren’t helping much.

A variation on this theme is to have a large number of static functions in an OO system. It’s the same problem just not explicitly called out as helpers.

Probably the worst thing you can do with code in a large system is duplicate stuff all over the place. It’s a recipe for creating bugs, not functionality. But avoiding encapsulation by using a weaker approach like helpers isn’t really better. There is never enough time to do a proper job coding anymore, so you have to make any work you do count. The more time you save, the more time you now have to do a better job.

Reinvention

Programmers are sometimes confused between reinventing a wheel and just simply rewriting it.

Let me explain…

Underneath all code is a model of operation that is often based on the interaction of sophisticated data structures. That is the core of the abstractions that drive the behavior. That is what makes the code work.

The more sophisticated the model, the more likely it works better (not always true, but more often true than not).

If you learn or read about those abstractions, then you can use that knowledge to faithfully write new implementations. That is just a rewrite. You’re not trying to invent something new, but are just leveraging all of the existing knowledge out there that you can get your hands on.

If you ignore what was done in the past, and essentially go right back to first principles, that is a reinvention. You will try to come up with your own sophisticated model that is in direct competition with what is already out there. If that existing work is already decades old, you are essentially going back in time to an earlier starting point, and you will have to find your own way from there, through all of the same problems again.

In simple cases that is not a big problem. You can craft a new GUI that looks somewhat like the older ones. They are fairly routine and usually conceptually simple.

In advanced cases though, like persistent storage, frameworks, languages, or operating systems it is a major leap backward. So much was done, discovered, and leveraged by a massive number of people, but you are just ignoring everything. It’s unlikely that by yourself, you have anything close to a tiny fraction of the time needed to relearn all that was already known. What you create will just be crude.

So we really don’t want to reinvent stuff, but that is quite different from just writing our own version of existing stuff. The key difference is that long before you start a rewrite, you had better spend a great deal of effort to acquire the state of the art first. If you are just blindly coding something that seems okay, you’re doing it wrong.

Given that distinction, it gets interesting when you apply it to smaller dependencies.

If there are some simple ones that you could have written yourself, then it is way better to write them yourself. Why? You’re not reinventing the wheel, you already have the knowledge and it is always better to directly control any code you have to put into production. If you wrote it, you understand it better, you can fix it faster, and it is likely more future-proof as well.

If there are complicated dependencies, with lots of mature features, then unless you obtain the precise knowledge of how they work, you are probably better off relying on them. But, as they are already complex technology, you still have to spend a lot of time to understand them, and you should leverage them to their fullest extent, instead of using subsets of a bunch of similar libraries.

In between, it likely depends on your experiences, the domain, and both the short-term and long-term goals of the organization. Having your own code is “value”, and building up a lot of value is usually super important, both for in-house and commercial projects. A ball of glue might be quick to build, but it is always an unstable nightmare and is easily replaced by the next ball of glue.

Software Factories

I am uncomfortable with the factory model of a software development shop.

I heard it from a seasoned tech exec decades ago. Effectively, the dev shop is a factory that takes in requirements and produces releases. Basically, it’s a bunch of software assembly lines.

The obvious problem is that it indirectly implies that programming is similar to blue-collar work. That you get a bunch of programmers together, tell them what to do, they do it, and then the results get shipped. It is sort of treating it as if it were all physical, not intellectual.

While that offends the idealist in me, it is pretty much the case now that most software development is routine. Libraries and frameworks deal with the hard parts, most coding is just gluing it all together to meet a specific goal. If you read the tutorials and basic documentation, any sort of missing answers can be found on sites like StackOverflow. You can learn the basic skills fairly quickly, and in most cases just keep applying them over and over again.

There is still a ‘rapid comprehension’ issue. That is, you might have found the answers, but you still need to understand them quickly. Active programmers are usually continuously learning stuff. It’s like being in school, and it is almost never-ending. Maybe if you stayed on the same codebase for over a decade there isn’t much left to learn, but for most coding positions, there is a vast unknown.

One of the tweaks to the factory model is that analysis is inside the factory, not outside of it. The requirements come in the front door, but the analysts venture out from the factory to investigate and flesh out the issues. They could be specialized roles like Business Analysts, but often they include all people heavily involved in the design like architects and development leads as well. You can’t really build a good solution if you don’t understand the problem.

The other major issue with the model is scaling. Big companies with lots of IT needs, tend to behave like a lot of little independent factories. This means they are each redoing the same fundamental work, over and over again. Besides being highly inefficient, it also leads to severe quality issues. The organization never gets competent at producing software, even if a few of its factories do. But they tend to disperse, taking all of that knowledge with them.

It would also be better to have some reliable horizontal factories at different levels that support the vertical factories for business needs. That is, you get groups of developers that write backend technical stuff, that is mandated to be used by the other developers. Obviously, enforcement is a huge problem, in that the quality of work for the lower-level developers has to be extremely high, or it negates the usefulness of the effort.

One of the good things about the factory model is that it clearly delineates the boundaries of development. That is, the factory produces updates and versions and stuff, but it is isolated from the act of ‘operating’ this stuff. It sort of implies that a well-run dev shop is rather quiet and focused on its work, even if the business itself is rambunctious and full of stress. Obviously, that would be far better for quality. You can’t code very well if you are constantly being interrupted. You don’t need perfect code, but the side effects of running bad code can be hugely expensive.

And that split between development and operations is super important. They are different mindsets, so pushing the devs heavily into operations issues just tends to just burn them out. They should be focused on engineering things to work correctly, and while they should obviously get a lot of feedback to ensure everything works as expected, they don’t need to be front and center in the ongoing operational issues. Rather they should complete their work, then ship it to others who will put it into production, monitor it, and deal with any unexpected behaviors.

So, while the factory model bothers me at times, there are aspects of it that I wish more people understood and paid attention to. You often see these development shops where everything is hopelessly blurred, people are stressed and confused, and the work is going very badly. We need to find better ways to structure the work so that it doesn’t cost so much and take a crazy amount of time to get done. We seemed to be getting worse at building software, not better.