Et cetera

Recently, I started queuing up posts for Thursdays at 9pm. Not really sure why, just seems like a good time to publish.

I figured there would be near zero readership over the holidays, people have much better things to do. So, initially, I was going to skip today, but I guess I’ll just ramble a little bit about the future anyways.

We’re at a strange juxtaposition in our history, where our overall trajectory doesn’t look great, but there are still a few things “in the wind” that could be epic game changers.

The biggest is that fusion is looking promising again. Unlimited clean energy would massively disrupt our societies. Lots of science fiction books out there explain why, but the shortest answer is that shared, expensive, infrastructure forces us all together. Break that, and give everybody dependable robots, and we’ll disperse everywhere. Maybe not so great for the planet, but it would definitely make it harder for people to manipulate and exploit many of us.

Another great pending innovation is self-driving vehicles. Forget cars, we could build mobile houses, which would be amazing. It’s like Howl's Moving Castle, but better. Your whole house is a mobile set of rooms that reconnect at different times, in different places. You could share parts of your home with your family or friends. Everything would be constantly mobile, particularly if cheap energy wasn’t a constraint.

In general, robotics are radically improving. They are going to cause massive class upheavals in our societies. Machines caused some trouble long ago, making it harder in some parts of the world for people to find good jobs that provide lifetime stability. Robots will be far more transformative. If they wipe out every sort of blue-collar occupation, there will be a great swath of people who are left without any options. Enough people that it could be a real problem.

But robots also have the potential to do great things. I could see massive salvage yards where millions of robots, some huge, others tiny, take all of our trash and gradually recycle it back into its raw elements. Decreasing-sized robots that separate and chop up stuff, until it has become so small that it can provide near purity. Swarms of little nanorobots partition stuff into different particles. Trash farming. We can’t do this today, the electricity cost is massive, but if that suddenly fell …

Some of the growing ramifications of AI are looking interesting as well. While it still seems to be a little rigid right now, a bunch of these new technologies are also pretty darn amazing. I’m just hoping that they will help lift us up and not just enable the worst exploits. Too many people want to control others for the purpose of extracting wealth; it’s pointless, but they persist. We don’t need more billionaires, instead, we really need to come together to fix our mistakes. To accomplish this we need to be collectively smarter, so maybe AI will actually support that somehow.

As for software, it’s probably time to finally get really serious about it. AI may wipe out programming, but if it doesn’t we really need to do a much better job of engineering these massive systems that rule our lives. As well, we should be actively preventing unscrupulous people from using software for their own malicious purposes. We can’t just keep on quickly hacking out stuff and then walking away. It’s just contributing to the instability of the world.

I guess that’s it. The future world could be really comfortable, smart, and even luxurious, or it may be a horrific police state where everything you do is monitored. Either way, it is software and energy that helps enable these futures.

Split Level

[opps, I had posted this early this month under a different name. Now it’s redundant like so much of our code. ]

The ‘problem’ space is the physical/digital issues and data that are facing a group of users for a given domain.

The ‘solution’ space is the set of all functionality embedded into a software system that can be used to address some of the issues in that problem space.

Keeping those two strictly separate from each other makes it far easier to design complex systems.

A ‘feature’ is something the users need to deal with one or more of their issues in the problem space. That feature then maps over to one or more parts of 'functionality' in the solution space.

It’s a bad idea to let any solution terminology infect the description of the problem specifications. It is usually a good idea to push a lot of the problem terminology into the solution design.

That is, a common feature in a lot of systems is the ability to manage a particular category of data. Which is often mapped to ‘new’, ‘edit’, and ‘delete’ GUI screens in the software. It would be a mistake to call “editing” that data a feature. It is not a feature, it is just one of the parts of the functionality needed to address “managing” that data. It would be good to be more precise about the naming of that data. So, it’s asymmetrical.

Now it may seem really pedantic to discuss it this way, but there are deep seeded reasons why it is better.

An obvious one is that just implementing part of the functionality is not actually implementing the whole feature. So, we see a lot of systems out there with ‘incomplete’ features and they are extremely frustrating to use. Partially solving any problems often just makes those problems worse.

Another good reason to do this is that the terminology and necessity in the problem space does not always map nicely into the solution space. In fact, usually, it is kind of horribly denormalized, if not entirely irrational. People want to work in a way that is comfortable for them, which is usually far more difficult to implement. Because of this, we often stumble into the problem of whether a ‘simplification’ of any kind is just effective for the programmer or whether it really benefits the user. It is rarely both. It’s a classic trade-off.

If you do some ‘analysis’, that should be purely in the problem space. If you do some ‘design’, then it quite obviously needs to be in the solution space. If they are kept distinct, then we have a way of not blurring the lines and injecting noise into the outputs. The users need to keep track of something happening in their business. That is their feature. If that actuates into a portal screen or a report or some other construction, that is part of the design of the solution.

One place that is super common to see a mess is in classic requirements. They are often a haphazard mix between features, implementations, designs, conventions, abstractions, etc. Pretty much anything from either space gets brutally mixed and matched and combined into a mess. This is the key reason why they don’t get cross-checked properly and validated early on. This leads to confusion, ambiguities, and tonnes of scope creep. It’s a bigger reason why a lot of systems derail than methodology. If the requirements are convoluted, then any attempts to clean them up without splitting out the spaces will fail too, and sometimes just make the confusion worse.

If you had a clean list of features and another clean list of functions, then you could just mechanically go through all of them and see which functions address which features. You can see if the feature is fully or partially addressed. You can see useless functions. You would also see which features are not addressed at all. That would let you double-check that the solution design will actually help to ensure it is a good fit.

A big problem with modern software development is that the definitions we use in our industry are vague, blurry, convoluted, and often deliberately scrambled.

Determinism

When you go to release a piece of software, you really, really, really want to know what that software will do in the wild. That is a big part of the mastery of programming.

People often ask for code that does very specific things. You write exactly that code. Each and every time it runs it will correctly do what is asked. Its behavior in ‘all’ circumstances is what you expect it to be.

There will be bugs, of course.

Depending on your skill level, there is a relatively fixed amount of targeted testing that can achieve any particular desired quality level.

For instance, if you are a strong programmer and you need near-perfect quality, depending on the size of the codebase, the testing might only require a few months or a couple of quarters of intensive effort to confirm that it always works correctly.

Aside from bugs, there are plenty of other things in code that can affect whether or not its execution will match your expectations. We’ll use the word ‘determinism ‘to describe these.

Code is deterministic if it only ever does what you expect. It is non-deterministic if its behavior is unpredictable; you don’t know how it works or what it will do when it runs.

A fully deterministic piece of code takes some inputs, applies a reasonable computation on them, and then produces some outputs. This happens each and every time you run the code. It seems simple enough.

So, the first point in making code deterministic, is that you understand and get control of all of the inputs.

We learned over fifty years ago that global variables were a bad idea. If you look at a function that relies on a global variable then not only do you have to worry about the inputs, but you also have to have some deep understanding of all of the possible states of that global. If it’s a signed integer, for instance, it could be any value between the minimum and maximum integer values.

For one calculation that doesn’t seem that bad, but as the code is following an execution tree for a bunch of functions, you can’t assume that the integer hasn’t changed for some reason. It may have been zero at the start, but somewhere else deep down in the stack it flipped to 42. You don’t know if that is possible, or if it happened just by looking at a small number of those functions. You’d have to look at everything that touches that global, and figure out if it ever got triggered. It is bad for single-threaded processes, it is far worse for multi-threaded code.

This is to say that referencing a global variable in your function adds some non-determinism to the function itself. Maybe a little, maybe a lot. Now, you aren’t entirely sure what to expect anymore when it executes. It’s not a massive problem for a single integer, but for each such occurrence, it only gets worse.

So any globals, whether they are primitive values, objects, singletons, etc. start to add non-determinism to each and every bit of code that utilizes them. They infect the code.

Oddly the fix isn’t hard.

We just make sure that every input in any function is declared as part of the function parameters. Then we know, way at the top, that all of the underlying functions are working on the same instance of that global and that they are referencing the same value, each and every time.

In that sense, “except for the top”, using globally accessible ‘variables’ is degrading the determinism of the code. Constants and immutable globals are okay, it’s just the stuff that varies that is dangerous.

We can further explore this by understanding that there is always a greater context playing out for our code. We can refer to that ‘execution context’ as ‘state’. If the only thing in that state is fixed data, then there is only 1 overall global state, and everything below that is deterministic. If there are N variables, each of which has M1, M2, ... Mq possible values, then the number of possible states is the M1*M2*...*Mq permutations, which is a crazy large number. Too many for us to imagine, too many for us to craft correct expectations about the execution.

But programmers use globals all of the time because it is easier. You just declare stuff once in some file, then keep referencing it everywhere. Oddly, shoving those globals into the function args is only slightly more work, but people still love to cut this corner aggressively.

Along with globals, a long time ago we realized that using ‘goto’ statements caused similar problems. They allow the flow of the code to just jump out to strange and unpredictable places. It’s the same problem with flow, that globals are to state. You can’t look at a function and produce correct expectations; you can not fully “determine” what that code is going to do without a lot of jumping around, searching, and contemplating other parts of the codebase. The flow of control itself is non-deterministic.

It’s not that the goto operations are all bad, after all, they did return to languages in limited circumstances like setjmp/longjmp and later in try/catch statements. Just that reckless usage cranks up the non-determinism, which makes it increasingly impossible to figure out what this stuff does in the wild.

The try/catch idiom for example does sometimes lightly re-introduce the goto problem. Any usage tends to split the outbound flow from the code. If there is a stack of code, each of which is doing one or more try/catch handling, then each one adds a *2 possible branch to the way the code will execute. A couple of *2 branches are easy to understand, but throwing lots of them everywhere all over the execution stack will explode the permutations beyond anyone’s ability to imagine the behavior, and thus make everything non-deterministic again.

That’s why the best practice is to always restrict try/catch handling to be only right at the bottom or right at the top. Put it in the bottom if you can do something meaningful right there, otherwise, let it go right up to the top and get dealt with just once. If you use try/catch diligently it cuts down on your own flow mechanics, but if you don’t apply any discipline the whole thing becomes increasingly non-deterministic.

As well as globals and gotos, non-determinism can be introduced at the computational and architectural levels too. The problem space itself might hold some non-deterministic or ambiguous elements, which can’t be tackled by brute force. Examples of these types of problems are quite complex, but in general, we tend to capture them as issues like transactional integrity, the dirty write problem, CAP, consensus, etc. They occur because the code cannot ever statically make the right choice, so, you need to wrap the theoretical aspects in some deep complexity to either make it deterministic or at least to make it as nearly deterministic as possible.

Deliberately ensuring determinism in your code is part of mastering coding. Making it readable, future-proof, and accurate work estimations are also necessary. Throwing together some code that kinda works is easy. Throwing together a lot of code that always works is difficult.

Definitions

Communication is vital for succeeding in any activity that requires the participation of more than one individual.

Everyone has to be on the same page, they have to coordinate their actions.

All forms of communication, including verbal and written, are essentially compressed data. That is, we use higher-level terminology to quickly discuss complex topics. Underneath, those terms rely on everyone expanding their own definitions.

Definitions have a denotation or connotation, essentially a meaning with a specific set of boundaries. But they also have colloquial meanings, which is a further set of attached understandings than can depend on context, territory, or culture.

Most definitions are victims of history. They’ve been banged up by past events. Once they have been bent out of shape, they start detracting from the communication. Because of this, most professional fields use their own strict definitions, which can be quite different from the common ones. It ensures that any communication is precise.

So for coordination, in the very best case, an entire group of people already know and understand the definitions of the terms that they are using to communicate. Then the information is passed around correctly.

Often at the heart of many conflicts is a misunderstanding of one or more of the underlying definitions. Two people will argue with each other because their definitions are not aligned. Clearing up those definitions will usually dissipate the issues.

Besides accidental disagreements, we also see people purposely mess with definitions. They stretch them in order to twist out their “logic” to press some personal agenda. It is a popular tactic in politics. It is fueled by intentionally vague definitions that are constantly in motion. By now many of these definitions are entirely nonsensical.

A leading indicator of any questionable argument is someone initially starting it by changing some of the known definitions. They may do this to clarify the terminology, which can be good, but they also may be trying to make it even cloudier, which is definitely not. It is difficult to discern between those two situations.

Usually, the difference is that people who want clarity things will tighten their definitions, so they’ll reduce the scope or boundaries. People who are attempting to misuse definitions will shift them around or weaken the definitions to include more stuff. They get less precise.

With a set of broken definitions, you can reconnect them together in any way you want to pretend to prove basically anything. If people don’t pick up the deception, they can be left believing that some falsehood is true. So, it’s a powerful weapon wielded by unsavory people.

This is why it is so important to be tight and consistent with definitions and to fully understand them. It’s why most introductory subject courses are basically just endless definitions. Definitions are the weak link in our ability to communicate. Bad ones can masquerade as primitives in “logic” but given they are broken, any derived conclusions are meaningless. To communicate or even extend our knowledge we need a strong base of definitions. It’s not pedantic, it is precise; there is a difference.

Problem and Solution Spaces

The ‘problem’ space is the physical/digital issues and data that are facing a group of users for a given domain.

The ‘solution’ space is the set of all functionality embedded into a software system that can be used to address some of the issues in that problem space.

Keeping those two strictly separate from each other makes it far easier to design complex systems.

A ‘feature’ is something the users need to deal with one or more of their issues in the problem space. That feature then maps over to one or more parts of functionality in the solution space.

It’s a bad idea to let any solution terminology infect the description of the problem specifications. It is usually a good idea to push a lot of the problem terminology into the solution design.

That is, a common feature in a lot of systems is the ability to manage a particular category of data. Which is often mapped to ‘new’, ‘edit’, and ‘delete’ GUI screens in the software. It would be a mistake to call “editing” that data a feature. It is not a feature, it is just one of the parts of the functionality needed to address “managing” that data. It would be good to be more precise about the naming of that data. So, it’s asymmetrical.

Now it may seem overly pedantic to discuss it this way, but there are deep seeded reasons why it is better.

An obvious one is that just implementing part of the functionality is not actually implementing the whole feature. So, we see a lot of systems out there with ‘incomplete’ features and they are extremely frustrating to use. Partially solving any problems often just makes those problems worse.

Another good reason to do this is that the terminology and necessity in the problem space does not always map nicely into the solution space. In fact, usually, it is kind of horribly denormalized, if not entirely irrational. People want to work in a way that is comfortable for them, which is usually far more difficult to implement. Because of this, we often stumble into the problem of whether a ‘simplification’ of any kind is just effective for the programmer or whether it really benefits the user. It is rarely both. It’s a classic trade-off.

If you do some ‘analysis’, that should be purely in the problem space. If you do some ‘design’, then it quite obviously needs to be in the solution space. If they are kept distinct, then we have a way of not blurring the lines and injecting noise into the outputs. The users need to keep track of something happening in their business. That is their feature. If that actuates into a portal screen or a report or some other construction, that is part of the design of the solution.

One place that is super common to see a mess is in classic requirements. They are often a haphazard mix between features, implementations, designs, conventions, abstractions, etc. Pretty much anything from either space gets brutally mixed and matched and combined into a swamp. This is the key reason why they don’t get cross-checked properly and validated early on. This leads to confusion, ambiguities, and tonnes of scope creep. It’s a bigger reason why a lot of systems derail than methodology. If the requirements are convoluted, then any attempts to clean them up without splitting out the spaces will fail too, and sometimes just make the confusion worse.

If you had a clean list of features and another clean list of functions, then you could just mechanically go through all of them and see which functions address which features. You can see if the feature is fully or partially addressed. You can see useless functions. You would also see which features are not addressed at all. That would let you double-check that the solution design will actually help to ensure it is a good fit.

A big problem with modern software development is that the definitions we use in our industry are vague, blurry, convoluted, and often deliberately scrambled.

Orientation

When working on a large piece of software, the single most important goal is needing to get a really good piece of code out there that has no immediate defects.

That is, software being a list of instructions, the underlying quality of that list is the key thing to focus on. If you need to do ten things, then neither less nor more than ten is correct. You need ten things, you should create code that does ten things.

Thus, the fundamental tenet of software engineering then is that it is far better to produce no code than to produce bad code. That is, not having done anything at all is better than having produced a crappy program, since that code will just make things worse.

So, there are 3 simple rules to help make the code good. Any and everything should be all about getting goodness into the code.

1. The code will solve the problems it was written to solve. Not partially, but completely.
2. Any unexpected weirdness external to the code will not keep the code from solving the problems it was written to solve.
3. As the focus expands to solve more and larger problems, the code can also expand to correctly solve more and larger problems.

Then #1 is all about analysis and design. You can’t solve something if you don’t know what it is that you are solving. If you do a poor job of figuring out the problem, then the solution will be partial, which is bad.

#2 is all about error handling and interfaces. Other computers or people cannot disrupt the code. It still works, at worst it slows down, but it still works. It will produce the solution, even in the craziest of circumstances. People will still be able to get the solution, even in the craziest of circumstances. It’s not enough to just get it to work, it has to always work in order to not be bad.

And #3 is about extensibility. The next release is not the last release. Systems just keep growing, and we want to make it easier to grow as it gets larger and more complex. So, the code is readable, the fundamental behaviors are documented, and it’s not so hard to onboard a bunch of new developers to build out more stuff that correctly fits into the old stuff.

If you cover those 3 rules, then you’ll get good code. But to do that the focus needs to be on covering those rules, if it is misdirected onto anything else, you’ll end up with bad code.

Software Development Power Tools

You can type out a million lines of code. It will take you quite a while and it will have an incredible amount of bugs, but you can do it.

Along the way, you can make all of the code as concrete as possible. Code out each and every special case separately. Make everything isolated, so that the scope of any change is minimized.

This is the brute-force approach to programming. You just go through the long and hard exercise of pounding it all out separately. Then you go through the long and painful exercise of testing each and every possible permutation. Right at the end, you will get a workable codebase.

It works, but it is not particularly efficient, so it doesn’t scale at all. You can use it for small and maybe medium-sized systems, but that’s it.

Or, you can learn how to use power tools.

They are the digital equivalent of construction workers' power tools, like nail guns, chainsaws, power sanders, or sawzalls. And like their physical counterparts, while they can massively increase efficiency, they do also require new skills and are quite dangerous. So you have to be very careful with them. The web is littered with plenty of stories of coders injuring themselves while recklessly playing around.

A software development power tool has a couple of basic attributes. First is that when applied correctly it will save you a lot of time in coding. The second is that because it requires less code, it will also require less testing and cause fewer operational issues. But it will take more cogitative effort while working. And the final attribute is that it will actually be harder to debug, although there will be way, way fewer bugs.

When power tools are used properly they will massively cut down on code, testing, bugs, etc. When they are used poorly they can waste a lot of time, cause far more bugs, cause more damage to the code, and occasionally derail entire projects.

We can start with the most obvious power tool category: reuse. If you retype in the same code over and over again it obviously takes way longer. But as the code changes, the different variations fall out of sync with each other and cause bugs. But reusing code isn’t easy. You have to make it safe, quick, and convenient for everyone or they won’t do it. So, while it is easy to say, it is actually hard to enforce.

The granddaddy of all power tools: abstraction. Instead of pounding out each and every special case, you take a few steps back and come up with some generalization that covers all of them correctly. Of course, if you go too far back it becomes useless, and if you swing out in the wrong direction it becomes convoluted. Finding a perfect abstraction is a near art form, but if you do, huge programs can shrink down into manageable engines.

Complexity has a tendency towards exponential growth, so the next power tool tackles it directly: encapsulation. It’s often described as information hiding too, which is similar. Instead of having the guts of everything intermixed as one big exposed mess, or redoing it all as silos, you package the lower parts as well-defined, trustable, reusable opaque boxes. Internally they do something complex, but outside some of the complexity is taken away from the caller. All major technologies, such as operating systems, databases, etc. do this for the programmers. It is used everywhere, sometimes well, sometimes poorly. Its companion power tool is ‘layers’, in that they are at full strength only if they encapsulate enough complexity to make them worthwhile.

Statically encoding anything is always fragile, so the next power tool gets around that: dynamic behavior. Most of the time in code, the computer itself does not need a lot of the knowledge that the programmer has carefully placed there. So, the practice is to write code that only places the absolute minimum constraints on itself. Instead of just keeping a list of integers, for example, you keep a list of “anything”. The elements in the list are dynamic, so the underlying types can vary. Besides data structures, you can apply these ideas all over the place. The backend might not know about the specifics of any of its data. It just blindly grabs structures in response to requests. The front end doesn’t have to know either. It gets structures, maps them to widgets and display elements, then interacts with the user. None of the code knows about or understands the domain; the data then is just some data. Of course, eventually, the rubber has to hit the road or all you have built is another domain-specific programming language, so there is somewhere in the system that actuates the configuration, such that those details get passed around as more data. A bit tricky to set up, but immensely powerful afterward. Many of the huge, popular, vendor products are variations on this, some are full dynamic workflow implementations.

The expression ‘premature optimization is the root of all evil’ is just a warning that this next approach is also a power tool: optimization. The tendency is to think that optimizations are algorithmic ways to cut down on computation, but really they are far more than that. For example, you can remember and reuse data you saw before, its full name is memoization, but a sub-variant of it, caching is often quite popular. Some approaches that people think are optimizations like space-time trade-offs are actually not. The resources have shifted, not gotten smaller. Most categories of optimizations are fairly well documented, but they actually require significant research in order to fully understand and apply them. Attempting to reinvent an existing optimization is often a setback of maybe 20 to 40 years of knowledge.

A cousin of optimizations: concurrency. It’s not technically an optimization, in that you're doing the same amount of work, you're just distributing it to more computing engines all at the same time. To get it right you have to strongly understand atomicity and locking. When it’s wrong, it often manifests as Heisenbugs, nearly impossible to find or correct. Multi-process work was difficult enough, but they removed the safeties to allow multi-threaded behavior. Dealing with it is mandatory to be able to scale. But small and many medium systems don’t need it at all, the significant management overhead it causes can outweigh the benefits.

At a more micro level: syntactic sugar. This occurs when there are two or more ways to encode the same thing in a programming language, but one has a drastically reduced fingerprint. Used properly it is less code and better readability. Used badly, it is near zero readability, and possibly a landmine. It’s the easier power tool to access, and it is often the first one that people start to play with, but it is also the one that is the most abused and has caused the most harm.

You can’t discuss power tools without discussing the obsequious one: paradigms. Despite fancy language features most code these days is really just simply Procedural. That is, there are lots of functions that are cobbled together for questionable reasons. Object Oriented, Functional, etc. are deep paradigms that are embedded directly into the underlying languages. To get value from them, you have to really learn how to use them correctly. It’s a lot of often conflicting knowledge, with many different eclectic flavors. Misusing a paradigm leads people down dark holes of wasted effort. Most often, you shouldn't even mix and match them, it rarely works well. Probably the best approach is to use one consistently, but if it starts to not fit nicely, tweak it everywhere until it improves.

An oldie but goodie: normalization. Usually associated with just the data, it can be applied to code as well. Instead of just tossing the data into any haphazard structure in the database, you apply a strict set of rules to rearrange it and make sure that it has useful properties, like no redundancy. It’s a bit obtuse, but even partially done it makes the code way smaller and cleaner which cuts down on bugs, hacks, and very expensive data fixes. Data forms the foundation of every codebase, building on a broken foundation never goes well.

There are probably a lot more power tools out there, and no doubt there are still some new ones yet to be discovered.

In the panicked rush to code, people often avoid power tools, but that is frequently a rather limiting direction. Even if it all starts out as small, almost any successful coding will continue to grow. Once it passes into medium size territory, if you still want to go farther, you have to use the techniques and skills that will allow you to keep going. To build large systems, strong organization and power tools are absolute necessities, although lots of people end up learning that the hard way.

Identity

There is a collection of rules, sort of spanning the planet, to deal with identity, loosely known as “know your client” (KYC). These are fairly well-known in the financial industry. We need something similar for social networks, although the C could stand for something else.

In the age of unlimited surveillance, while it is sometimes important to verify someone’s identity, we also have to make sure that privacy is also preserved. Everyone should be able to explicitly choose how much they reveal about themselves to others. It’s a delicate balance.

My sense is that there is a direct tie between fame, wealth, and desired influence. We can leverage that to craft a few simple rules.

First is that trustworthy verification of any individual or corporation is actually expensive. You pretty much need a bunch of different, totally independent, ways to really confirm it. It’s not a one-shot thing, the potential for identity theft means that it could change at any time. If the verification is tied to others or in a chain, someone, somewhere will figure out how to break one link and then corrupt it.

Second is that money needs to be on the line. If something is spoofed, someone is penalized. If the punishments aren’t real or enforced the system breaks down.

Third is that actual verification is a specialized skill. So, the people performing it need to be licensed.

In a lot of ways, it is similar to insurance. Without enforcement, any charlatan will offer it, only to renege on the payouts later.

To get there, any organization wanting to claim verified users would have to register with an international institution, probably part of the UN. They would have to pay a lot, and may also have to put up a sizable deposit as a stake.

If at any time, it was shown that their verification of identities was untrustworthy, not only would they lose their certification, but so would each and every person or company that they verified. Because of this, people will be very careful about whom they pay to do their verifications. If it is cheap, it is also very risky, it may fall apart quickly.

Then for everyone else, their accounts are ‘unverified’ even if they are tied to a digital asset like an email or phone.

So, from a technical perspective, it is very similar, if not nearly identical to certificates, but with harsher consequences for failure.

So, then, if any sort of social platform wants or needs to claim that its users are actually verified, it would have to get the specifics of the verification credentials from the person, double-check them with the verifier, and then keep them very secure. If the verifier of an account got into trouble, the social network would need to automatically, in real-time, remove any notice of verification.

For all other users, they would be classified as unverified (quite possibly a dog, or cat, or something).

As far as promoting content, all social networks would be restricted from doing any sort of promotions on unverified accounts. That is unless you pay to get verified, the stuff you put out there will only be visible to your friends. It won’t appear in searches, top lists, etc. Your connections might decide to pass on what you say, but anybody not directly connected to the unverified post would never be aware of it. The scope will be limited. Each hop to the next set of people would have to be manually triggered by one of the connections, one at a time. Even verified accounts can’t just widely promote a non-verified tweet.

Now that may seem to favor rich or famous people, after all, they have lots of money to get verified, and they do get a larger voice, but it’s not too bad. There is still a chance for everyone else to get their contributions to go viral, based on their network of connections, it is just somewhat less than it used to be. It would also be slower too, as it would only occur one hop at a time. This is the cost of not letting the really bad things also go viral.

In that sense, the general rule would be that no social networking site, of any type, can offer promotional or viral features to unverified users. If they did, they would be heavily fined and it would be enforced internationally. Unverified users could still join, most often for free, and still, contribute content, but it would be somewhat harder and slower for their content to spread. Growing unverified connections for new accounts would also be more difficult as well, as they are harder to find. This throttling does not stop the medium from letting the truth get free, but it does force it through a much longer process before it explodes virally.

In terms of allowing anonymous whistle-blowers, rather obvious they would only use unverified accounts. Anyone else that believes what they are saying can no longer ‘link’ to their posts, they would have to add what they say as their own content themselves. So a verified journalist might see something questionable, but can’t just blindly pass it through, which means that they would not just rush to be first, but be forced to have to confirm the information before they used it. People may lament the death of the truth being quickly accessible, but while that was an incredibly good thing, bad actors took advantage of it as well, in order to flood the world with lies. If we don’t want the lies anymore, and we probably don’t, then we have to return to some means of throttling them.

Lost

I wasted the whole day. I thought the problem was somewhere else in the code, so I spent the entire day there tiring to figure out why it wasn’t working. But the problem was in some other completely different part of the code.

You’d think after thirty years of heavy coding that wouldn’t happen to someone with as much experience as I have. But it does.

It probably happens way less than when I was younger, but it still happens.

And that’s why the way we often try to estimate the amount of work is hopeless (but fixable).

My problem yesterday was that I was working on a large disconnected group of changes, all at the same time. Bouncing back and forth between them. The latest thing I was doing was not working, but the real problem was something I did a few days before to some other part of the code.

I was bouncing, not because of time -- it isn’t really more efficient -- but really because it was boring, and there were some deep technical problems that I haven’t yet figured out how to properly handle. So, I’d work on it a bit, jump to something easier, get that done, and then return. But this week, I have a couple of those tricky problems all at the same time, so it got a little messy.

It didn’t help that I was having an ‘off day’. Kinda down in my mood; kinda slow in my thinking. If you’ve been coding hard for a long time, sometimes your brain just needs a break. You hit this level of mental tiredness, often described as burnout. So, you need to take a bit of a breather; pace yourself.

That is, not every day is a “heavy coding” day. Somedays you just want to do boring, stupid, or trivial things. It’s because of that that I often keep two different to-do lists. One for the hard stuff and one for the boring stuff.

So, what does this have to do with estimations?

Well, if the work you are going to do spans a longer period of time, and you are still at the stage in your career where you really do need to focus on only one thing at a time, then there are going to be lost days in the middle of your effort.

So, you can’t just say it will take X days. Some of those days -- and you don’t know which ones -- will get messed up by life, moods, exhaustion, and other things. It’s not going to take exactly X days. In fact, you don’t have any idea how many days it will take.

But all is not lost. You can often know that if it went really well, it might take Y days. And if it goes really badly it could take as long as Z days. So, honestly, the work will take between Y and Z days, you’ll know afterward.

That is, estimates for intellectual work should not be a fixed period of time, instead, they should always be best and worse cases. A range. Two values.

If you get into the habit of thinking of them as ranges, then collecting them together and converting them into scheduling becomes a whole lot easier.

Informality

Although the universe is bounded by its physics, it is flexible enough to accommodate what is essentially a massive number of unimaginable situations. That is a fun way of saying “almost anything can happen”.

So, let us describe that potential as being ‘informal’.

We think there are a set of rules which dictates behavior, but we do understand that there are all sorts of things that can happen “outside” of our known rule set. They may occur at infinitesimally small probabilities, like say once in a billion years, but that doesn’t mean that they can’t happen. They just might not happen in our lifetimes.

Contrast that with mathematics.

We have constructed mathematics abstractly, as a ‘formal’ set of rules and axioms. These explicitly bind any mathematical objects, context, and derived objects, so that anything that occurs outside of them is effectively declared ‘invalid’; not part of the system. Let us refer to that as ‘formal’. We created these objects, the rules, etc. and we can visualize and walk through the mechanics ourselves, and in doing so we strictly enforce the rules.

There are some mathematical objects like ‘infinity’ or concepts like ‘perfection’ that we can use and make part of these formal arrangements, but they do not necessarily derive from our real-world experiences, just from our intellectual capabilities.

In that sense, we can talk about each branch of mathematics as being one or more formal systems, and similarly, we can define our real, existing universe as a really big informal one.

With that framing, we take any mathematical ‘relationships’ and use them to describe things we see in reality, such as physics, accounting, statistics, or even economics. While those relationships mostly fit and explain what we are seeing, the formal math itself is always just an approximation to the underlying informal mechanics. Formal relationships are too rigid to fully express informality.

That is, even though we might describe a physics formula as a ‘law’ since reality is informal, it is entirely possible that while it mostly fits, there could be some unknown and inconceivable circumstance where it does not. If we are unaware of any such circumstance, we tend to believe that the fit is ‘perfect’, but rather obviously if that isn’t a valid real-world concept either then that is a flaw in our understanding, communications, and perceptions.

The rather obvious example of this is gravity. For a long time, people knew it was there as an obvious visible effect, but it took Sir Issac Newton to describe the relationship. Then later Albert Einstein refined it, and they are still trying to corral that within Quantum Physics. Each time, the mathematical relationships got tighter, but also more complex.

So, it gives us a great way of understanding the relationships we are seeing in the world around us. We take formal mathematical relationships and see if they fit informational circumstances. If the fit looks good, we add it to our collective intelligence and build on those foundations. But really, we should know that no formal relationship will ever 100% fit into an informal circumstance, or at least we should always leave room for doubt or more precision.

History is littered with us applying increasingly complex mathematical relationships which do advance our capabilities but later are supplanted by even better, more precise mathematical relationships. It is a never-ending process to just keep improving the precision of our knowledge and expectations.

Now, that happens regardless of what occurs in mathematics. And it is often the case that mathematicians are busy exploring new objects and formal systems long before we ever find a way to relate them back to our world and our lives. My favorite example is Évariste Galois who created Group Theory back in 1829, which was used since the 1980s to help provide scratch resistance in the form of error-correcting codes for encoding music on CDs. There may have been earlier examples of this mathematic branch getting applied in our lives, but certainly this one effect billions of people, most of whom were obvious of what was going on. Sometimes it takes a while to match the formal relationships back to the informal universe.

No doubt, there are formal mathematical relationships that we understand that will never be applied back to the world around us. Just because we can craft a formal system, doesn’t mean it corresponds with some aspect of our informality. That’s the corollary to the fit possibly never being perfect. There is a disconnect there.

What’s really interesting about this gulf between formality and informality is computers.

As the CPU runs, it enforces the formal running of computations, which are Turing-complete. So, the execution of code within a computer is strictly formal. The program runs, or it stops. But, that execution is running on top of hardware, which is bounded by our reality, so it is actually informal. That is, the hardware can glitch, which would throw off the software, breaking the illusion of formality. it doesn’t happen often, but it does happen.

Thus, once we started playing with computers, we rather accidentally found a way to bind our abstract formal systems to our physical informal one. That has a bizarre implication.

Formal systems then are not necessarily just fully abstract things that exist in our minds. Mathematics may not be just the product of our imaginations. It tends to imply that the formality of our abstract systems themselves has some, although limited, binding to the rules that underpin the universe. That is our imaginations as physical constructs may also be bounded by the universe.

In that sense, maybe under all of that informality lies a strictly formal set of rules? Not sure we’ll be able to figure that out in our short lifetimes, but it does seem like there is something below the probabilistic nature of quantum physics and that we haven’t quite reached the bottom yet.

A more obvious consequence of the gulf comes from using computers to solve real-world problems. The core of the software needs to be formal, but the problem space itself is informal. So, we end up mapping that informality onto formalized versions of data and code as solutions, and we often tend to do it statically since that is easier for most people to understand. But of course, both the informal and possibly dynamic nature of the problem space usually throws a wrench into our plans.

Follow Suit

The very best developers always follow suit on any existing codebases.

The expression ‘follow suit’ comes from card games. If the last card was a Spade, you follow with a Spade. If instead, the last card was Clubs, you follow with Clubs.

In coding, it means that if there is an already existing codebase, with a bunch of styles, conventions, idioms, paradigms, and dependencies, then any code that you add will fit in perfectly. That is, it leverages any underlying existing code, looks the same as all of the other code and is in the correct architectural location.

Why is this important?

If the code is a pile of different conventions, styles, etc. it is nothing more than a mess. So, if you go into a codebase and make even more of a mess, although some of your contributions are positive, overall they are negative. You’re not helping.

Messy code is always really buggy code, annoying to the users, and is stupidly expensive to fix or extend.

Now, it is true that if you don’t follow suit, you can save yourself lots of time and get right down to work. But while that is time you saved for yourself today, it is also technical debt that you just dumped on someone else. Professionally, pushing your problems onto other people so you don’t have to deal with them is not acceptable.

The other thing is that if you rewrite 3 months worth of work that was already done, completed, and tested, you have just wasted 3 months of time. You’re not moving the project forward, you’re just making it inefficient.

Now if you have personal preferences for say a particular coding style, it is essentially subjective. Get out of your comfort zone, and at least try the other style. Maybe it’s better, maybe it’s worse, but if you can only code one way, then you are a rather limited one-trick pony, which isn’t so great for your career prospects.

If you are the first coder into the codebase, you have a choice of ‘suit’. But if you are not first, you do not have that choice, so don’t pretend like you do.

It may be that some of the things in the code, like its idioms, are new to you. That is a good thing, a chance to learn, not a reason to avoid it.

Now if you believe the other code is “wrong” then now you are actually obligated to fix it. All of it. So, it’s not a good idea to judge some work if you are not going to have the time or ability to redo it. It’s not “wrong”, it’s just code and it follows a particular convention, and you need to follow suit. If you’ve developed a hate-on for a particular way of doing things, that is your issue, not a coding one, you should try and fix yourself.

In some cases, the codebase does have legitimate issues that really do need correcting. If you set about to correct them, you can’t just do so locally, that creates a mess. You have to do so for the entire project, across the whole codebase. Not only is that a lot of work, but you also have to get agreement from any other coders on the project, and management, to back the time necessary to do the work. It is entirely possible. it is good practice, and while it is often mundane and boring, that type of refactoring is a very strong skill. In most large codebases, there is a lot of cleaning up that really should be done, but people are avoiding it. Obviously, it can’t just be a subjective quirk, you need to have solid reasons why spending the time to make these changes is worth it.

In the case where the code is just an unrepentant mess, the best practice is to pick one of the many suits that are already there and get everyone to agree that it is the best one. Then, wherever you go in the mess, if you touch a file, you are now responsible for updating it. Slowly, that will bring down the degree of messiness, until the last bit of effort is finally tangible. While it takes a long time and is slow, you are not avoiding it, so you are still acting professionally.

Knowable

Some people believe that the universe is intrinsically unknowable.

That is there will always be mysterious things happening around us that we won’t be able to explain. It’s a somewhat romantic perspective on the world.

Other people believe the universe operates by a series of nearly-strict formal rules, deep underneath.

This means that any and every behavior or event is explainable, it’s just that we either don’t have deep enough knowledge yet, or the frequency of occurrence is so tiny that we haven’t been able to imagine the event before it occurred.

From that view, It’s not that the universe is random, but rather that our encounters with it are probabilistic. Things that happen, that you didn’t expect to happen, only happen rarely. So any surprises are a lack of expectations, not some mysterious forces.

In that second perspective, if you view the world with objectivity, you can gradually build up a better sense of the types of things that can and will occur and in doing so that brings you a bit closer to understanding our surrounding reality. You become a realist, not hoping for the best, nor complaining about the worst. Just accepting things as they are.

If you see the universe that way, then all things of interest are on some trajectory. You can then make inferences about where they are going, but also understand that any sort of unusual collision may change their course. Then you can catalog any of the known possible collisions and their frequencies. Obviously, common collisions are part of your assumed trajectory, but less common ones deserve consideration too.

If you understand the allowable paths for any given trajectory, you can make arrangements to route through them to the desired result. That is, you proactively know where you are going, while reactively steering through the known and occasionally unexpected obstacles to get there.

It also makes it easier to understand what just happened. Look for some uncommon collision and reinterpret its frequency as needed. A bunch of trajectory shifts in unlikely places can interact to alter seemingly unrelated trajectories, the proverbial perfect storm.

A mysterious universe is more appealing to some people, but collectively we do seem to have already acquired enough knowledge to dispute it.

Our next great feat is to be able to leverage what we know, and will soon learn, to keep our own trajectory from spiraling down. We’ve mastered living in harsh climates but have not figured out how to prevent unintended side effects. We shape the world around us for our own convenience, but it is still destructive. If we can sort out these mistakes while picking better destinations, we can live up to our intellectual potential.

Bugs

Most bugs in our code have yet to be discovered.

They exist, either as mistakes, misunderstandings, typoos, or cheats. But the circumstances that would bring them into the light, as a pending development priority, have just not occurred yet.

There was a popular movement a while back that insisted that a limited form of testing would magically ensure that the code was bug-free. There was another less popular movement that doing a tonne of external work could prove the ideas behind the code were right. Both of these approaches are predicated on the idea that all bugs can be removed.

Pretty much the best and only reasonable definition of a ‘bug’ is that they are some annoying behavior of software that bugs someone. That is an incredibly wide definition that would include bugs for things like ugly or awkward interfaces, mismatches in conventions, inconsistencies, under-simplifications, and over-simplifications.

The healthy way to see bugs is that software is and will never be “perfect”. At best, if you work on it for a long time you might converge closer to perfection, but really you’ll never get there in your lifetime. So, bugs are an inevitable fact of building software. They exist, there are lots of them, and whenever you get a chance you should correct them, even if they are small or currently unnoticed.

The quality of any code is the number of bugs and the readability of the code itself. That is, it is the short-term usage, and the long-term extensibility, of the stuff you are building. The work to remove bugs is an exponential curve. It is easy to remove easy bugs, but it is a crazy amount of work to find and correct the difficult ones. Some bugs are just out of reach of the time frame. You may or may not know about them, but you’d never get the time to fix them either way.

Because bugs happen, any sort of release/deployment process needs to deal with them. That is, there is a strict, slow, correct way to release the next upgrade, and there is a fast, dirty, somewhat risky way to just get a patch into the wild. You have to have both, but the default is to always take the longer, safer way first.

There are lots of things coders can do to prevent and catch bugs. Usually, they are ‘habit’ driven. That is, without knowing any better, some of the ways people are constructing code is far more likely to be buggy. But if they just made small changes to the way they were working, fewer bugs would happen. As well, there are lots of ways beyond independent ‘tests’ that would help detect and prevent bugs from getting out into the wild. We have to accept bugs, but we can also change our working habits to reduce their frequency.

Naming

Naming is hard. It is the beginning of all confusion and laziness in code, and it is usually where most bugs hide. If the code is littered with bad names, you should be very suspicious of it.

Oddly, the hardness of the problem is because people expect it to come easily. So they get frustrated when it doesn’t. Then they just cheat and use really bad names instead.

As people see other people’s badly named mistakes, they accidentally follow suit, so the conventions really tend to degrade with each generation. And since they are mistakes, it becomes even harder to understand why they were chosen, making it all even more confusing.

But naming is an explicit indication that you understand what you are doing, and that you think about it clearly. The two most important characteristics of good programmers: the mechanics are clear in your head (unique) and you understand what is happening (precise). With those two things, most names, most of the time, are fairly obvious. If you also understand the degree of generality you need, then you know how to properly lift the names as well.

How do you know if the names are bad?

Easy. Go verbally explain your code to another programmer.

If your names are self-explanatory, you will use them directly while you are talking. If each name needs a description, explanation, or some other help, then they are bad. More to the point, if what you are trying to do needs any overly complex explanations and those explanations cannot be directly abstracted away in one step from the work itself (e.g. objects, models, patterns, idioms, data structures, etc.), then the whole thing is messy and convoluted, and likely does not work in the way you expect it to.

Because naming is hard but fundamental, it is one of the core skills that you should work on developing very early in your career. If you wait until you’ve learned other stuff first, your naming habits will be so poor that the stuff you produce won’t work even if you know how to do it properly.

Operational Issues

If there is a problem with a system in “production”, rather obviously that should be recorded with all of the relevant facts and hopefully an ability to reproduce it.

Then the problem should be triaged:

• Is it an operational problem? Such as configuration or resources?
• Is it a data administration problem? Bad or stale data?
• Is it a development problem? Our code or a vendor’s code is incorrect?

There should be a run book that lists all old and new problems this way.

If a problem is operational, say we’ve run out of some type of resource, then 2 things need to occur:

1. Bump up the resource, and restart the system
2. Analyze the problem and make recommendations to avoid it in the future.

If the problem is data administration, then there are a few options:

• Was the data changed recently? Who changed it and why?
• Is it stale? Why did it fall out of sync?
• Is there a collision between usage? Both old and new values are needed at the same time?
• Who is responsible for changing the data? Who is responsible for making sure it is correct?

If it is a development problem, then:

• Is there a workaround?
• How important is it that this gets fixed?
• When can we schedule the analysis, design, coding, and testing efforts necessary to change it?

In the case of resources such as operating system configuration parameters, the first time they occur for any system it will be a surprise, but they should be logged both against the system itself and the underlying tech stack, so that later even if it happens in a completely different system, the solution to correct it quickly is known and already vetted.

If it is a well-known tech stack issue, then operations can address it immediately, and only later let the developers know that it had occurred.

If the problem is infrequent or mysterious, then operations may ask for developer involvement in order to do a full investigation. If the problem is repeating, trivial, or obvious, then they should be able to handle it on their own. If a developer needs to get involved, then they often need full and complete access to production, so it is not something you want to occur very often and it is expensive.

For common and reoccurring problems, operations should be empowered to be able to handle them immediately, on their own. For any system, the default behavior should be to reboot immediately. If reboots aren’t “safe” then that needs to be corrected right away (before any other new work commences).

As well as reactively responding to any problems with the system, operations need to be proactive as well. They should set up their own tooling that fully monitors all of the systems they are running, and alerts them to limited resource issues long before they occur. Non-trivial usage issues come from the users.

Upgrades and bug fixes should be tested and queued up by development, but deployed by operations at their convenience. Since operations are actively monitoring all systems, they are best able to decide when any changes are made. Operations are also responsible for tracking any underlying infrastructural changes, assessing the potential impacts, and scheduling them, although they may have to consult with vendors or in-house development to get more information.

Data Engineering

The foundation of all software systems is persistent data.

That is, a big part of any solution provided by a software system is the ability to digitize events, inventories, and conversations. Then collect them together and make them reliably available in the digital realm.

Like anything else, these foundations have to be in place first, before you can build any other type of computation on top. Building on top of broken foundations is a waste of time and resources.

An organization only needs only one copy of any of the possible trillions of digital records. They need to know where it is located, and unless it is accessed frequently they don’t need speed. If it takes a few minutes to get to infrequent data, that is fine. In some cases, an hour might be acceptable as well.

Obviously, size, frequency, and usage are critical things that need to be understood.

For every different type of record, at least one person in the company needs to really understand it. Its structure, its usage, its frequency, and any special cases that will be seen during collection. If you just collect data blindly, the quality of that data will always be very poor. You get out of data, the effort you put into making sure it is right.

Often beside the primary copy of the record, there can be a lot of secondary copies getting created as well. Sometimes for performance, more often because of politics. That is fine if all of them are ‘read-only’, but it is a really, really bad mistake if people are updating those secondary copies.

You have to capture the data as it exists. You can’t flatten structural data without losing information. You often can’t inflate it back due to ambiguities. So, if you spend the effort to capture it into a primary database, it is always worth any time and effort to get the structure correct. Collecting a lot of broken databases is a waste of resources.

If you have just one primary copy of good-quality data, then building out any other computations and interfaces on top of it is straightforward. If you find that what you are building is instead extremely complex, then it is often because of the poor foundations. It is way, way more efficient to fix the foundations first, instead of just piling more problems on top. The house of cards only ever gets worse, it will not fix itself and the real problem is below.

So, the core of data engineering is to be able to craft models for the incoming data that correctly list out the size, frequency, usage, and structure of each different ‘type’ of record. All of them. Then the other part is to ensure that for any organization there is only one primary source and that any secondary sources are always read-only. If that is in place, crafting databases, ETLs, or any other sort of data piping will go well. With that as a foundation, building anything on top will be cheaper and far better quality.

Dynamic Systems

A picture is worth a thousand words, as they like to say. But it is also frozen in time. It doesn’t change, or at least it shouldn’t. 

And it’s a great analogy to describe things being ‘static’. Basically, a photograph is a directed static capture of just one specific moment of our dynamic reality. For a lot of pictures, pretty much right afterward at least one little thing has changed, the picture is no longer reproducible. Reality has moved on, it will never be the same again.

Now it’s worth noting that a video is a long sequence of pictures played together so that we can see a larger set of changes. While it is still static externally, it isn’t a moment anymore, it is now a period of time.  So, when you view it, it appears to be dynamic, but you can keep replaying it over and over again, so it isn't really.

Things that are dynamic can constantly change. They can change across all possible dimensions. Obviously, if we were going to try to model that, it would be impossible. 

But we can make some of the dimensions dynamic, while leaving others to be static, such as what happens with a video. Then things are not ‘entirely’ dynamic, but there are still dynamic aspects that are captured, at least for some period of time.

All of that is fairly easy to understand, but it gets a little less so when applied to mapping aspects of reality onto software. Fundamentally, what we want to do is capture some information about the world around us with our computers. It’s just that it’s highly multi-dimensional, and many of those dimensions are intrinsically dynamic. 

We should stop and talk about the multi-dimensional aspect of digital information. We mostly perceive our world as four dimensions, so we could capture any point in time and space, and combine these together to get descriptions of any type of objects. We do have higher dimensional theories, but we tend to think of our world in terms of 4D.

In software though, we have a large number of ‘variables’; that is things that can and do vary. Each one only has a finite number of different possibilities and at any one time there are only ever a fixed number of them, but since we tend to use a crazy large number of variables, it is worth treating each and every one as though it was essentially its own dimension. We can boil that down to a subset of variables that are so ewhat independent of each other, but since that is still a fairly large number it doesn't really change our perspective. It is a huge number of dimensions.

A static program then is one where the number of dimensions and the spaces it occupies, cannot change. For example, you write a problem with exactly 3 integer variables and the result is thay are added together, then we know the entire input space (all triples permuted from integer min to max) and the output space (an integer from min to max) are both fixed. The entire possible behaviour is static. 

A dynamic program however can introduce new variables. 

The easiest way to do this is to keep a list of integers and not have a maximum list size. At some point the list will grow too large for the hardware, but we’ll ignore that for the moment. While each integer has a fixed space, the number of them is essentially ‘nearly’ infinite, which we will call very large. So, the overall space of the input if it were just a list to be added together is very large. If the output were another list, then it too would be very large. 

In that sense a single variable is one dimensional, but a list of them adds a new dimension. That’s not a surprise, but weirdly a matrix doesn’t add a second one, two lists or a spreadsheet do, and a dag seems to add more than one. That is, the expressive power of data structures is multi-dimensional, but rather than being fixed static dimensions like the variables, each dimension is a “very large” one. And it is that huge bump in expressive power that makes the whole system dynamic.

But that isn't the only way to make systems dynamic. Another way is to make the variables variable. That is, instead of only an integer, the value can be any valid primitive type in the language. But more fun is to let it also be any user constructed composite variables, e.g. structs or objects. It is fully polymorphic. Then the code doesn’t know or care about the type or structure of the data which means the code can be used for an essentially unlimited number of computations. So, rather than statically encode one block for each possible type of variable, you encode the block once for all types, primitive or composite or data structure. 

If that is the heart of the code, basically an engine, then you can apply that code to a huge set of problems, making the usage of the code itself dynamic.

So, we can make some of the data dimensions dynamic and we can make some of the code dynamic. And we can pass that attribute into any of the interfaces used to interact with the system. There can employ the same code and data techniques to make the interface dynamic.

Now that we can construct dynamic computations, we can map real world dynamic behaviours onto these, so that we model them correctly. As things swing around in the real world, we can capture that correctly in the model, and build systems around that. 

We can tell if something is mismatched in a system. 

If it was encoded statically, but it suffers from scope creep all of the time, that is a clear indication that there is at least one dynamic dimension missing. 

In fact, unless the coverage of the model never changes, either some dynamic attribute was missed or the interface is incomplete. But for that latter case, any "missing features” are easy to implement. However, if something was incorrectly made static however, the code and data changes are usually big, difficult, and/or expensive. 

So we can really test the suitability of any underlying model used in a software solution by examining the requested list of changes, classifying them as domain increases, incomplete interfaces, or invalid modelling. New domain territory needs to extend the code and data to a new part of the problem space, incomplete interfaces and invalid modelling were covered above.

There is also a sub-branch of mathematics called “dynamic programming” which is a bit confusing in that it predates computer programming, but its use of the term dynamic is similar to this discussion. You can encode some algebraic expressions that when applied will dynamically converge on an answer. I believe that you can encode both the code and the data for this statically, so we have yet another category, which is effectively a dynamic computation. It’s no doubt related to the mathematical concept of uncomputable, but that knowledge is too deep for this discussion, other than alluding to the notion that dynamic behaviour may be able to spring from seemingly static roots.

Some things in reality are static. They essentially will not change for the expected lifetime of the system (usually 30 years). Other things are intrinsically dynamic. They can’t be locked down. In order to build good systems that don’t have ugly side effects, we need to implement any dynamic behavior with dynamic data or code. Trying to cheat that with static implementations will generate unintentional and undesirable side-effects with both the construction and usage of the system.

Phone vs Cable

There are only two basic paradigms for communicating between things (threads, processes, containers, machines, hardware) in software.

The first is based on explicit requests, basically like a telephone. One computer phones up another computer, they talk, then the line is closed.

The other is broadcast based like cable TV. A source computer sends out a signal over a wire, everyone else just listens to the different channels.

Request-based communication is very specific, but also expensive. That is, you can request a piece of data, but both the server and client have to be actively involved. If there is a small number of clients this works quite well, but it usually fails to scale.

Broadcast-based communication is very general, but also relatively cheap. One server can stream out tonnes of different channels. If it’s capable, it can do it steadily.

Request-based systems degrade under a heavy load. If too many people connect, they tend to either slow down for everyone or drop connections.

Broadcast systems are independent of load. If they are set up to handle N channels, then they can consistently do that going forward. As long as the server can handle sending out M channels you can go from N to M.

If you need to ensure that the two machines actually reliably communicated something to each other then requests are far better. The client can ensure that it actually gets the data, or that it will continually retry until it is retrieved. The server knows exactly what it got and from which client.

For broadcast, the server often doesn’t even know who the clients are, how many of them are out there, or what they have received. For a client, if it misses part of the stream, and there is no historic cache of the older broadcasts, then it will not be aware of the data.

Over the decades, there have been lots of technologies that try to combine both paradigms. They might receive a broadcast, but then be able to initiate a connection as well. Or it is request based but receives a stream of something like events or updates later. While that gives the best of both worlds, it does still open up the servers to performance problems, just that they are less frequent.

There is a slight difference in error handling. For requests, the client needs to keep retrying until it is successful. For broadcasts, the client needs to be aware of its own uptime, and do extra processing in order to fill in any gaps that occurred.

Phrased as “phone vs cable” it is pretty easy to remember which basic properties are tied to which paradigms. Sometimes it is obvious which one you need, but sometimes the trade-offs are difficult. Usually, the difficulties are mixed with issues about integrity, frequency, and correctness.

Helpers or Encapsulation

I was having a discussion with another developer recently. He suggested that we could create a library of ‘helpers’.

I like creating libraries for most things, but this suggestion turned me off. It just sounded wrong.

A while ago I was working on a medium-sized system where the developers basically went crazy with ‘helpers’. They had them everywhere, for everything.

Generally, if you take any ideal in programming and apply it in an over-the-top extreme manner it doesn’t work out very well, and that was no exception.

It basically destroyed the readability, and the code in the helpers was haphazard, all over the place. So without bouncing into lots of different files and directories, you wouldn’t get any sense of what the code was actually trying to do. And far worse, that fragmentation was hiding some pretty significant bugs, so really it was a huge mess.

But breaking things down is one of the core ideas of good software development, so why did the helper-fest go so horribly wrong?

You want to build up a lot of reusable pieces, gradually moving up from low-level operations into higher-level domain primitives. You want this so that when you go to build new stuff, you can most often do it from a trusted set of existing lower stuff. You’ve built a lot of the parts already, you’ve tested them, and they are being heavily used. So, if you leverage your earlier work you’ll save yourself massive amounts of time.

But you also need to encapsulate any of the complexity. If you don’t it will grow out of control.

Basically, you want to stick something complex into a box, hide it away, and then keep reusing it. Once you have solved a specific coding problem, you don’t want to waste time by having to solve it again.

Encapsulation is always more than just code. There are some data, constants, or config parameters of some type that goes inside the box as well. You might even have a bit of state in there too, but you have to be very careful with that. As long as some of the underlying details never leave the box, you’ve at least encapsulated something.

So, a good reusable lower component is encapsulated code, data, and the mechanics needed to do some functionality, that is nicely wrapped and hidden away from the caller. You see this commonly in language libraries, string handling is often a good example. You don’t get caught in messing with an array of individual characters, instead, you do common operations directly on the abstraction notion of a ‘string’.

Basically, you’ve removed a piece of complexity and replaced that with a box that adds only a little bit of extra complexity. Overall, complexity is higher, but at any given level you have lowered it.

As long as you name it correctly, it is reusable. Later someone can find it, use it, and trust it, without having to get lost in those underlying details.

The way most people write helpers though, they are just pure code fragments wrapped with a function. Really just some idiom or a clump of code that people are using often. It’s more like a cut and paste, but without making explicit copies.

So that’s where helpers get into trouble. Basically, they’re often disjoint code fragments, arbitrarily chosen, and both because they do not encapsulate, and because they are not well-defined primitives, you could reuse the code but the overall complexity is still going up. It doesn’t hide things, really it just reduces retyping. You gain a little bit from calling it, but you lose a lot more from its inherent complexity, especially if you need to read it later.

And that’s pretty much what I saw in that earlier project. Readability was shattered because the lines drawn around the helper code fragments were arbitrary. You can’t even guess what the calls will do.

In that earlier case though, a lot of the helpers also modified globals, so the side effects were scary and rather unpredictable. But even if that wasn’t true, the general ideas around helpers may help reduce pasted code, but do not encapsulate any complexity, so they really aren’t helping much.

A variation on this theme is to have a large number of static functions in an OO system. It’s the same problem just not explicitly called out as helpers.

Probably the worst thing you can do with code in a large system is duplicate stuff all over the place. It’s a recipe for creating bugs, not functionality. But avoiding encapsulation by using a weaker approach like helpers isn’t really better. There is never enough time to do a proper job coding anymore, so you have to make any work you do count. The more time you save, the more time you now have to do a better job.

Reinvention

Programmers are sometimes confused between reinventing a wheel and just simply rewriting it.

Let me explain…

Underneath all code is a model of operation that is often based on the interaction of sophisticated data structures. That is the core of the abstractions that drive the behavior. That is what makes the code work.

The more sophisticated the model, the more likely it works better (not always true, but more often true than not).

If you learn or read about those abstractions, then you can use that knowledge to faithfully write new implementations. That is just a rewrite. You’re not trying to invent something new, but are just leveraging all of the existing knowledge out there that you can get your hands on.

If you ignore what was done in the past, and essentially go right back to first principles, that is a reinvention. You will try to come up with your own sophisticated model that is in direct competition with what is already out there. If that existing work is already decades old, you are essentially going back in time to an earlier starting point, and you will have to find your own way from there, through all of the same problems again.

In simple cases that is not a big problem. You can craft a new GUI that looks somewhat like the older ones. They are fairly routine and usually conceptually simple.

In advanced cases though, like persistent storage, frameworks, languages, or operating systems it is a major leap backward. So much was done, discovered, and leveraged by a massive number of people, but you are just ignoring everything. It’s unlikely that by yourself, you have anything close to a tiny fraction of the time needed to relearn all that was already known. What you create will just be crude.

So we really don’t want to reinvent stuff, but that is quite different from just writing our own version of existing stuff. The key difference is that long before you start a rewrite, you had better spend a great deal of effort to acquire the state of the art first. If you are just blindly coding something that seems okay, you’re doing it wrong.

Given that distinction, it gets interesting when you apply it to smaller dependencies.

If there are some simple ones that you could have written yourself, then it is way better to write them yourself. Why? You’re not reinventing the wheel, you already have the knowledge and it is always better to directly control any code you have to put into production. If you wrote it, you understand it better, you can fix it faster, and it is likely more future-proof as well.

If there are complicated dependencies, with lots of mature features, then unless you obtain the precise knowledge of how they work, you are probably better off relying on them. But, as they are already complex technology, you still have to spend a lot of time to understand them, and you should leverage them to their fullest extent, instead of using subsets of a bunch of similar libraries.

In between, it likely depends on your experiences, the domain, and both the short-term and long-term goals of the organization. Having your own code is “value”, and building up a lot of value is usually super important, both for in-house and commercial projects. A ball of glue might be quick to build, but it is always an unstable nightmare and is easily replaced by the next ball of glue.

Software Factories

I am uncomfortable with the factory model of a software development shop.

I heard it from a seasoned tech exec decades ago. Effectively, the dev shop is a factory that takes in requirements and produces releases. Basically, it’s a bunch of software assembly lines.

The obvious problem is that it indirectly implies that programming is similar to blue-collar work. That you get a bunch of programmers together, tell them what to do, they do it, and then the results get shipped. It is sort of treating it as if it were all physical, not intellectual.

While that offends the idealist in me, it is pretty much the case now that most software development is routine. Libraries and frameworks deal with the hard parts, most coding is just gluing it all together to meet a specific goal. If you read the tutorials and basic documentation, any sort of missing answers can be found on sites like StackOverflow. You can learn the basic skills fairly quickly, and in most cases just keep applying them over and over again.

There is still a ‘rapid comprehension’ issue. That is, you might have found the answers, but you still need to understand them quickly. Active programmers are usually continuously learning stuff. It’s like being in school, and it is almost never-ending. Maybe if you stayed on the same codebase for over a decade there isn’t much left to learn, but for most coding positions, there is a vast unknown.

One of the tweaks to the factory model is that analysis is inside the factory, not outside of it. The requirements come in the front door, but the analysts venture out from the factory to investigate and flesh out the issues. They could be specialized roles like Business Analysts, but often they include all people heavily involved in the design like architects and development leads as well. You can’t really build a good solution if you don’t understand the problem.

The other major issue with the model is scaling. Big companies with lots of IT needs, tend to behave like a lot of little independent factories. This means they are each redoing the same fundamental work, over and over again. Besides being highly inefficient, it also leads to severe quality issues. The organization never gets competent at producing software, even if a few of its factories do. But they tend to disperse, taking all of that knowledge with them.

It would also be better to have some reliable horizontal factories at different levels that support the vertical factories for business needs. That is, you get groups of developers that write backend technical stuff, that is mandated to be used by the other developers. Obviously, enforcement is a huge problem, in that the quality of work for the lower-level developers has to be extremely high, or it negates the usefulness of the effort.

One of the good things about the factory model is that it clearly delineates the boundaries of development. That is, the factory produces updates and versions and stuff, but it is isolated from the act of ‘operating’ this stuff. It sort of implies that a well-run dev shop is rather quiet and focused on its work, even if the business itself is rambunctious and full of stress. Obviously, that would be far better for quality. You can’t code very well if you are constantly being interrupted. You don’t need perfect code, but the side effects of running bad code can be hugely expensive.

And that split between development and operations is super important. They are different mindsets, so pushing the devs heavily into operations issues just tends to just burn them out. They should be focused on engineering things to work correctly, and while they should obviously get a lot of feedback to ensure everything works as expected, they don’t need to be front and center in the ongoing operational issues. Rather they should complete their work, then ship it to others who will put it into production, monitor it, and deal with any unexpected behaviors.

So, while the factory model bothers me at times, there are aspects of it that I wish more people understood and paid attention to. You often see these development shops where everything is hopelessly blurred, people are stressed and confused, and the work is going very badly. We need to find better ways to structure the work so that it doesn’t cost so much and take a crazy amount of time to get done. We seemed to be getting worse at building software, not better.

Would you like Fries with that?

I often run into software developers who have very poor relationships with their users.

I’m sure there are a few users-from-hell out there, but often the users are okay, and still, the relationship is quite bad. That’s unfortunate and unnecessary, and it just contributes to stress on both sides. Mostly, it can be avoided.

The first part of fixing the relationship is acceptance.

The software industry will tell you that you are an artisan, a craftsperson, a magician, or at least super intelligent. From this many software developers come to believe that they should be worshipped. That the users should be thankful for any tiny amount of effort that they make. The developers think they are special, above it all.

But the truth is far less romantic. The users have a problem. It’s a problem that a computer can help solve. So they hired a bunch of people to build things that solve their problems. Those people are highly skilled and hopefully professional, but they are still in their positions just to build solutions for other people’s problems.

In that sense, from the user perspective, while it is different from going to a fast food restaurant and ordering a meal, it isn’t really that different. They ask the developers to solve specific problems, then the developers figure it out, and give them back a solution. The users often fund, initiate, and drive the work; they do have a reasonable expectation of ‘service’.

“PROGRAMMING IS NOT A SERVICE INDUSTRY!!” I can hear some readers screaming.

Well, it is not entirely a service industry and there are certainly programming jobs where there are zero service components, but since most software is used by people to solve their problems then most of it has significant service components. We deliver features, it’s built right into our job descriptions.

I know it is very hard to accept that, but accepting it is the first major step to not having a contentious relationship with the users.

If the users suddenly want you to make some quick hacks to an existing screen that seem unnecessary to you, you probably should not say “would you like fries with that?” but you should definitely think it. It will put the conversation in the right frame for you.

Now in no way am I recommending that you just blindly do whatever crazy thing the users want you to do. You don’t need to take them literally.

Often their “solutionizing” is just the way they are trying to express some aspect of their problem to you. You need to see through what they are saying to what they actually need. When you do see it you should feel free to redirect them to better solutions, ones that fit better into the overall context.

In that sense, if you have engaged with them and listened to their problems, you can help lead them in a more positive direction. But you still have to solve their problems.

Saying “no!” or “I’m too busy...” isn’t just rude it is also self-defeating. Making up some lane excuse isn’t any better. Even if you don’t know what is driving their desires, they are still expressing a need, and you have to spend some time figuring it out.

Extreme candor isn’t necessary, but you need to be honest with people for them to believe you. If they believe you, they will eventually trust you. And if they don’t trust you, they certainly are not going to take your advice; things won’t go well.

Of course, if you fix this, it will be way better in the future, but you still need to always treat them with respect.

You want to stick to just the facts, be as honest as you can, and rephrase stuff into their terminology. You want to let them know what is happening, but not make excuses. If there are good, viable, options you can present them, but if not, don’t phrase it as a question. If there isn’t a choice don’t offer one. If you don’t know something, admit it. If someone got sloppy, admit that too. It’s okay to say “that release didn’t go as planned” instead of pretending that it worked.

We’re not perfect and some things that seemed easy turn out to be brutally hard or impossible. If you're honest and they trust you, they’ll appreciate you letting them know. Not everything, and not to blame other people, just more of an understanding with them.

You know that it is your job to provide usable solutions to their problems; if they know that too the respect will be mutual. And that will erase a lot of unnecessary stress and give you the ability to concentrate harder on your work. Just don’t forget to ask a lot of questions, they do understand their problems a lot better than you ever will.

End-points and Computations

There are lots of different ways to decompose large software projects.

A strong decomposition that is applied consistently across a system forms the base of good organization, which make the development smoother and provides better quality.

One way to look at the different types of code in any large system is to separate it between end-points and computations.

We’ll start computations.

If you have a bunch of inputs, you can apply some work to them, and you’ll end up with a bunch of outputs. That is a simple, rather pure, stateless computation.

Way down on the nearly trivial scale, we have operators like addition. You take 2 integers, add them together and provide the result. You can go slightly higher up to something like string concatenation, where you join two strings to form a larger one.

But it also applies to much higher, larger groups of instructions. For instance, you might calculate some complex metric like a bond yield, from the description and current time series around the bond in a market. Way more information than addition or concatenation, but still the same general idea. It’s just a computation.

It’s stateless, and everything you need to compute successfully comes in from the inputs. Then it either works or it gives you a reason for not being successful.

Described that way, you can see that ‘compiling’ for a language like C or Golang is in itself also just a computation. You give it the ‘source’ and you end up with a binary of some type or a list of very specific errors.

But we can go even higher. You might give some piece of code a URL, and some navigation stuff, and it will return a clump of data like JSON. It's still a computation, just one part of it is distributed. It triggers one or more other machines to do their computations based on the input you sent it.

So you could structure the code that calls someone else’s REST API as a series of stateless computations. And if the API were somewhat stateful itself, you can just take the output of one call and use it as the input for another, and still keep it somewhat stateless. At least each of the calls is stateless, even if the combined interaction is not.

We can also see that going to some large backend for persistent data, say an RDBMS or NoSQL database, is the same. We might give it an id for something, and it returns all of the associated data with that id, in a particular structure. Still a computation, and still devoid of state on each call.

Then that leads us to the definition of an end-point. Really it is any leftover code that is itself not a computation.

For instance, in the backend rest API, there is some routing code that bonds the URL to the code you want it to execute. Sort of a computation, but not really. It’s just the end-point mechanics to route incoming things to the right handlers. You could pull out any simple computations from the mechanisms used to actually trigger the code.

A GUI might have a bunch of buttons that people can press. As they do, sometimes a ‘context’ builds up. Then at some point that leaves the interface end-points and triggers the desired computation. Maybe if it’s a web app, the app itself is mostly end-points, and the backend directs it to the correct computation.

So, any end-point code is stateful, contextual, configurable, etc. Often quite messy. All of the other bits of code that are necessary to wire up stuff to users or other computers, to run correctly. It could include operational issues, platform issues, configuration, etc.

And it tends to be the code that runs into the most difficulty.

It’s not that hard to write a computation, and while it might take a bit of work to get a multi-party distributed computation working correctly, it is fairly easy to test its behavior.

It is hard though to set up a bunch of end-points and make sure that they are durable enough to withstand the chaos around them. So, end-points tend to have a lot more bugs. They are the front line, where all of the problems originate.

So, now if you can clearly separate these two different types of code for a large system, it opens up a lot of good organizational properties.

For instance, you know to put all of your computations into shared libraries, so that a lot of other people can use them too. But you also know that the end-points are specific, and tend to be ugly and redundant. So, you don’t waste a lot of time trying to figure out how to reuse them. They tend to be single-use. At best maybe you provide a skeleton or template or something to get people up and going faster.

If you lean on that perspective, you realize that minimizing end-points is a great thing, and maximizing the computations is good too.

When we have talked about building up reusable lego blocks from the ground up, it usually means the computations. Where we have talked about just writing things up quickly, it usually means the end-points. And if you have a lot of thin end-points separated from libraries of shared computations, you have a great deal of flexibility in how you will deploy stuff, but also the ability to leverage the bulk of the work you have already done.