Orientation

When working on a large piece of software, the single most important goal is needing to get a really good piece of code out there that has no immediate defects.

That is, software being a list of instructions, the underlying quality of that list is the key thing to focus on. If you need to do ten things, then neither less nor more than ten is correct. You need ten things, you should create code that does ten things.

Thus, the fundamental tenet of software engineering then is that it is far better to produce no code than to produce bad code. That is, not having done anything at all is better than having produced a crappy program, since that code will just make things worse.

So, there are 3 simple rules to help make the code good. Any and everything should be all about getting goodness into the code.

1. The code will solve the problems it was written to solve. Not partially, but completely.
2. Any unexpected weirdness external to the code will not keep the code from solving the problems it was written to solve.
3. As the focus expands to solve more and larger problems, the code can also expand to correctly solve more and larger problems.

Then #1 is all about analysis and design. You can’t solve something if you don’t know what it is that you are solving. If you do a poor job of figuring out the problem, then the solution will be partial, which is bad.

#2 is all about error handling and interfaces. Other computers or people cannot disrupt the code. It still works, at worst it slows down, but it still works. It will produce the solution, even in the craziest of circumstances. People will still be able to get the solution, even in the craziest of circumstances. It’s not enough to just get it to work, it has to always work in order to not be bad.

And #3 is about extensibility. The next release is not the last release. Systems just keep growing, and we want to make it easier to grow as it gets larger and more complex. So, the code is readable, the fundamental behaviors are documented, and it’s not so hard to onboard a bunch of new developers to build out more stuff that correctly fits into the old stuff.

If you cover those 3 rules, then you’ll get good code. But to do that the focus needs to be on covering those rules, if it is misdirected onto anything else, you’ll end up with bad code.

Software Development Power Tools

You can type out a million lines of code. It will take you quite a while and it will have an incredible amount of bugs, but you can do it.

Along the way, you can make all of the code as concrete as possible. Code out each and every special case separately. Make everything isolated, so that the scope of any change is minimized.

This is the brute-force approach to programming. You just go through the long and hard exercise of pounding it all out separately. Then you go through the long and painful exercise of testing each and every possible permutation. Right at the end, you will get a workable codebase.

It works, but it is not particularly efficient, so it doesn’t scale at all. You can use it for small and maybe medium-sized systems, but that’s it.

Or, you can learn how to use power tools.

They are the digital equivalent of construction workers' power tools, like nail guns, chainsaws, power sanders, or sawzalls. And like their physical counterparts, while they can massively increase efficiency, they do also require new skills and are quite dangerous. So you have to be very careful with them. The web is littered with plenty of stories of coders injuring themselves while recklessly playing around.

A software development power tool has a couple of basic attributes. First is that when applied correctly it will save you a lot of time in coding. The second is that because it requires less code, it will also require less testing and cause fewer operational issues. But it will take more cogitative effort while working. And the final attribute is that it will actually be harder to debug, although there will be way, way fewer bugs.

When power tools are used properly they will massively cut down on code, testing, bugs, etc. When they are used poorly they can waste a lot of time, cause far more bugs, cause more damage to the code, and occasionally derail entire projects.

We can start with the most obvious power tool category: reuse. If you retype in the same code over and over again it obviously takes way longer. But as the code changes, the different variations fall out of sync with each other and cause bugs. But reusing code isn’t easy. You have to make it safe, quick, and convenient for everyone or they won’t do it. So, while it is easy to say, it is actually hard to enforce.

The granddaddy of all power tools: abstraction. Instead of pounding out each and every special case, you take a few steps back and come up with some generalization that covers all of them correctly. Of course, if you go too far back it becomes useless, and if you swing out in the wrong direction it becomes convoluted. Finding a perfect abstraction is a near art form, but if you do, huge programs can shrink down into manageable engines.

Complexity has a tendency towards exponential growth, so the next power tool tackles it directly: encapsulation. It’s often described as information hiding too, which is similar. Instead of having the guts of everything intermixed as one big exposed mess, or redoing it all as silos, you package the lower parts as well-defined, trustable, reusable opaque boxes. Internally they do something complex, but outside some of the complexity is taken away from the caller. All major technologies, such as operating systems, databases, etc. do this for the programmers. It is used everywhere, sometimes well, sometimes poorly. Its companion power tool is ‘layers’, in that they are at full strength only if they encapsulate enough complexity to make them worthwhile.

Statically encoding anything is always fragile, so the next power tool gets around that: dynamic behavior. Most of the time in code, the computer itself does not need a lot of the knowledge that the programmer has carefully placed there. So, the practice is to write code that only places the absolute minimum constraints on itself. Instead of just keeping a list of integers, for example, you keep a list of “anything”. The elements in the list are dynamic, so the underlying types can vary. Besides data structures, you can apply these ideas all over the place. The backend might not know about the specifics of any of its data. It just blindly grabs structures in response to requests. The front end doesn’t have to know either. It gets structures, maps them to widgets and display elements, then interacts with the user. None of the code knows about or understands the domain; the data then is just some data. Of course, eventually, the rubber has to hit the road or all you have built is another domain-specific programming language, so there is somewhere in the system that actuates the configuration, such that those details get passed around as more data. A bit tricky to set up, but immensely powerful afterward. Many of the huge, popular, vendor products are variations on this, some are full dynamic workflow implementations.

The expression ‘premature optimization is the root of all evil’ is just a warning that this next approach is also a power tool: optimization. The tendency is to think that optimizations are algorithmic ways to cut down on computation, but really they are far more than that. For example, you can remember and reuse data you saw before, its full name is memoization, but a sub-variant of it, caching is often quite popular. Some approaches that people think are optimizations like space-time trade-offs are actually not. The resources have shifted, not gotten smaller. Most categories of optimizations are fairly well documented, but they actually require significant research in order to fully understand and apply them. Attempting to reinvent an existing optimization is often a setback of maybe 20 to 40 years of knowledge.

A cousin of optimizations: concurrency. It’s not technically an optimization, in that you're doing the same amount of work, you're just distributing it to more computing engines all at the same time. To get it right you have to strongly understand atomicity and locking. When it’s wrong, it often manifests as Heisenbugs, nearly impossible to find or correct. Multi-process work was difficult enough, but they removed the safeties to allow multi-threaded behavior. Dealing with it is mandatory to be able to scale. But small and many medium systems don’t need it at all, the significant management overhead it causes can outweigh the benefits.

At a more micro level: syntactic sugar. This occurs when there are two or more ways to encode the same thing in a programming language, but one has a drastically reduced fingerprint. Used properly it is less code and better readability. Used badly, it is near zero readability, and possibly a landmine. It’s the easier power tool to access, and it is often the first one that people start to play with, but it is also the one that is the most abused and has caused the most harm.

You can’t discuss power tools without discussing the obsequious one: paradigms. Despite fancy language features most code these days is really just simply Procedural. That is, there are lots of functions that are cobbled together for questionable reasons. Object Oriented, Functional, etc. are deep paradigms that are embedded directly into the underlying languages. To get value from them, you have to really learn how to use them correctly. It’s a lot of often conflicting knowledge, with many different eclectic flavors. Misusing a paradigm leads people down dark holes of wasted effort. Most often, you shouldn't even mix and match them, it rarely works well. Probably the best approach is to use one consistently, but if it starts to not fit nicely, tweak it everywhere until it improves.

An oldie but goodie: normalization. Usually associated with just the data, it can be applied to code as well. Instead of just tossing the data into any haphazard structure in the database, you apply a strict set of rules to rearrange it and make sure that it has useful properties, like no redundancy. It’s a bit obtuse, but even partially done it makes the code way smaller and cleaner which cuts down on bugs, hacks, and very expensive data fixes. Data forms the foundation of every codebase, building on a broken foundation never goes well.

There are probably a lot more power tools out there, and no doubt there are still some new ones yet to be discovered.

In the panicked rush to code, people often avoid power tools, but that is frequently a rather limiting direction. Even if it all starts out as small, almost any successful coding will continue to grow. Once it passes into medium size territory, if you still want to go farther, you have to use the techniques and skills that will allow you to keep going. To build large systems, strong organization and power tools are absolute necessities, although lots of people end up learning that the hard way.

Identity

There is a collection of rules, sort of spanning the planet, to deal with identity, loosely known as “know your client” (KYC). These are fairly well-known in the financial industry. We need something similar for social networks, although the C could stand for something else.

In the age of unlimited surveillance, while it is sometimes important to verify someone’s identity, we also have to make sure that privacy is also preserved. Everyone should be able to explicitly choose how much they reveal about themselves to others. It’s a delicate balance.

My sense is that there is a direct tie between fame, wealth, and desired influence. We can leverage that to craft a few simple rules.

First is that trustworthy verification of any individual or corporation is actually expensive. You pretty much need a bunch of different, totally independent, ways to really confirm it. It’s not a one-shot thing, the potential for identity theft means that it could change at any time. If the verification is tied to others or in a chain, someone, somewhere will figure out how to break one link and then corrupt it.

Second is that money needs to be on the line. If something is spoofed, someone is penalized. If the punishments aren’t real or enforced the system breaks down.

Third is that actual verification is a specialized skill. So, the people performing it need to be licensed.

In a lot of ways, it is similar to insurance. Without enforcement, any charlatan will offer it, only to renege on the payouts later.

To get there, any organization wanting to claim verified users would have to register with an international institution, probably part of the UN. They would have to pay a lot, and may also have to put up a sizable deposit as a stake.

If at any time, it was shown that their verification of identities was untrustworthy, not only would they lose their certification, but so would each and every person or company that they verified. Because of this, people will be very careful about whom they pay to do their verifications. If it is cheap, it is also very risky, it may fall apart quickly.

Then for everyone else, their accounts are ‘unverified’ even if they are tied to a digital asset like an email or phone.

So, from a technical perspective, it is very similar, if not nearly identical to certificates, but with harsher consequences for failure.

So, then, if any sort of social platform wants or needs to claim that its users are actually verified, it would have to get the specifics of the verification credentials from the person, double-check them with the verifier, and then keep them very secure. If the verifier of an account got into trouble, the social network would need to automatically, in real-time, remove any notice of verification.

For all other users, they would be classified as unverified (quite possibly a dog, or cat, or something).

As far as promoting content, all social networks would be restricted from doing any sort of promotions on unverified accounts. That is unless you pay to get verified, the stuff you put out there will only be visible to your friends. It won’t appear in searches, top lists, etc. Your connections might decide to pass on what you say, but anybody not directly connected to the unverified post would never be aware of it. The scope will be limited. Each hop to the next set of people would have to be manually triggered by one of the connections, one at a time. Even verified accounts can’t just widely promote a non-verified tweet.

Now that may seem to favor rich or famous people, after all, they have lots of money to get verified, and they do get a larger voice, but it’s not too bad. There is still a chance for everyone else to get their contributions to go viral, based on their network of connections, it is just somewhat less than it used to be. It would also be slower too, as it would only occur one hop at a time. This is the cost of not letting the really bad things also go viral.

In that sense, the general rule would be that no social networking site, of any type, can offer promotional or viral features to unverified users. If they did, they would be heavily fined and it would be enforced internationally. Unverified users could still join, most often for free, and still, contribute content, but it would be somewhat harder and slower for their content to spread. Growing unverified connections for new accounts would also be more difficult as well, as they are harder to find. This throttling does not stop the medium from letting the truth get free, but it does force it through a much longer process before it explodes virally.

In terms of allowing anonymous whistle-blowers, rather obvious they would only use unverified accounts. Anyone else that believes what they are saying can no longer ‘link’ to their posts, they would have to add what they say as their own content themselves. So a verified journalist might see something questionable, but can’t just blindly pass it through, which means that they would not just rush to be first, but be forced to have to confirm the information before they used it. People may lament the death of the truth being quickly accessible, but while that was an incredibly good thing, bad actors took advantage of it as well, in order to flood the world with lies. If we don’t want the lies anymore, and we probably don’t, then we have to return to some means of throttling them.

Lost

I wasted the whole day. I thought the problem was somewhere else in the code, so I spent the entire day there tiring to figure out why it wasn’t working. But the problem was in some other completely different part of the code.

You’d think after thirty years of heavy coding that wouldn’t happen to someone with as much experience as I have. But it does.

It probably happens way less than when I was younger, but it still happens.

And that’s why the way we often try to estimate the amount of work is hopeless (but fixable).

My problem yesterday was that I was working on a large disconnected group of changes, all at the same time. Bouncing back and forth between them. The latest thing I was doing was not working, but the real problem was something I did a few days before to some other part of the code.

I was bouncing, not because of time -- it isn’t really more efficient -- but really because it was boring, and there were some deep technical problems that I haven’t yet figured out how to properly handle. So, I’d work on it a bit, jump to something easier, get that done, and then return. But this week, I have a couple of those tricky problems all at the same time, so it got a little messy.

It didn’t help that I was having an ‘off day’. Kinda down in my mood; kinda slow in my thinking. If you’ve been coding hard for a long time, sometimes your brain just needs a break. You hit this level of mental tiredness, often described as burnout. So, you need to take a bit of a breather; pace yourself.

That is, not every day is a “heavy coding” day. Somedays you just want to do boring, stupid, or trivial things. It’s because of that that I often keep two different to-do lists. One for the hard stuff and one for the boring stuff.

So, what does this have to do with estimations?

Well, if the work you are going to do spans a longer period of time, and you are still at the stage in your career where you really do need to focus on only one thing at a time, then there are going to be lost days in the middle of your effort.

So, you can’t just say it will take X days. Some of those days -- and you don’t know which ones -- will get messed up by life, moods, exhaustion, and other things. It’s not going to take exactly X days. In fact, you don’t have any idea how many days it will take.

But all is not lost. You can often know that if it went really well, it might take Y days. And if it goes really badly it could take as long as Z days. So, honestly, the work will take between Y and Z days, you’ll know afterward.

That is, estimates for intellectual work should not be a fixed period of time, instead, they should always be best and worse cases. A range. Two values.

If you get into the habit of thinking of them as ranges, then collecting them together and converting them into scheduling becomes a whole lot easier.

Informality

Although the universe is bounded by its physics, it is flexible enough to accommodate what is essentially a massive number of unimaginable situations. That is a fun way of saying “almost anything can happen”.

So, let us describe that potential as being ‘informal’.

We think there are a set of rules which dictates behavior, but we do understand that there are all sorts of things that can happen “outside” of our known rule set. They may occur at infinitesimally small probabilities, like say once in a billion years, but that doesn’t mean that they can’t happen. They just might not happen in our lifetimes.

Contrast that with mathematics.

We have constructed mathematics abstractly, as a ‘formal’ set of rules and axioms. These explicitly bind any mathematical objects, context, and derived objects, so that anything that occurs outside of them is effectively declared ‘invalid’; not part of the system. Let us refer to that as ‘formal’. We created these objects, the rules, etc. and we can visualize and walk through the mechanics ourselves, and in doing so we strictly enforce the rules.

There are some mathematical objects like ‘infinity’ or concepts like ‘perfection’ that we can use and make part of these formal arrangements, but they do not necessarily derive from our real-world experiences, just from our intellectual capabilities.

In that sense, we can talk about each branch of mathematics as being one or more formal systems, and similarly, we can define our real, existing universe as a really big informal one.

With that framing, we take any mathematical ‘relationships’ and use them to describe things we see in reality, such as physics, accounting, statistics, or even economics. While those relationships mostly fit and explain what we are seeing, the formal math itself is always just an approximation to the underlying informal mechanics. Formal relationships are too rigid to fully express informality.

That is, even though we might describe a physics formula as a ‘law’ since reality is informal, it is entirely possible that while it mostly fits, there could be some unknown and inconceivable circumstance where it does not. If we are unaware of any such circumstance, we tend to believe that the fit is ‘perfect’, but rather obviously if that isn’t a valid real-world concept either then that is a flaw in our understanding, communications, and perceptions.

The rather obvious example of this is gravity. For a long time, people knew it was there as an obvious visible effect, but it took Sir Issac Newton to describe the relationship. Then later Albert Einstein refined it, and they are still trying to corral that within Quantum Physics. Each time, the mathematical relationships got tighter, but also more complex.

So, it gives us a great way of understanding the relationships we are seeing in the world around us. We take formal mathematical relationships and see if they fit informational circumstances. If the fit looks good, we add it to our collective intelligence and build on those foundations. But really, we should know that no formal relationship will ever 100% fit into an informal circumstance, or at least we should always leave room for doubt or more precision.

History is littered with us applying increasingly complex mathematical relationships which do advance our capabilities but later are supplanted by even better, more precise mathematical relationships. It is a never-ending process to just keep improving the precision of our knowledge and expectations.

Now, that happens regardless of what occurs in mathematics. And it is often the case that mathematicians are busy exploring new objects and formal systems long before we ever find a way to relate them back to our world and our lives. My favorite example is Évariste Galois who created Group Theory back in 1829, which was used since the 1980s to help provide scratch resistance in the form of error-correcting codes for encoding music on CDs. There may have been earlier examples of this mathematic branch getting applied in our lives, but certainly this one effect billions of people, most of whom were obvious of what was going on. Sometimes it takes a while to match the formal relationships back to the informal universe.

No doubt, there are formal mathematical relationships that we understand that will never be applied back to the world around us. Just because we can craft a formal system, doesn’t mean it corresponds with some aspect of our informality. That’s the corollary to the fit possibly never being perfect. There is a disconnect there.

What’s really interesting about this gulf between formality and informality is computers.

As the CPU runs, it enforces the formal running of computations, which are Turing-complete. So, the execution of code within a computer is strictly formal. The program runs, or it stops. But, that execution is running on top of hardware, which is bounded by our reality, so it is actually informal. That is, the hardware can glitch, which would throw off the software, breaking the illusion of formality. it doesn’t happen often, but it does happen.

Thus, once we started playing with computers, we rather accidentally found a way to bind our abstract formal systems to our physical informal one. That has a bizarre implication.

Formal systems then are not necessarily just fully abstract things that exist in our minds. Mathematics may not be just the product of our imaginations. It tends to imply that the formality of our abstract systems themselves has some, although limited, binding to the rules that underpin the universe. That is our imaginations as physical constructs may also be bounded by the universe.

In that sense, maybe under all of that informality lies a strictly formal set of rules? Not sure we’ll be able to figure that out in our short lifetimes, but it does seem like there is something below the probabilistic nature of quantum physics and that we haven’t quite reached the bottom yet.

A more obvious consequence of the gulf comes from using computers to solve real-world problems. The core of the software needs to be formal, but the problem space itself is informal. So, we end up mapping that informality onto formalized versions of data and code as solutions, and we often tend to do it statically since that is easier for most people to understand. But of course, both the informal and possibly dynamic nature of the problem space usually throws a wrench into our plans.

Follow Suit

The very best developers always follow suit on any existing codebases.

The expression ‘follow suit’ comes from card games. If the last card was a Spade, you follow with a Spade. If instead, the last card was Clubs, you follow with Clubs.

In coding, it means that if there is an already existing codebase, with a bunch of styles, conventions, idioms, paradigms, and dependencies, then any code that you add will fit in perfectly. That is, it leverages any underlying existing code, looks the same as all of the other code and is in the correct architectural location.

Why is this important?

If the code is a pile of different conventions, styles, etc. it is nothing more than a mess. So, if you go into a codebase and make even more of a mess, although some of your contributions are positive, overall they are negative. You’re not helping.

Messy code is always really buggy code, annoying to the users, and is stupidly expensive to fix or extend.

Now, it is true that if you don’t follow suit, you can save yourself lots of time and get right down to work. But while that is time you saved for yourself today, it is also technical debt that you just dumped on someone else. Professionally, pushing your problems onto other people so you don’t have to deal with them is not acceptable.

The other thing is that if you rewrite 3 months worth of work that was already done, completed, and tested, you have just wasted 3 months of time. You’re not moving the project forward, you’re just making it inefficient.

Now if you have personal preferences for say a particular coding style, it is essentially subjective. Get out of your comfort zone, and at least try the other style. Maybe it’s better, maybe it’s worse, but if you can only code one way, then you are a rather limited one-trick pony, which isn’t so great for your career prospects.

If you are the first coder into the codebase, you have a choice of ‘suit’. But if you are not first, you do not have that choice, so don’t pretend like you do.

It may be that some of the things in the code, like its idioms, are new to you. That is a good thing, a chance to learn, not a reason to avoid it.

Now if you believe the other code is “wrong” then now you are actually obligated to fix it. All of it. So, it’s not a good idea to judge some work if you are not going to have the time or ability to redo it. It’s not “wrong”, it’s just code and it follows a particular convention, and you need to follow suit. If you’ve developed a hate-on for a particular way of doing things, that is your issue, not a coding one, you should try and fix yourself.

In some cases, the codebase does have legitimate issues that really do need correcting. If you set about to correct them, you can’t just do so locally, that creates a mess. You have to do so for the entire project, across the whole codebase. Not only is that a lot of work, but you also have to get agreement from any other coders on the project, and management, to back the time necessary to do the work. It is entirely possible. it is good practice, and while it is often mundane and boring, that type of refactoring is a very strong skill. In most large codebases, there is a lot of cleaning up that really should be done, but people are avoiding it. Obviously, it can’t just be a subjective quirk, you need to have solid reasons why spending the time to make these changes is worth it.

In the case where the code is just an unrepentant mess, the best practice is to pick one of the many suits that are already there and get everyone to agree that it is the best one. Then, wherever you go in the mess, if you touch a file, you are now responsible for updating it. Slowly, that will bring down the degree of messiness, until the last bit of effort is finally tangible. While it takes a long time and is slow, you are not avoiding it, so you are still acting professionally.