Oversimplication

For a long time, the software industry turned against sophisticated power techniques like abstraction.

Partly because getting a little abstract is difficult for some people, but also because it slows down the number of lines they can grind. As an added bonus, it is a bit harder to debug.

People felt that rapidly churning out super simple, near-trivial, brute force code was the best approach.

Strangely, though, a great deal of the actual foundation of software rests on some pretty complex abstractions. Little of what is now possible with computers would be there or work properly if it didn’t. A lot of that code came from an age when people still cared about getting it correct. It’s not ‘legacy’, it’s ‘classic’.

As I watched these oversimplification trends take hold and gain strength, I was getting increasingly worried about the frailty of all of the software that we depend on. Everything is so buggy these days that it’s a wonder that anything works. A few decades back, people would dispute that claim and say it was crazy, but now it’s way worse and so obvious the tables have turned.

There are plenty of known problems out there where any and all attempts to brute-force some crude logic will fail. The scary part is that it may not ‘completely fail’. It may just look like it’s working, work some of the time, but then fail right in the middle of consequential moments.

Software has many essentially ‘physical’ boundaries. You can’t do anything to get past them; they are hard and fast limits. It’s also not particularly malleable, and certainly gets rapidly less malleable as the codebase grows. The crap at the bottom gets frozen there as more stuff gets piled on top. A digital form of gravity, I guess.

Any sort of ‘just jump in and pound it out’ philosophy is a recipe for failure. By the time you’ve clued into your most obvious mistakes, and there will be obvious mistakes, it is too late to go back and fix them. Then building on top of that shaky mess is doomed.

If the problem you are trying to solve is complicated, then any viable real solution to that problem is at least equally complicated. We’ll call this the equality rule.

Techniques like abstraction and generalization are the only means to take that unmanageable complexity and bring it down to a reasonable level where you can implement it correctly. All your other attempts to ‘hack through’ the issues are just a waste of time.

But some people are often so focused on trying to create the simplest solution possible that they end up ignoring the problem they are trying to solve. So, it’s disconnected. The solution is wonderful, but useless.

Instead, it’s way better to go grab a sheet of actual paper and start writing out any proposed solution first. Sketch some mechanics. Think of it as modelling clay, like what they use for car design. Then take that proposed solution and go back to properly explore the problem. If they don’t match up, go back to the design board. A sketch in time saves buckets of wasted coding. It’s orders of magnitude faster than coding up a bad solution, throwing it away, coding up another, throwing that away, ad nauseam.

Grungy coding habits like that are oddly okay for demos and quick prototypes, but it only works because the code is tiny. Scale screws with everything, and any code that is good enough will always grow. Maybe we should call that the destiny rule. If the code kinda works, then it is fated for continued expansion until it doesn’t work anymore.

The oddest part, though, is that the motivation to keep it simple is great. The thing should be as simple as possible; it’s just that any distance way over or way under that bar is exponentially worse. You need to hit the bar, and that bar is positioned by the equality rule.

If there is any sort of method to the madness, it is to start simple. But know immediately that it is too oversimplified. Bad design. Keep going back to the problem to figure out why it won’t work in specific situations. Put those complexities carefully back into the design. Iterate, over and over again.

If you think that might take too long, and you really don’t have decades to catch up with the state of the art, the shortcut is not to ignore the problem, but rather to jump out and do some reading on the state of the art. Leverage what other people have already figured out. Read a bit, learn a lot. Revise what you actually think simple really means. Know that if a kink in the problem space stumbles too close to a boundary, then find the best fit possible within your timeframe to actually address it, or at least carve out some space for it to be corrected later.

What you don’t want to do is hobble something together out of components you barely understand. While that keeps your code simple, what rests below explodes with overcomplexity.

It’s worth noting that the crazy balancing act necessary for modern software design is not something you can learn instantaneously. Because of this, the fastest way to gain knowledge is always mentoring. Get on a good project, work with good teams, and get someone who knows a bit to explain their rationale. Don’t be skeptical, at least not at first. Everyone is wrong, but some people are less wrong, thus closer to the knowledge you are seeking.

While it's tempting to believe that there is always a simple software solution for everything, it’s been proven incorrect so many times that it has become boring and forgotten, again and again. If you want to build software that really solves people’s problems for them, then that software has to really solve people’s problems for them. If their problems have become crazy complicated, then it’s obvious that anything simple will not even come close to fitting properly. I really should call ‘the equality rule’ a law or something...

Ordering

With complex work, the order in which you do it hugely affects the quality, speed, and accuracy.

If there are ten subtasks, you do not really have the option to do them in any order. You need to start with some tasks first, before you do the others. They are dependent.

That’s pretty obvious if you are building a house. You can’t just do the roof first if you don’t have the walls up. And you can’t get the walls up until you have completed the foundations.

The crazy part about software is that although it is exactly like that too, most people incorrectly believe that because it is “digital”, it is somehow different. Somehow it magically escapes any dependencies, so starting in any order is fine. It is not.

As a consequence, you get situations that are equivalent to the stakeholders getting obsessed over the paint colour for a penthouse, when the foundations for the apartment building haven’t even been finalized in design yet, totally out of order.

\While we would not expect non-technical people to understand or be concerned about ordering the work, it is critical for developers to do this correctly. Maybe the second most important thing after organization.

The stakeholders only see the outside of the top; the rest of the box is invisible to them. They always assume it is trivial.

But internally, software is closest to a house of cards. It is far from trivial, where everything rests on what is below it. That orientation sets its dependencies. An interface that calls a backend, for example, is way up there. The persistence that the backend uses is far below. With that in mind, you can assess the “height” of all components.

Some architectures may flip certain parts of the game, but there is still some sort of relative height there. Figure out those heights, and you get a quick rough sketch of the ordering.

The biggest notion is that everything is built on something else. Those times when a few independent components can be built in parallel, in isolation from each other, are rare anomalies. Take advantage when you find them, but don’t expect to find them and definitely don’t assume they exist. Go the other way: assume there are far more dependencies for anything than you are currently aware of, since that is most likely to be true.

Set a height, plan as best as you can, start at the bottom, then work upwards.

When you get it right, development will go smoothly along.

When you get it wrong, there will be lots of frustration and drama. Simple changes will become huge, oversized issues. The schedule will keep getting pushed back. Way too much stress, and the quality will bottom out, causing downstream operational problems which just keep it all boiling.

Not always, but frequently, if there is a lot of drama in a development project, a key ingredient is bad ordering. That is an issue that developers can and should control. It is one of the places where they are directly responsible for the outcomes.

It’s a grey zone, though, in that plenty of management tries to control it too. It is contested ground, mostly because of timelines.

What experience teaches is that the less you say about the tasks and their order, the more control you will have over them, which you need. It’s better to frame any project scheduling around outside events, not internal ones. That’s why along the way I learned to separate ‘features’ from ‘functionality’, where one is a set of things the user needs, while the other is the specific way each one is implemented. That disconnect is not there to over-complicate things, but instead to disconnect the requirements from the work in order to retain control over the ordering of the work.

When and where some non-technical management has stuck their fingers too deeply into the pie, because they don’t care about dependencies, their ordering choices have been disastrous. Erratic and prone to burning crazy amounts of time on wasted efforts that were started far too early. And although they are the problem, they keep complaining that it's developers that are too slow, which is often beyond frustrating.

Ordering matters. It matters a lot. Once you’ve worked on a well-ordered project, it's difficult to deal with a chaotic one again. The shortest path to getting the work done is oddly very strict. A few other paths are similar, but most others are far worse. Stay on a good path; keep people from distracting you.

Yak Shaving

The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.

We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.

Yak shaving is actually a symptom of disorganization and/or ordering problems.

If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.

So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.

Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.

First, get your workspace in order. Do you have all the tools you need? Are they configured properly?

Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.

For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.

When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.

The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.

But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.

It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.

Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.

There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.

In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.

For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.

The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.

Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.

It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.

Setbacks

In the mid-eighties, I was very excited about the potential of computers. Little personal computers were disrupting those massive, expensive beasts lurking in basements. It seemed like computers could integrate into our daily lives, change the world, and make it better.

Computer Science and Informatics were having a heyday; innovation was overflowing. There seemed to be monthly progress.

But over the decades, that progress was plagued by some pretty bad setbacks.

The first one I really saw was the rise of Microsoft.

They had been the David that took down the IBM Goliath, but the moment that they got traction, they got nasty. Over and over again we’d watch as some small group of people would labour to prove some new innovative technology was viable. But just before they could stabilize, Microsoft would swoop in with an army of kids and eat their business.

I was horrified, often quite angry. Instead of software progressing, any new buds were quickly snuffed out by products of far lesser quality, but way more senseless features. Microsoft then applied the rest of its growing monopoly to crush these small competitors.

Just before this era, the heart of innovation had shifted from academics into a small group of energetic people trying to get established. They had good intentions. Mostly these were wiped out. Innovation quickly died as a result; it was too risky, and after all that pain you’d just lose anyway.

That only changed when the next setback took hold.

The Internet had been around for a while, but the Web on top exploded into a huge bubble.

On the plus side, it brought most consumers to software, but it also embraced chaos. If Microsoft killed innovation, the Web killed quality. Whacking out crude software at high speeds was favoured. The ideas shifted from helping people to just exploiting them. The product was no longer software, but a cut of the money, control over people and surveillance. Software was created as fast and crude.

In a quick tumble, we switched from empathy to pure snake oil. If you could trick a million or more people into your broken product, you were famous.

Almost as a counter to the corruption of the Web, mobile apps nearly reversed that trend. For a while, but then everyone else took a page from the Microsoft warbook and quickly bought or crushed anything even mildly innovative or decent quality.

As the mobile dust settled, another setback had been quietly brewing since the dotcom bomb took hold.

Forget quality and innovation, Agile preached mindless coding at high speed.

If you just grind out enough code this week for whatever reactive drama was dominating, the belief was that your chaos would somehow converge into something useful. That lack of direction and foresight was misguided. You can’t randomly build an apartment building; it would obviously collapse, so why would people think that might work for any large technology structure?

Instead of learning from the mistakes of the past, the software industry doubled down on its madness.

Innovative sites vanished, replaced by grungy interfaces that only captured sellable data. People seem to just adjust to a steady flow of disruptive bugs. Using computers got insanely complicated, buttressed by a flood of questionable how-to pages.

Instead of software becoming more intuitive, people just followed opaque, incoherent recipes as if they were magic spells. This produced a mountain of incomprehensible code piled high on mountains of incomprehensible code. It kinda worked, but only if you stayed very close to the magic recipes. Deviate even a little bit at your own risk. Underlying knowledge of how things worked got decimated.

For a while there, it seemed like we’d finally get enough breathing room to be able to fix the industry’s problems. Maybe this time we’d be able to figure out how to build stuff that isn’t crap, but then LLMs appeared, and it started all over again.

Now we live in an era where we are totally dependent on the messes we’ve created but no longer understand. No longer just a house of cards, we suffer from skyscrapers of cards.

And a new generation of people thinks, again, that doubling down on our lack of understanding and handling that over to random traversals of illegally compiled huge lumps of questionable data will fix our problems. But it’s really just another repeat of the same mistakes.

We keep getting distracted from trying to actually figure out how to engineer massive systems that are both reliable and trustworthy. So long as we keep avoiding that, we’ll keep tumbling over the cliffs.

Flow Design

One of the pitfalls in software design is for people to only concentrate on the narrow case where everything works as desired, but ignore all of the other cases where it has gone partially or totally wrong.

Data flow analysis can help avoid this.

The idea is that given some complex scenario for software mechanics, we can fully iterate all of the issues in advance, so we know what we need to solve.

To do this, we can examine the flow of the data.

In a simple example, the data is something a user knows. They enter that into a screen. On completion, the data is sent to the back end, which then persists it.

So the flow is:

User -> screen -> request -> backend -> persistence -> response -> user

Two key problems:

• Data is invalid
• One of the two backend resources is down

If the data is invalid, it is a computational problem. But it shows up in three places:

• screen
• request
• persistence

We’d like all three to be identical. If they do differ, then the messages need to be specific about at which level.

We’d like to give the user a chance to correct the problem without having to start all over again. They really appreciate not having their time wasted.

There are at least three computers involved in this flow. If the first one is down, the flow is blocked, so we don’t need to worry about it.

If either of the other two is down, most of the time we want to wait for a little bit, just in case they become available again, so we sleep for a bit and retry.

If it's a long disruption at either computer, we want to let the user know. Since they may communicate it to support personnel, we want to be clear about which of the two resources is down. So there are two similar error messages for the user, but distinguishable for diagnostic purposes.

If the attempt to persist is successful, we also want to let the user know. But the backend is an intermediary, so it could fail just after the persistence worked and accidentally hide the success message. It was persisted, but the user gets informed about a failure instead. We need to avoid that somehow.

So, there are six arrows, four of which are problems that should be solved in the software.

One way of getting around this is to collapse the arrows as much as possible. Distributed programming is fraught with complex error handling and transaction integrity problems. Avoiding it is the best way of solving it.

Keep in mind that this is a tiny, common, simple example. But it does include all of the spinning pieces and issues. If you code all of the issues to work correctly, then you can extend that approach to much more complex scenarios. If you get this working, then filling in the missing deterministic computations is easier.

Uncertaincy

While each line of code is pretty simple, and a few of them are not much worse, assembling big collections of code that behaves as expected is crazy difficult.

I used to think it was just scale that was the problem. Remembering a list of five things isn’t too bad; remembering a list of fifty things is beyond most people’s ability.

That can be solved with organization. Write them down, make it easily accessible. But even when the work is organized, it is still courting disaster.

I realized that any means of organizing stuff falls apart with growth. You neatly organize fifty things; it doesn’t work anymore long before you get up to five hundred. Constantly reorganizing stuff is always necessary, but people try to avoid it.

Still, even in the best huge codebase that’s been meticulously cleaned, it is still difficult. I think now that all of these smaller problems are tied together by one larger one: uncertainty. No matter how experienced you are as a programmer, lots of aspects of the work are uncertain.

The foundations and environment are too complex to fully understand. The technologies all have issues. They are wrapped up in histories and complex dynamics. The domain is messy, and understanding is constrained by the slow reveal. Each little step only shows the next step down.

Together, the norm is that you don’t know way more than you actually know. It’s worse when you are a junior, but it never, ever goes away. The explosive growth of the software industry guarantees it.

The difference between success and failure often seems to be how the developers deal with uncertainty. Do they accept it, or just pretend like it doesn’t exist?

For me, I know that I don’t know. But there are a bunch of things that I currently don’t know, but I really, really need to know them. These are the focal points. I can not complete and release the code until I know these things. Every other problem is immaterial. Resolve the key uncertainties and the rest will fall into place.

These uncertainties are often the properties implied by the way the solution is specified.

A trivial example is any sort of feature that lets a user save something and get it back later.

That obviously implies persistence, since there is no way to guarantee that an instance of the program, a thread, or a session or whatever is guaranteed to be live throughout that arbitrary time period. You have to take the data ‘now’, then return it ‘later’.

But it can get worse. If the data spans different resources during persistence, it requires transaction integrity. If it is dependent on external data, it involves synchronization. If it's a lot of data, it eats through storage and requires optimizations. To implement one simple feature may require resolving dozens of uncertainties or worse.

So it comes down to knowing that there are a massive number of things you don’t know or don’t really understand, then deciding which of those you must understand in order to get out the next release. Once you fill in those details, the others either reveal themselves or end up as some risk that you’ve decided is acceptable.

We’re forced to gamble sometimes; the alternative is a potentially bottomless pit of personalities and politics. Some details you can’t pull from the environment; some decisions get hobbled by personalities. For these uncertainties, you randomly pick an answer, shove it out into the world to see what happens, then pay attention to the feedback in order to really resolve it. That is the shortest path.

Dealing with the tornado of uncertainties whirling around you is what makes software development difficult. If you try to pretend that you are not caught up in this, the denial will only make it worse. If you accept that, unfortunately, this is the way it is right now, then at least you can adopt strategies that are more likely to produce better outcomes. You might not know, but that is fine; you still have to get the work done as well as possible.

Properties

Code is a set of instructions for a computer to follow.

But code also has additional properties, beyond its runtime behaviour. There are lots of them.

A simple one is readability. Some code is cryptic; it takes a lot of investigation to figure out what it will do. Some code is self-explanatory. It’s written in a clear style so that most programmers can quickly figure out what it will do. It has extra comments added to fill in any missing explanations.

Another one is idempotent. The first time you call that code, something changes. All subsequent calls with the same inputs won't change it again. It’s a useful property to protect against unintentional repeated calls.

Another one is stateless. There are no internal variables that persist between calls. All the outputs are the results of pure computations on the inputs.

Optimization is also a property. Given the same inputs, some variations on the code will produce the expected outputs with less resources. Common resources include memory, CPU, GPU, disk, and network. This code usually exploits deeper dependencies within the data or the environment to get to the results faster.

Secure is a property too. Some code can be triggered by anyone; some code only by a strictly controlled small set of fully identified people.

Generalization is another interesting property. Within the code is considerable flexibility to accept a full range of different inputs and to produce a larger range of outputs. Instead of having dozens or hundreds of versions of nearly identical code, there is one version with a more complex flow that works its way through the different sets of computations.

Its cousin is abstraction. The code isn’t specific in its instructions. It pulls back from the actual logic to solve the problem at a higher conceptual level. Then there is often a way to bind that output back to the specifics, so the results end up the same. Its strength is that it is far less code than just grinding it out by brute force.

Safety is a property too. The author expected problems like bugs or bad data and added additional logic to detect, prevent, or correct the issues. The code has been explicitly constructed to prevent these bad outcomes. This is often called self-defensive as well.

Extensible is another less common one. The code is structured in a way to help ensure that future changes are less cognitively demanding. This often employs organizational techniques like symmetry or table driving. Adding new logic is usually just duplicating a few lines and making fairly safe little changes. Future changes take a few minutes.

While these properties don’t change the outputs of the code when it runs, they do strengthen its usability. They make deploying and using the code far less risky. They head off common problems, and make the development and operations of the code less stressful.

Most of them don’t require a lot of extra work, and all of them save a lot of time later in diagnosing and dealing with the usual problems. The fewer of these properties the code has, the more fragile it is when used. It’s important to add these in where they will enhance the development and usage of the code. Just calculating outputs with overly fragile code is not fully solving the problems.

Methodology

My very first experiences with different methodologies for software development date back to the early nineties. I’ve lived through some heavyweight ones over the decades, including those in healthcare. 

Realistically, we need methodologies, even if we don't like to be confined by them. Too many loose strings in a project will eventually tangle.

Which is why I liked the emerging lightweight ideas at the turn of the century. But I found that over time their intent turned away from what I thought was reasonable.

So, I like to propose my own methodology. I guess I’ll call it “Adaptive”.

I realize that the word ‘adaptive’ is already taken for a specific type of ‘project management’, but this methodology is somewhat aligned to that, so they are complementary, while still very different. This is about the act of building complex things, not about the show that surrounds it.

The methodology starts as lightweight, but as quality or funding constraints are needed, it gets heavier. So, it’s almost weightless if you need to whip out a demo, but it’s pretty darn heavy if the code will end up in a medical device.

It values quality over quantity. That is, the intent of a software development project is not to create a mass of code; rather, it is to get the best code you can to solve the problem as precisely as you can. If the solution needs 150K to work properly, then the closer you get to that size, with enough quality to keep it useful, the better it is going.

It values organization over speed. No matter what you create, if it ends up in a giant pile of disorganized stuff, it is useless. Organization is, at minimum, having a place for everything and then putting everything in its place. The development and operations of the solution both need to be kept organized. A big part of organization is knowing where to put things, so there needs to be enough effort devoted to that to allow lots of people to know where things go.

It values engineering over evolution. That is, if there is a state of the art for implementing some specific part of the solution, then the people involved will go out and learn more about those prior efforts. They won’t reinvent the wheel just to save a little bit of time. Crude is not good. They know that the knowledge to do the work is out there, and they’ll attempt to find it first. Once they know, they are free to bend it to their will, or extend it, or even go off in a different direction, but first they have to know.

It values fit over ease of effort. If there are two ways to implement something, a simple one and a complex one, the best implementation isn’t driven by the technology or by the developers' preferences; rather, it is driven by which choice makes the solution the best fit for the users. It’s from their perspective, not the people building the solution. The point of software is that it gets used, so degrading its usefulness to save a little time or concentration doesn’t make sense. What is done is always done to make the users' lives better, even if it is hard and painful for the developers.

As for the specifics, since there are two main axes to the effort, there are two main roles for any project.

On the solution side, there is a lead developer. An expert in software development. They are fully responsible for the development of the software. They have a long, extensive background in building stuff that was complicated enough to act as a reasonable solution for similar problems. It’s not about energy or intent, but rather it is about having been through work in the past that is close enough to what needs to be done now, so that they will make the best choices for the project to keep it from derailing.

On the problem side, there is a domain champion. An expert in the domain problems. They are the person who is managing all aspects of the development from the outside and has considerable expertise in the problem itself. In an enterprise, they are at least a PM / BA combination with actual real-world experience in the business. They have spent time in the domain trenches. They have to have fairly deep knowledge in the domain and its practices, since this is where the details get difficult. They handle the external issues.

The domain expert is responsible for and the expert on all aspects of the problem. They know what the related problems are; they understand them. They have some idea of how to correctly deal with aspects of the problem, but they are not experts in solving it. So they know what the problem is, and have some ideas about how they think a computer might help deal with those issues. They control or have raised money to get the problem solved.

The developer is responsible for and in control of all aspects of the solution. The internals of development. If the solution does not correctly solve the problems, it is their fault. They know how to build it, how to deploy it, and how to keep it running over time. They understand how well it actually fits into the problem space. That is, they can’t just build a generic app and claim it ‘could’ solve the problems; they have to build something that ‘does’ solve the problems, so they can’t ignore understanding the problems themselves.

In that sense, the domain expert sets out a series of features they would like to see in the solution, while the software expert implements these as functionality. It’s a partnership. The software expert understands a lot of the nature of the problems that drive the solution, while the domain expert understands the limits and costs of technologies. But the domain expert has the final word on all domain issues, and the software expert has the final word on all technical ones.

The domain expert can say “your proposed solution is not actually solving our problems”, and the software expert can say “at best we can only approximate a solution to that given problem”. The path forward lies between these two points.

As for specific development processes, it is incumbent on the software expert to keep in constant communication with their own development team. How they choose to do that is up to them. Maybe it’s frequent meetings, or one-on-ones, or some combination. These are internal development product conversations; they should be private away from the domain conversations. What’s important is that they are constantly and actively communicating. The development team cannot be left in the dark, and if they are stuck on something, they need to raise it to the lead immediately.

If the internal project communication breaks, the project will derail.

The software expert and the domain expert are always working together. Neither the problem aspects nor the solution aspects have priority over each other. The idea is that they both need to be balanced. Part of the problem is often ‘time’. If a solution arrives too late, that is a big deal. But, unfortunately, since most development work is iterative over years of effort, prioritizing the correct order of the development work is on the software side. This is a contradiction that both sides have to resolve. It is necessary to avoid wasting time doing the wrong work at the wrong time, while still being timely. So, the two experts are in constant negotiation.

Small ambiguities or misunderstood domain and technical issues pop up on a regular basis. It’s normal. The sooner they are pushed over the fence, the better. While this methodology is not reactive, it does prioritize timeliness; but both sides should try to get ahead of the ball wherever possible. That is, the work may suddenly need to change in response to the ground shifting. It is not appropriate to sit and wait on a showstopper, for example, as the effort going on in the meantime may just become makework.

In very small projects, the lead developer might also be the domain expert, but only if they actually have the background to know enough about the domain. That is, they can’t be a faux domain expert; they have to be a real one. They don’t have to know the total full width of the domain, but they do have to know the width of the domain problems they are intending to solve.

The converse is not true. A domain expert with no significant technical background, given the scale of the project, cannot act as the lead developer. Whacking out some Excel spreadsheets or demos does not qualify someone to run a medium-sized development; the consequences will be disastrous.

A project gets started with the intent to solve a problem. Thus, the domain expert is there from day one. Once the project is in motion, however, the technical issues tend to slowly dominate. Most often, it is difficult to prioritize which features get done first, given whatever the limited time constraints.

Once there is an existing codebase, it has a number of deficiencies, and there is also some desire to grow certain parts of the solution. It’s a bit easier. But the effort slows down, as the work itself should get better; more precise.

There is always a long-term plan, as there is a defined problem, and an expectation that a specific collection of features will act as a reasonable solution to solve them. The initial round of development may only be scheduled for six months, but the long-term plan may extend out five or more years. Both should be discussed; both should be on the table. The value in any software solution comes not from its existence, but from its ability to be there, over and over again, as you need it. That is, even if the initial version is crude, if it does have value, it will gradually evolve into something significantly more complex. If it doesn’t have that value, it was probably a waste of time.

That there is both a short-term and a long-term for all projects is important to keeping the effort organized and to ensuring that the work overall is as effective as it can be. Which is why any sort of work compression like ‘reuse’ or ‘abstraction’ is critical. Once some work is completed, you’ll need to leverage it later, not redo it.

Any and all codebases will eventually reach a maturity point where the effort to extend them is so slow and painful that the life of the codebase is effectively frozen. As well, all codebases need constant maintenance, since the underlying technology is always changing at a crazy rate. If the work gets into that ugly ball of mud stage too early, then it will suffer a premature death. It will waste a lot of money; cost way more. Obviously, for many little demos this is not important, but for any medium-sized system or larger that lots of people rely on, it is highly damaging. If a line of business is reliant on the software, it is extraordinarily destructive.

At the core, though, writing software for the sake of writing software may be an interesting performance art, but it is not the intent of any of these larger development efforts. For these, the software is written explicitly as a solution to some real-world problems. So the code that goes into that solution must fully solve the problems and match their complexity. You shouldn’t over-engineer, but you also shouldn’t oversimplify.

Development projects are inherently iterative. That is, it is a never-ending discrete set of development efforts that can go on for years, or even decades. The frequency of demos and releases is driven by an external timetable and is often influenced by issues such as confidence and trust, particularly in the early stages of development. This means that the length of time for each round of development is variable and shifts. On the one hand, if the release periods are too short, there is more work involved because work is done out of sequence and more likely to be disorganized, but if they are too long, the funding or interest may fade. So, there is no perfect immutable length of time for each iteration. For big work, longer is better; for maintaining trust, shorter is better. On top of this, most funding comes with significant strings, so there is some minimal amount of tracking that is also necessary to maintain confidence. This is not extra work; both the technical and domain sides need to be confident that the big expense of development is worth the risk.

But it’s worth noting that excessive tracking will gum up the effort and the morale. Nobody likes being micromanaged, and any attempt to track the time for each and every little task is a classically bad form. The technology lead needs to keep presenting evidence that the work is progressing as expected, and they need to be diligent about raising any potential show stoppers early and often. The domain expert needs significant patience and to understand that even with the best of specifications, development teams, and technologies, there are still a great deal of uncertainties that may cause significant disruptions, particularly if the pace of development has been rushed.

Overly frequent tiny releases are both expensive and disruptive to the users, so they are not a good thing. People don’t need a new version of an app with just 200 lines of new code to fix minor, mostly unencountered problems. Unless it is an emergency, new releases should at minimum provide revised functionality or infrastructure shifts. That is, most of the time, a user should see some obvious changes in each release. If it’s not visible to them, and they have not been affected, it is just a disruption, not a positive.

Some iterations should always be pure cleanup. The bane of software projects is disorganization. As the project grows, its size invalidates earlier organization efforts. So, even regular consistent cleanup attempts will not be enough to keep the code from reaching the end of its life. Thus, organization itself is both the primary feature and main functionality of every software project. If it’s not there, then the value of the effort is extraordinarily low.

Even for the best lead developers and domain experts, there will be plenty of unknown unknowns. Which means that throughout the entire life of the project, there will be new and better information coming in which needs to be acted on. That is, while we know the technology itself is always changing, the understanding of the domain problems is often in flux as well. Managing change is essentially part of managing the development complexity of any software. Expect change, embrace change, and try to spend as little time as possible fighting about when those changes get reflected properly in the software.

As well, everyone involved in any software development project will grow. They will learn more, they will understand more, and many of the details will emerge. Even for late-stage development where the domain is nearly fully covered, the depth of what is understood is bottomless. Working on software is an exercise in constant learning; it never ends.

In some development efforts, there are significant trust issues. When this is the case, then all major decisions need to be papered, and they need sign-offs. If there is trust, particularly between the lead developer and the domain expert, then it can be bypassed. But if neither side insists on sign-offs, then both are equally guilty later when there is a disagreement. If the sign-off isn’t papered, it can be verbal. If it didn’t happen, then one side moved forward without informing the other. So, you can work on a verbal set of features, functionality, specifications, design, etc. only if both sides are in full trust mode. But if it results in a disruptive disagreement, the weight has to change going forward, and the decisions have to be written out in an email, a document, or any other medium that allows for one party to sign off on the terms of the effort.

It is trust that allows a methodology to be lightweight. If that is broken, it can no longer be lightweight and needs to be as heavy as the conflicts that stand in between the communications. That is obviously costly in time, so trying to stay as lightweight as possible is more effective.

It should also be noted that quality comes from not rushing through the work. So, if enhanced quality is a part of the requirements for solving the problem, then it necessarily slows down the effort. Basic quality for a rushed software project is a pretty low bar. Somewhere around expecting monthly big failures, if not weekly. If that is too much, the development effort needs to be slower; more tracking or oversight will only make the issues worse.

Quality control is an essential part of all software development. Since coding is highly mistake-prone, it is critical to ensure that all code is always properly tested before getting into releases. No untested code should ever end up in the operations environment; that is an avoidable and a seriously bad mistake.

The degree and type of testing depend heavily on the required quality of the effort and the time available for building it. Some code needs incredibly intense testing; some just needs a light amount of smoke testing. That is driven more by the problem domain, not the technical one. That is, writing an app for entertainment may not need much testing, but writing one to be included in a medical device, airplane, or any other strict domain needs a lot and many different types of testing.

So, as well as tracking and sign-offs, heavyweight processes often involve adding strong one-to-one relationships between the specified functionality and its testing in order to verify that it was implemented as described. That is, it ensures that the development process did not accidentally or intentionally deviate from what was specified and is known to be needed for the problem. It also ensures that there is no poorly tested functionality stuffed into the codebase.

In many cases, if the software can glitch with low impact, and the timeline is super tight, the testing can be minimal. But if that results in significant runtime problems, then the testing requirements need to increase. As long as there is any sort of upset from bugs escaping, the testing needs to be enhanced until finally the right level is found. That obviously means that putting effort into automating the testing itself is important, since it is more reliable. But it also means that the type of testing chosen is critical too. If partially testing a few of the components isn’t sufficient, and too many problems are occurring, then it needs more system, regression, or even reference testing.

Testing is not an afterthought; it is tied directly to the problem. That is, some problems need better quality, some are more lenient. It is influenced by the development effort as well. Some teams are better at double-checking their work; some are more reckless. The lead developer should know what the development needs and what the team is capable of. The domain expert should set a minimal level of quality that doesn’t contradict their time expectations.

As a big part of this methodology is in providing solutions that satisfy people’s ability to deal with their problems, part of the trust issues between developers and them is the need to act professionally. That is not just in writing code of reasonable quality or putting in the time to understand the domain perspectives, but also in the words and terminology used to describe the process as well. Because of that, for the parts of the process that developers may or may not need, it is important not to use gamified terminology for them. Some of the earlier lightweight methodologies tried to make the names sound fun or even cute, but that only degrades the way others see the profession. The names and terminology used need to look professional from the outside perspective because that is how developers get people to trust them, which allows them to keep the processes between them and stakeholders lightweight, which we know is at least a little less stressful for developers.

In summary, if a development project is adaptive, you should be able to easily identify the domain expert and the lead developer. Both of them should know what they are responsible for, and be good at avoiding stepping over those lines. As the issues have arisen, more heavyweight processes have been added, but the processes are as lightweight as possible. Both the problem descriptions and the code are organized; you can look through the details and see that they have been cataloged and handled appropriately. The code is always tested enough, and there are few operational issues. Whatever higher-weighted processes like some tracking are there, they are well organized and are not choking out the momentum. The train is on the tracks and moving forward at the best pace it can given the life phase of the project.

So for developers, the idea is that you have to know what the long-term solution is that you are building, while you have to keep adapting to the short-term disruptions that you encounter along the way. It’s trying to be proactive, while not letting the reactive issues knock everything too far off course. It's knowing that there is at least one person who really understands the problem you are trying to solve and another who really understands utilizing technology to really solve those problems, and that they rarely are the same person. It’s not over-engineered solutions that veered way off course, nor overly reactive little partial solutions that are piled together into a mess. That the long-term anchor keeps the short-term responses in line to extend out the lifespan of the effort, and that that is the most cost-effective way to proceed.

Predictions

Being able to predict what’s coming up in the future allows you to make better decisions right now. Better decisions will give you more chances to actually change the current direction rather than just getting caught up in the flow.

The first part of enhancing predictions is to accept that there is only one objective reality. Although it surrounds all of us, we only ever see a tiny sliver of it. Because of that, it is not uncommon for people to believe that their sliver is somehow personalized just for them. It is not. We might sometimes perceive the world as somewhat random, but that’s only because we are caught by surprise.

The next part is to understand that all things follow a trajectory. They will continue on their merry way unless they are interrupted by something unusual. From our perspective, reality is an informal system; all sorts of surprising things can occur. It is constrained, but these constraints are beyond our ability to formalize them. Although this allows for a lot of wiggle room on what might actually happen, it’s still constrained. Some things just can’t happen; they violate some very deep underlying rules about how the physics of the universe will play out.

In that sense, there are plenty of things that are possible, but also highly unlikely. And there are plenty of things that are absolutely impossible, although we might not have full certainty that they are not just unlikely.

So if we observe any phenomena long enough, while we can’t discount that surprising things may occur, if they don’t, then it will follow the trajectory that it has been on already. That is, things will mostly continue as expected based on their past behaviour.

That can be a bit tricky though, if the past behaviour has a rather complicated path through a very long period of time. That’s a lot of complexity to absorb.

Thus, to get a reasonable prediction, you need to observe the actual path through a context that is large enough to include all of the non-surprising but significant influences. Pull back far enough to include all of the dominant variations and influences, watch the movement, and then sketch out the default path forward. If nothing unusual throws it off course, you know where it is headed.

If you are surprised, it could just be an extremely rare event, or that your context was just not large enough to be accurate. Most often, it was too narrow a context.

In that sense, being able to predict lots of things around you is an indication that your knowledge extends far beyond your own sliver. You can learn to predict larger phenomena or farther out in the future as you acquire a deeper understanding of reality. They are correlated.

In the general sense, getting better at predictions will make your actions go smoother. You can leave a little room for the informality, but mostly get the results that you desire. It also helps with insurance. You might not be able to precisely predict infrequent events, but at least you can plan for them in order to mitigate their disruptions. This is a better strategy than just being purely reactive, in that it prepares you to better navigate any obstacles while keeping you aligned to a longer-term goal. When you can anticipate your bad days, you can prepare enough in advance to make them easier to deal with.

Hedging

A long time ago, I remember someone trying to convince me that sufficient testing would prevent ‘all’ bugs. All you need are tests.

That’s never been the case for me, even after all of these decades. 

At the very best, with a gifted team, we were super careful, had strict standards, and hyper-intense reviews, but we still had one bug escape. Worse, though, was that the effort took considerably longer than most people would even guess. It was the only time that I’ve ever not felt rushed during development. So it’s rare.

Ever since then, the default for me has been to anticipate that we won’t have enough time and that the code will be half-baked at best.

That’s okay if you expect it, but it changes your practices and habits. You learn to hedge any and all bets, always.

For example, you always assume there will be lots of bugs and make sure you’ve scheduled in enough time to find and fix them. If you track it, you can often get a sort of accurate guesstimate on how many to expect or at least an upper and lower limit.

You also want scaffolding. That is code that indirectly asserts, verifies, stops, counts, or in any other way validates the incoming data or calling sequences. You have it turned on during development, on for some testing, and off for production.

Full end-to-end system testing is always the best, but unit testing works for overly complex components. There is never a one-type-of-testing-that-finds-all bullet. Testing is expensive; you have to match the different types based on your time and quality requirements.

You want good logs and lots of auditing. From the logs, you should be able to see any and all internal state changes. While having one is best, they do get added all over the place. Logs are the main way to fully resolve real production issues.

For auditing, you need to know who did what. For some domains, a few weeks would suffice, but for some, you need years or decades worth of it.

You need a fast flag switcher. It lets you swap out functionality on the fly in production. It’s for extreme emergencies only, but time compression makes these far more likely.

You want to table-drive every piece of code that involves high uncertainties. That’s the most flexible arrangement possible, and it tends to be resilient against update mistakes.

You need two release processes. The right way and the fast way. You should never use the fast way unless you have no choice.

You want as much reuse as you have the time to get. It pays for itself a bit later, but by then you’ll really appreciate it. Particularly if you have to whip out some revised functionality, and you can do that nearly instantaneously with battle-hardened code. Way better than doing the bug dance again.

You should always be suspicious of any code that you depend on but don’t understand or didn’t write. It tends to bite you when you least expect it. The more you learn about how it works, the better things will go. If you can do it properly yourself, your life will be easier.

You want to assume at any time that you only have half of the picture, and that it is about to change. So, hedging some code with expected variability is more than reasonable.

If you really understand why the code solves a specific problem, you can vet it as you write it. If it's all Greek to you, it's more likely that it will need excessive changes in the future. It often gets lost in translation. You can factor it to make it easier to change.

Initially, you might get praise for grinding out tons of static brute force code, but that quickly changes as the problems and the high costs of fixing them emerge. It’s better if they accuse you of being slow, but later the drama just doesn’t happen. They’ll catch up eventually.

At all times, you just know that it's going to get difficult and somebody somewhere will always be upset. You shouldn’t develop a lack of empathy; instead, you should set up the game to quickly respond with increased precision. The second it's discovered to be wrong, you’re right on top of the fix. That gives you a much better reputation than if you shrug angrily and say, “I don’t know”.

Data Stitching

If you collect some data with respect to a specific context, when there are significant changes in that context, the original data no longer fits seamlessly with any newly collected data.

One way to handle this is to change all of the old data, updating it to the new changes. This works, but it could take quite a while in production and can be extremely risky. A change in massive data can take days or even weeks to run. A small mistake could effectively destroy the old data. 

The other way to handle this is at ‘presentation’. You split the computations, doing something slightly different for the older data.

All you need is a precise ‘line’ between them to test for. Then, to keep the code sane, you make both sides polymorphic and hide the test. When some code asks for the final data, if that request crosses the line, it gets handled correctly.

The way you don’t want to do this is by hardcoding the test somewhere high. Doing this once may not cause too much damage, but if the context has changed once already, it is highly likely to change again. A few times is really ugly; more than that is a classic bug sinkhole.

The overall idea is to always preserve what you collect and stitch it together with polymorphism. Then you can double-check it for correctness, so you have the best possible options for getting it right, and you don’t leave around landmines for later efforts.

For anyone worried about performance, presentations are almost always fixed-sized small data sets; the extra costs of the tests and the polymorphism are usually hardly noticeable. If there is some other type of dynamic large-scale bulk interface over the data, then it’s an architectural problem, not a data one. That is, if the performance is really noticeable, then you’re probably doing something else in the software that you shouldn’t be doing.

Data stitching is often avoided, but when done reasonably, it ensures that the data appears correctly and that it is straightforward to fix later if it doesn’t. Depending on the data, it often requires a domain expert to verify, which usually happens very late in the effort, so it's best to expect last-minute changes.

Language Ideas

Decades ago, our project had a convention where we never directly used ANSI C primitive types, unless it was trivial in a loop. But for any and all other variables, everything had to be typedef’d explicitly. Everything.

Some people would see that as excessive and way over the top, but it actually turned out to be a very good habit that helps with getting really high quality.

If you went to call something, you’d need an X, and that would force you to go into the code to find out how to create one. If the code is all ints, strings, doubles, etc., then people skip that effort and just find some hacky way to kludge the value they need. But now you can’t.

I’ve often thought that I’d like to see a language that has zero primitive types for variables. None. If you need a variable type, you have to declare it.

But to add to the fun, there aren’t even open types like ‘int’. If you needed an integer, you’d create the type, but as you did, you’d have to explicitly specify the range.

For example:

type Counter: integer 0..INTMAX

It would be more fun for strings, as they are not open either:

type Token: string [a-zA-Z]*

Each would be constrained by an RE state machine.

What I think would happen, maybe naturally, is that programmers would get bored with creating a million different types and a billion conversions between them. So, instead, they’d start to pack things together into larger structures all the time.

type UserName:

First: string [a-zA-Z]*

Last: string [a-zA-Z]*

And for any discerning programmers out there who suspect that using just alphabetical characters is not wide enough to handle all of these types of names across the planet, staring at this declaration would trigger a need to further investigate and correct the model.

But the trick would be that in correcting the model, in that one place, it would also be corrected everywhere else. So the benefit would be that all of the validation code, both at the interface and at persistence, that would have needed to have also been updated when the model changes, would actually not need to be changed, since it is all implicit in the language. A recompile would do the trick.

I’d go even further, though. I’d not have any primitive data structures in the language’s library; they’d all be baked into the language too.

For example:

type UserList linkedlist:

Entries: User all

Sorted by User.UserName.Last

If it’s starting to smell like SQL, I apologize. The type semantics is essentially declarative, but the rest of the language syntax would be imperative, with some extended paradigm on top. Probably more like Golang than any of the other OO or FP variants.

The point, though, is that you would use the type mechanism to build up larger and larger data structures, and all of the base ones, like lists, trees, dags, graphs, and even hypergraphs, would already be there. For fun, you’d have all others like stacks, queues, and pagodas. All with various options, but to use more advanced ones, you would have to explicitly declare that they were not just trivial implementations.

That would ensure that anyone reading would not have to infer anything about the underlying implementations. The crudest thing would always be the default thing. Any special ability, property, enhancement, or optimization would have to be explicitly mentioned. It would minimize confusion.It would aid in avoiding bloat.

Over the decades, the trend in languages was to make them more dynamic and to minimize expressions. The result of adding those freedoms in the hands of a disciplined programmer was great, but the abuses heavily outweighed the elegant examples. Learning from that, I think we should find more ways to restrict the freedoms, but not cripple the expressiveness. That is, you’re not forced to type in reams of boring boilerplate, but you also can’t write code that is cryptic enough to win the obfuscated C code contest either. The code you have to write always makes its intent clear. Readability is part of the language.

The Craft of Programming

Back in the eighties, when I was a student, there were only two choices: you used ‘vi’ or you used ‘emacs’.

I picked vi, my roommate picked emacs. The stuff he did with his editor was way cooler, but over the decades, vi has pretty much been everywhere I have stumbled.

I assumed it would fall out of fashion, but it persisted. Vim and all of those embeddings in the IDEs keep it alive.

I read something about how someone thought that using a complicated editor like vi was unnecessary. They implied that it was a waste of time to learn lots of its features. There were so many other things to learn, why not just use a lame mouse-based editor and move on?

For me, the answer is that it is all about craft.

When you build a large program, it’s a lot more than just throwing together the thousands of instructions necessary to tell the computer what to do. The code is just the output, but if you are hasty and careless on your way there, the whole thing ends up as a house of cards; any little breeze will knock it over. That is, it isn’t just coming up with some code; it is coming up with good, solid code that is readable and will survive all of the craziness that the world throws at it.

I’d rather have 150K lines of something that is super tight, abstract, solid, dependable than have 1M lines of stuff that could be served in an Italian restaurant. A whole lot of bad code is just a few steps closer to an apocalypse than a system.

So, it’s not about how much code you can create or how fast you can create it. Instead, it is about how carefully you embed precision into that code. Not for tomorrow, but for its entire lifespan. You want strong solutions to the real problems you need to solve, and you want them encoded in a way that you can keep them moving forward and leverage them for all sorts of related problems.

In the act of doing that sometimes tedious, thoughtful, and precise work, you want to develop habits that are more likely to produce high-quality output. Quality matters, not quantity.

Getting back to vi, if you are concerned about your code, you care very deeply about how it is ordered in any sort of file. It should not be random; you have come up with some consistent reason why one function is placed ahead of the others. In the daily grind, you have probably violated that order accidentally a few times. In most editors, you can use the mouse to highlight a chunk of code, hit ctrl-X to cut it, then ctrl-V to paste it back where it should go. It works, but it’s a touch haphazard. You could be interrupted, or get confused, or any other disruption that might interfere with the outcome you desire. In vi, you ‘mark’ the start, then ‘mark’ the end, then issue a few commands to move the code into place. It’s a small difference, but it’s transactional integrity. It either happens or it does not. If you get interrupted, it doesn’t matter; nothing is lost, nothing is messed up. If you're unsure about where to place the code, you can scroll around with the keyboard until you are sure. It isn’t any slower than using the mouse, but it is a whole lot more precise.

That’s a trivial example, for sure, but it’s just one of those things that you can do with a good tool when you’ve invested the time to understand it fully. Being a good programmer, a reliable one that is always there in a crisis, is the act of investing a whole lot of time in honing these acts of precision. It’s not that you can whack out a whole lot of code that you barely understand; it is that when you are given a piece of work, you can ensure that that work is done satisfactorily, even in environments that are difficult.

That is the essence of good programmers. They are the ones that everybody leans on, especially when it gets difficult, because they know that they’ll be there and make sure it is at least as good as it can get right now. They are reliable. Not always efficient, never fast, but always making sure that the work going out the door is solid.

We’ve always had the expression “garbage in, garbage out” in reference to the necessity to have good input data, but it also applies to code. If you get a group of people that churn millions of lines of questionable code, it’s just a field of landmines waiting to go off. If you get a group of people that solve complex problems with the least amount of code they can, carefully, then it doesn’t take that long before that quality of work starts to pay for itself. The craft of programming isn’t coding; the craft is in providing solutions, it just so happens that most of them involve code in some way.

Structureless

The most common mistake I have seen in big ugly balls of mud is to try to capture data without enough structure.

Chopping up some incoming data into a lot of little strongly typed fields is a pain. Sometimes it seems like an unwarranted pain. You may need to get a mailing address. Why not just give the user a big textbox to fill in?

The problem isn’t that the users can’t carefully type in the text with appropriate structure; it’s that sometimes they won’t.

And the code to parse unstructured text is stupidly complicated. They can type anything; you have to be able to apply some type of structure to each and every possible variation. Since there are an infinite number of those, you are going to lose.

You can add a ton of validation, but if it’s not rigorous within a fully interactive interface, then if there is any tiny way to bypass it, it was a total waste of effort.

You could scream at the users and make them format it correctly, but as time wears on, unless you keep screaming, eventually that practice will degrade. It will just delay the inevitable.

Which is to say that in a computer, a big box full of text is absolutely nothing more than a big box full of text. It has no other use or value. That makes it useful for someone putting a “personal note” somewhere in a report, or something like that, but there is nothing beyond that. It’s not really data; it’s just an extra external comment of some type.

If what you intend to do is collect data with a very specific structure, you should never subvert text boxes to do that for you. It’s not a shortcut; you haven’t “figured it out”, you just made a very bad mistake. And sadly, doing it right wasn’t that much more time.

Likewise, if you are using some questionable software, and it has a lot of text boxes, so you come up with a clever idea about how to put structured data into those, just because you can’t or don’t know how to change the program, then it isn’t brilliant. It’s just a sloppy, hacky way of trying to get around some other code. One step worse than duct tape.

Data is not data without structure. Untyped data is not data. A big string of characters is a mess. Mostly.

If it originated under the strict control of some code somewhere, and the pathway was closed and guarded, then sure, two programs can use strings and complicated parsers to pass data back and forth. It starts with structure, is transported in an unstructured container, and then it is restructured again. But if the pathway is open or one end of that game is a human, then a big string of stuff is just potential garbage. At some point, usually in the not-too-distant future, someone will fill that string with a problem, and it will end up wasting a lot of time.

Do note that there are subtle variations on this. A user might use one program to render XML that they upload manually into another. In a case like this, the human isn’t the endpoint; they are just the transfer medium. It's an open pathway, but still just between two programs. And you can close down that pathway by only strictly accepting XML with a very specific schema. Since you can verify that and reject garbage as necessary, it becomes closed.

The foundation of all software systems is to collect data. The full and complete structure of that data, as it relates to the real and digital worlds, is an essential part of that data.

There are extraordinary times when it makes sense to only sub-model some external data and live with the consequences of that choice. But the default should always be that if the program needs some data, it is key to its usage and computations, then the data needs to be fully and correctly modelled. That is, it needs the right structures, for example, not shoving a tree into a list, and it needs all of the individual fields in that data to be very strongly typed. It needs this in order to make the correct choices with its instructions based on what is actually there. It can’t be vague or ambiguous; computers are not smart enough to grok external context. They can only act on the very specific information that they have.

It’s also worth noting that while parsing may look easy, and in some tiny instances, it is not that complicated, it should always be considered a hard problem to properly solve. As such, if you can avoid parsing, or at least push it to a human somewhere, then you will get far fewer bugs and the code will be more likely to behave as expected. If you do have to venture into parsing, then it is one of the coding areas where reading a boatload of stuff in advance will pay off huge dividends. Trial and error with parsing is a massive bug generator.

Software Systems

I use the term ‘software system’ loosely. I usually intend it to mean: all of the boundaries for a set of related solutions that have been or will be implemented with software.

In that sense, it is less about the technical parts of the ‘system’ and more about how they all come together to help people.

I do this mostly because I tend to visualize a ‘problem space’ as a flat 2D terrain. It is a convenient oversimplification. It is a big, wide, open, empty field of grass which spans over related problems.

When I am doing greenfield work, I see the start as picking one spot in that field. You start there, building up enough structure to be useful. First, you lay down some common foundations, then you start adding in functionality that implements the features you know will help solve the problem.

As you do this, people will see the effort and start making suggestions. Some will want to go off in one direction, while others will prioritize the opposite way.

The trick to keeping it all as usable as possible is to slowly expand out your borders, but not in too many directions all at once.

Someone once told me that for software, you should never pick a path unless you are willing to walk it. From this perspective, it usually means that you won’t expand into another area in the field haphazardly. If you do choose to go there, it needs to be done correctly. That is, adding a few really good, solid features is way better than adding a million lame ones.

The same is true for the data. If you need new data, you add it carefully, properly structured, or not at all.

Overall, though, you start at one specific spot and keep growing. If it’s a good set of programs and people find it valuable, you’ll probably be at it for years, if not decades. So, it’s really crucial to its lifespan that the work you do in the very early days is as good as it can be. It needs to be neat, tidy, organized, and carefully thought out.

With that in mind, calling the current work and any of the rather obvious future work a ‘system’ works quite well. The system isn’t the code, but rather it is all of the territory that the code is trying to cover at some point. You might build a system for handling the account problems in a large corporation, for example. There might be lots of included pieces, and even some nearly stand-alone sub-systems, but they are all trying to fit together to deal with the same problems.

So, it’s similar to seeing the forest through the trees. The boundaries of the expected work are the system, but the system may not stretch right up to those boundaries yet.

From this view, it makes it easier to understand a bottom-up implementation. You might not know all of the features that people will ask for, but you should have a reasonable sense of the territory you are covering right now. Lots of that territory is similar, so building reusable components and engines will really help in getting more ground covered at a faster rate.

The classic example is reporting. You know people will need it at some point, so instead of just hardcoding a couple of static examples, it would be better to either offload it to somewhere else as exported data, or write some generic engine that is flexible enough to cover its rather massive width. The trick is not to write a lot of code, but rather to leverage any code you do write to cover the largest parts of that territory. In software, a little foresight goes a long way.

Thinking of a software system this way really helps in making a lot of the implementation decisions. If your field doesn’t cover having a million users, then designing an architecture to support that scale doesn’t make any sense.

More importantly, if you are located in one corner of the field, then trying to expand way over to the other side of a nearby hill doesn’t make a lot of sense either. That’s far enough away that it is clearly another system, another project and thus another codebase.

Building even medium-sized software is surprisingly complicated, so finding ways to frame it nicely really helps with making better decisions. Since time is a precious resource, getting to the right code as quickly as possible is important. Seeing it all as a system occupying some territory in an endless field is a good guide.

Round Holes

A classic expression describes shoving a square peg into a round hole. Basically, you’re matching the wrong part with the wrong location.

I see this often in software architecture. Sometimes I like to use the term impedance mismatch. There is a component that someone is suggesting for use in a different variation of its problem space. It fits badly.

Sometimes the issue is access. There are hard limits on the availability of stacks, libraries, frameworks, tools, and services. In some organizations, they need to be vetted and approved first. This can be very slow, but I still think a good idea; using too many technologies is a nightmare.

In some cases, it is knowledge. People tend to gravitate to the things they already know, so they’ll prefer technologies they’ve used in the past, even if it means forcing them into place. That makes little sense if the architect has that preference, but it’s a different development team that does the construction.

Sometimes it is a misunderstanding. The marketing for the component says it will do everything, perfectly, but the reality is that it is far more than a stretch. Hastily cobbled together features to help sales. Still, the people funding the effort got swayed, so now everyone else is forced to jam weak pieces into the wrong place.

The habit I’ve always encouraged is to look carefully and admit that a round hole is, in fact, just a round hole. 

I don’t start with the square pegs; that is my last step. Not surprisingly, this can frustrate people, in that there might not be any round pegs available. They don’t exist or can’t be used. For me, though, I still want to know that the hole is round, even if I can’t fill it correctly right now, or in some cases, ever.

But if you can do that, and imagine for any bunch of holes what would fit rather perfectly, first, before getting lost in the messiness, it will really help both simplify the effort and get it as good as possible. 

Otherwise, you risk unintentionally creating a Rube Goldberg machine. For people unfamiliar with those machines, they are works of art that are deliberately overcomplicated. Just a collection of mismatching pieces made to do something interesting. They make great entertainment, but are not something that you’d want to have to rely on.

I’ve seen that too often in enterprise architecture, systems built out of odd, mismatching components sloppily glued together into a giant house of cards. That, paired with excessive brute force for the glue, tends to generate an endless amount of support and bug fixing, while never really working correctly. The system exists, but it is just off by enough that it would be better if it didn’t. It’s a time sink. Now, instead of a solution, it is an ill-placed speed bump.

Often, to avoid that fate, I want to just look at the way the data needs to flow around at the high, rather abstract level. 

You need to get the major entities from other sources, persist it all, and then deliver to interfaces, reporting, other systems, etc. If you understand the amount of information, its timeliness, and frequency, you start to get a sense of the minimum pegs you need below it. If you can grok that, then you can start the torturous phase of trying to see what is actually available and whether or not it is close enough to be workable. But if you go the other way and pick the components first, you’ll quickly get lost in gluing together odd parts for no reason.

It’s the same form of thinking that is needed to get good, simple, clean code, too. You have to see it first from a top-down perspective, before trying to build it up from what’s already available. It’s really the only way to keep from getting lost, but also to leverage reuse, encapsulation, etc. You want to know the scope of the problem first, come up with a near-perfect solution, and then map that impossible solution back to things that are possible. It probably sounds a bit crazy to people who can’t see it that way, but it is a perspective that anyone can learn to leverage. A superpower of sorts.

So we can get there with three easy questions. 

• What is the ‘full’ scope of the problem? 
• What would solve this perfectly? 
• What’s available to approximate that perfect solution?

In an enterprise that might be building up a replacement system for tracking some type of inventory or case management. The primary features are pretty well known; the useful secondary ones are findable with a bit of investigation.

Perfection might be a dynamic data store to accommodate wide but slowly changing shallow data. The users need a nice GUI to get at this and keep control. The incoming data is real-time, vibrates occasionally, so a queue would protect it and help with integrity. The system feeds a few others that specialize in other forms of management. It’s always a smallish number of people. It should all run in a managed environment.

This then is the hole that needs to be filled in with whatever technologies are available now, in the future, or can be suitably crafted in a “reasonable” time.

Contrast that with something where the data rarely changes, there are millions of users constantly accessing it, and they are the primary source of the data. It’s a very different hole that likely needs industrial-strength pegs in order to keep it going. It’s not a system running on one or two boxes, but requires a large cluster of machines all cooperating to cope with its huge and variable load. The scale is so large that there is no overlap with that first medium system, so it’s unlikely that they should share any common technologies. It’s more of a star-shaped hole, needs special stuff to fill it.

The converse is also true, in that any of the technologies suitable for the second design would be grossly over-engineered for the first one. You can’t just cherry-pick a few and shove them into place. One is a 2D circle that needs to be painted, the other is a 3D hole that needs to be filled.

In that sense, you learn as much as you can about the full width of the problem, then let your imagination run wild with getting it perfect. With those boundaries in place, you can start picking the fewest number of pieces that come close to filling it. There will be ugliness and rough edges, but you’ve found them early and minimized them, which is the best you can do if you can’t just build it all from the metal to the top.

Versioning

If you start from the premise that a system is just a series of access points into a vast array of computations, then if you accept that there will always be a huge number of changes to this underlying code, you see why this is messy.

All the computer is really doing is taking a bunch of inputs, grinding through computation, then spitting it out. But we often end up changing these computations, sometimes because we had obvious or subtle bugs, sometimes because we’ve acquired new knowledge about how to do the work better, more accurately, or much faster.

At the high level, people interact with all of these access points, apply some variability to them, and then set the computations in motion. It might take a millisecond, an hour, or even a few days. The interaction might be rapid (real-time-ish), or it might just be infrequent. What is important to these people is that they can trust that the computer does the thing that they expect it to do. Trust is the bedrock.

We can skip over any sort of difference; the output is either a blob of text or a pretty little graphic of some type, it’s all just variations on presentation. The text could be typed and structured, which doesn’t matter either.

In order to trust the system (app, program, plugin, etc.), one key property is that the behaviour has to be ‘stable’. It should not change day to day, hour to hour. If you used it yesterday to do something, then you expect that with the same inputs, it will do pretty much the exact same thing (determinism). An added expectation is that if there were changes, then mostly those changes would be adding more features, not changing the old ones.

This is the core of what we call backward compatibility. Most programmers think of it in terms of APIs they are calling that are stable, but really, it is an overall property of the system itself. It is backward compatible if and only if any interactions, code, or humans are fully preserved after endless updates. If it worked ten years ago, it will work exactly the same today.

There is a loose exception for bug fixes, particularly bugs that have rendered the functionality to be useless. These are obviously not expected to be backward compatible, as the old behaviour does not correctly match expectations, so it needs to be changed to something else.

This relation is expressed nicely in using three-digit version numbers.

The last digit is bumped up for 1 or more bug fixes. Going from n.n.10 to n.n.11 means that at least one bug was fixed, maybe a dozen of them.

The second last digit is listed as a minor enhancement, but honestly, it is really just there for added functionality. Nothing else changed, nothing was deleted. So, if n.10.n is bumped up to n.11.n you expect backward compatibility for all of the existing functionality, and there is now some new functionality included.

That leaves the first digit to clearly state that you have broken backward compatibility. It is essentially a flag. There is some change that is major enough that the user is not necessarily going to be able to expect the code to be deterministic. Something big changed, and it will be noticed.

If people were strict in the usage of version numbers, and if they respected the notion that any 0.n.n version was just an initial demo or test instance, then even if the system was under active development for years, if it was backward compatible, then 1.9343523.14 would be a reasonable version number. 9M times new features were added, but the rest is still intact. Lots of stuff has been added, but all of it is backward compatible. The last round of added features needed 14 tries to get the bugs knocked out. All of these should have been in testing.

As it is with user interfaces, it is true for any pure computational dependencies below. Libraries, frameworks, languages, tools, etc. Strict usage of the version numbers is enough to get a very strong sense of both how the development is going and whether the authors even understand backward compatibility.

Probably the most embarrassing self-inflicted mistake a software developer can make is to push out a release that immediately crashes due to an underlying dependency change. If they were doing things reasonably, this would never occur. At minimum, an embarrassing non-backward-compatible library change would get picked up in testing. Untested code should never, ever get into a release. Subtle changes could slip through, but at the bare minimum, the work should have been smoke tested to catch exactly this sort of mistake. But the stronger habit is to only upgrade questionable libraries at the beginning of a long development cycle, while also doing lots of non-destructive refactoring. That is, before dumping in new stuff, you tidy up the junk from the last release and update some of the libraries. Run it a lot yourself until you are sure it is stable, then you go to town to add new stuff.

If the library is any good, and it has done an excellent job at being backward compatible, this is extremely low risk. You can kinda cheat the game sometimes. But if it is some dodgy little thing written by a couple of people as an advertising attempt, then you would have to wrap it in very expensive testing for each and every little thing you’ve used it for. It’s this that makes most libraries not worth integrating, either because the testing is too much work or the risk is just way too high. Reading the code and applying some of its better ideas is more suitable.

Often, you can get a sense of the quality of the library just by looking at the version numbers. For instance, 24.3.2 is a suspicious number if the work is only a couple of years old. They’re not taking backward compatibility very seriously; they are high risk.

It comes across with some of the larger tech stacks, too. If there is a major version bump that fundamentally breaks all backward compatibility, but someone has the newer and older versions haphazardly laid on top of each other, you pretty much know that the confusion caused by being too loose with the versioning is going to cause a lot of chaos that will either waste a lot of your time or result in embarrassing bugs. If the break was wide enough, the new work should really abandon the ‘brand’ of the old work. They are two different things, even if that means it is harder now for the new version to get a lot of traction. Just because you decided to change it doesn’t mean everyone else in the world should change too. Once you’ve committed to a particular set of computations, you have to stay committed and only grow from there. You can’t just pick up and move to some other spot farther away and claim it is the same work; it is not.

Backward compatibility is hard, really hard, which is why everyone loves to cheat the game so much. But it is an essential property of stability, which is necessary for trust. If you want to do a good job providing some complex computations to others, it is going to be hard. There is no way to avoid it. If you do the hard work, then you can communicate it quite clearly with the version numbers. That will let people know that your work is serious.

Feedback

Recently, my blog has been getting a lot more views. Unfortunately, a lot of the incoming fields for these reads are just tagged with ‘Other’.

That tells me that the traffic is not coming from the older established sites I know, like HackerNews, but is either fake traffic or newer sites that I haven’t seen. It would be nice to know which is correct. Are people actually reading these posts?

So, if you are reading this, I’d really appreciate you taking a moment to comment. Anonymous is fine, and since my comments are screened before they are published, feel free to say ‘do not publish’ if you want. A ping is good; mentioning the source helps.

A long time ago, I briefly dreamed of monetizing my writing, but as I realized that the way to do that is to effectively change what I am saying, I decided not to do that.

I’ve always had to be careful not to upset any of my current employers, but beyond that, I write what I know, either from firsthand experience or from conversations with others.

Because of that, and my limited writing style, it’s never been a popular blog, but I still feel, after decades, that I want to get what I understand down somewhere. Maybe people read it, maybe not. It’s okay.

The software development industry varies hugely, so not surprisingly, plenty of other people have had very different experiences in their careers, but I also do suspect that there is way too much propaganda out there that is deliberately trying to mislead people. It’s an immature, messy and often ugly industry.

With all that in mind, if you could take a moment to say ‘Hi,’ at least I’ll know if you really exist or are just a figment of the web’s imagination.

UPDATE: Ok, I got a few responses, which is great. Thanks! Seems like at least some of the traffic is RSS and Atom, which doesn't show up in the stats. It might be those views where I do get a country and browser type, but that still leaves a great deal of traffic as Other. I guess I'll never know if those are real or not.

If anyone has suggestions about future topics, that would be great too. I feel like I am getting too repetitive in my old age :-)