Code is a set of instructions for a computer to follow.
But code also has additional properties, beyond its runtime behaviour. There are lots of them.
A simple one is readability. Some code is cryptic; it takes a lot of investigation to figure out what it will do. Some code is self-explanatory. It’s written in a clear style so that most programmers can quickly figure out what it will do. It has extra comments added to fill in any missing explanations.
Another one is idempotent. The first time you call that code, something changes. All subsequent calls with the same inputs won't change it again. It’s a useful property to protect against unintentional repeated calls.
Another one is stateless. There are no internal variables that persist between calls. All the outputs are the results of pure computations on the inputs.
Optimization is also a property. Given the same inputs, some variations on the code will produce the expected outputs with less resources. Common resources include memory, CPU, GPU, disk, and network. This code usually exploits deeper dependencies within the data or the environment to get to the results faster.
Secure is a property too. Some code can be triggered by anyone; some code only by a strictly controlled small set of fully identified people.
Generalization is another interesting property. Within the code is considerable flexibility to accept a full range of different inputs and to produce a larger range of outputs. Instead of having dozens or hundreds of versions of nearly identical code, there is one version with a more complex flow that works its way through the different sets of computations.
Its cousin is abstraction. The code isn’t specific in its instructions. It pulls back from the actual logic to solve the problem at a higher conceptual level. Then there is often a way to bind that output back to the specifics, so the results end up the same. Its strength is that it is far less code than just grinding it out by brute force.
Safety is a property too. The author expected problems like bugs or bad data and added additional logic to detect, prevent, or correct the issues. The code has been explicitly constructed to prevent these bad outcomes. This is often called self-defensive as well.
Extensible is another less common one. The code is structured in a way to help ensure that future changes are less cognitively demanding. This often employs organizational techniques like symmetry or table driving. Adding new logic is usually just duplicating a few lines and making fairly safe little changes. Future changes take a few minutes.
While these properties don’t change the outputs of the code when it runs, they do strengthen its usability. They make deploying and using the code far less risky. They head off common problems, and make the development and operations of the code less stressful.
Most of them don’t require a lot of extra work, and all of them save a lot of time later in diagnosing and dealing with the usual problems. The fewer of these properties the code has, the more fragile it is when used. It’s important to add these in where they will enhance the development and usage of the code. Just calculating outputs with overly fragile code is not fully solving the problems.
Thursday, August 6, 2026
Thursday, July 30, 2026
Methodology
My very first experiences with different methodologies for software development date back to the early nineties. I’ve lived through some heavyweight ones over the decades, including those in healthcare.
Realistically, we need methodologies, even if we don't like to be confined by them. Too many loose strings in a project will eventually tangle.
So, I like to propose my own methodology. I guess I’ll call it “Adaptive”.
I realize that the word ‘adaptive’ is already taken for a specific type of ‘project management’, but this methodology is somewhat aligned to that, so they are complementary, while still very different. This is about the act of building complex things, not about the show that surrounds it.
The methodology starts as lightweight, but as quality or funding constraints are needed, it gets heavier. So, it’s almost weightless if you need to whip out a demo, but it’s pretty darn heavy if the code will end up in a medical device.
It values quality over quantity. That is, the intent of a software development project is not to create a mass of code; rather, it is to get the best code you can to solve the problem as precisely as you can. If the solution needs 150K to work properly, then the closer you get to that size, with enough quality to keep it useful, the better it is going.
It values organization over speed. No matter what you create, if it ends up in a giant pile of disorganized stuff, it is useless. Organization is, at minimum, having a place for everything and then putting everything in its place. The development and operations of the solution both need to be kept organized. A big part of organization is knowing where to put things, so there needs to be enough effort devoted to that to allow lots of people to know where things go.
It values engineering over evolution. That is, if there is a state of the art for implementing some specific part of the solution, then the people involved will go out and learn more about those prior efforts. They won’t reinvent the wheel just to save a little bit of time. Crude is not good. They know that the knowledge to do the work is out there, and they’ll attempt to find it first. Once they know, they are free to bend it to their will, or extend it, or even go off in a different direction, but first they have to know.
It values fit over ease of effort. If there are two ways to implement something, a simple one and a complex one, the best implementation isn’t driven by the technology or by the developers' preferences; rather, it is driven by which choice makes the solution the best fit for the users. It’s from their perspective, not the people building the solution. The point of software is that it gets used, so degrading its usefulness to save a little time or concentration doesn’t make sense. What is done is always done to make the users' lives better, even if it is hard and painful for the developers.
As for the specifics, since there are two main axes to the effort, there are two main roles for any project.
On the solution side, there is a lead developer. An expert in software development. They are fully responsible for the development of the software. They have a long, extensive background in building stuff that was complicated enough to act as a reasonable solution for similar problems. It’s not about energy or intent, but rather it is about having been through work in the past that is close enough to what needs to be done now, so that they will make the best choices for the project to keep it from derailing.
On the problem side, there is a domain champion. An expert in the domain problems. They are the person who is managing all aspects of the development from the outside and has considerable expertise in the problem itself. In an enterprise, they are at least a PM / BA combination with actual real-world experience in the business. They have spent time in the domain trenches. They have to have fairly deep knowledge in the domain and its practices, since this is where the details get difficult. They handle the external issues.
The domain expert is responsible for and the expert on all aspects of the problem. They know what the related problems are; they understand them. They have some idea of how to correctly deal with aspects of the problem, but they are not experts in solving it. So they know what the problem is, and have some ideas about how they think a computer might help deal with those issues. They control or have raised money to get the problem solved.
The developer is responsible for and in control of all aspects of the solution. The internals of development. If the solution does not correctly solve the problems, it is their fault. They know how to build it, how to deploy it, and how to keep it running over time. They understand how well it actually fits into the problem space. That is, they can’t just build a generic app and claim it ‘could’ solve the problems; they have to build something that ‘does’ solve the problems, so they can’t ignore understanding the problems themselves.
In that sense, the domain expert sets out a series of features they would like to see in the solution, while the software expert implements these as functionality. It’s a partnership. The software expert understands a lot of the nature of the problems that drive the solution, while the domain expert understands the limits and costs of technologies. But the domain expert has the final word on all domain issues, and the software expert has the final word on all technical ones.
The domain expert can say “your proposed solution is not actually solving our problems”, and the software expert can say “at best we can only approximate a solution to that given problem”. The path forward lies between these two points.
As for specific development processes, it is incumbent on the software expert to keep in constant communication with their own development team. How they choose to do that is up to them. Maybe it’s frequent meetings, or one-on-ones, or some combination. These are internal development product conversations; they should be private away from the domain conversations. What’s important is that they are constantly and actively communicating. The development team cannot be left in the dark, and if they are stuck on something, they need to raise it to the lead immediately.
If the internal project communication breaks, the project will derail.
The software expert and the domain expert are always working together. Neither the problem aspects nor the solution aspects have priority over each other. The idea is that they both need to be balanced. Part of the problem is often ‘time’. If a solution arrives too late, that is a big deal. But, unfortunately, since most development work is iterative over years of effort, prioritizing the correct order of the development work is on the software side. This is a contradiction that both sides have to resolve. It is necessary to avoid wasting time doing the wrong work at the wrong time, while still being timely. So, the two experts are in constant negotiation.
Small ambiguities or misunderstood domain and technical issues pop up on a regular basis. It’s normal. The sooner they are pushed over the fence, the better. While this methodology is not reactive, it does prioritize timeliness; but both sides should try to get ahead of the ball wherever possible. That is, the work may suddenly need to change in response to the ground shifting. It is not appropriate to sit and wait on a showstopper, for example, as the effort going on in the meantime may just become makework.
In very small projects, the lead developer might also be the domain expert, but only if they actually have the background to know enough about the domain. That is, they can’t be a faux domain expert; they have to be a real one. They don’t have to know the total full width of the domain, but they do have to know the width of the domain problems they are intending to solve.
The converse is not true. A domain expert with no significant technical background, given the scale of the project, cannot act as the lead developer. Whacking out some Excel spreadsheets or demos does not qualify someone to run a medium-sized development; the consequences will be disastrous.
A project gets started with the intent to solve a problem. Thus, the domain expert is there from day one. Once the project is in motion, however, the technical issues tend to slowly dominate. Most often, it is difficult to prioritize which features get done first, given whatever the limited time constraints.
Once there is an existing codebase, it has a number of deficiencies, and there is also some desire to grow certain parts of the solution. It’s a bit easier. But the effort slows down, as the work itself should get better; more precise.
There is always a long-term plan, as there is a defined problem, and an expectation that a specific collection of features will act as a reasonable solution to solve them. The initial round of development may only be scheduled for six months, but the long-term plan may extend out five or more years. Both should be discussed; both should be on the table. The value in any software solution comes not from its existence, but from its ability to be there, over and over again, as you need it. That is, even if the initial version is crude, if it does have value, it will gradually evolve into something significantly more complex. If it doesn’t have that value, it was probably a waste of time.
That there is both a short-term and a long-term for all projects is important to keeping the effort organized and to ensuring that the work overall is as effective as it can be. Which is why any sort of work compression like ‘reuse’ or ‘abstraction’ is critical. Once some work is completed, you’ll need to leverage it later, not redo it.
Any and all codebases will eventually reach a maturity point where the effort to extend them is so slow and painful that the life of the codebase is effectively frozen. As well, all codebases need constant maintenance, since the underlying technology is always changing at a crazy rate. If the work gets into that ugly ball of mud stage too early, then it will suffer a premature death. It will waste a lot of money; cost way more. Obviously, for many little demos this is not important, but for any medium-sized system or larger that lots of people rely on, it is highly damaging. If a line of business is reliant on the software, it is extraordinarily destructive.
At the core, though, writing software for the sake of writing software may be an interesting performance art, but it is not the intent of any of these larger development efforts. For these, the software is written explicitly as a solution to some real-world problems. So the code that goes into that solution must fully solve the problems and match their complexity. You shouldn’t over-engineer, but you also shouldn’t oversimplify.
Development projects are inherently iterative. That is, it is a never-ending discrete set of development efforts that can go on for years, or even decades. The frequency of demos and releases is driven by an external timetable and is often influenced by issues such as confidence and trust, particularly in the early stages of development. This means that the length of time for each round of development is variable and shifts. On the one hand, if the release periods are too short, there is more work involved because work is done out of sequence and more likely to be disorganized, but if they are too long, the funding or interest may fade. So, there is no perfect immutable length of time for each iteration. For big work, longer is better; for maintaining trust, shorter is better. On top of this, most funding comes with significant strings, so there is some minimal amount of tracking that is also necessary to maintain confidence. This is not extra work; both the technical and domain sides need to be confident that the big expense of development is worth the risk.
But it’s worth noting that excessive tracking will gum up the effort and the morale. Nobody likes being micromanaged, and any attempt to track the time for each and every little task is a classically bad form. The technology lead needs to keep presenting evidence that the work is progressing as expected, and they need to be diligent about raising any potential show stoppers early and often. The domain expert needs significant patience and to understand that even with the best of specifications, development teams, and technologies, there are still a great deal of uncertainties that may cause significant disruptions, particularly if the pace of development has been rushed.
Overly frequent tiny releases are both expensive and disruptive to the users, so they are not a good thing. People don’t need a new version of an app with just 200 lines of new code to fix minor, mostly unencountered problems. Unless it is an emergency, new releases should at minimum provide revised functionality or infrastructure shifts. That is, most of the time, a user should see some obvious changes in each release. If it’s not visible to them, and they have not been affected, it is just a disruption, not a positive.
Some iterations should always be pure cleanup. The bane of software projects is disorganization. As the project grows, its size invalidates earlier organization efforts. So, even regular consistent cleanup attempts will not be enough to keep the code from reaching the end of its life. Thus, organization itself is both the primary feature and main functionality of every software project. If it’s not there, then the value of the effort is extraordinarily low.
Even for the best lead developers and domain experts, there will be plenty of unknown unknowns. Which means that throughout the entire life of the project, there will be new and better information coming in which needs to be acted on. That is, while we know the technology itself is always changing, the understanding of the domain problems is often in flux as well. Managing change is essentially part of managing the development complexity of any software. Expect change, embrace change, and try to spend as little time as possible fighting about when those changes get reflected properly in the software.
As well, everyone involved in any software development project will grow. They will learn more, they will understand more, and many of the details will emerge. Even for late-stage development where the domain is nearly fully covered, the depth of what is understood is bottomless. Working on software is an exercise in constant learning; it never ends.
In some development efforts, there are significant trust issues. When this is the case, then all major decisions need to be papered, and they need sign-offs. If there is trust, particularly between the lead developer and the domain expert, then it can be bypassed. But if neither side insists on sign-offs, then both are equally guilty later when there is a disagreement. If the sign-off isn’t papered, it can be verbal. If it didn’t happen, then one side moved forward without informing the other. So, you can work on a verbal set of features, functionality, specifications, design, etc. only if both sides are in full trust mode. But if it results in a disruptive disagreement, the weight has to change going forward, and the decisions have to be written out in an email, a document, or any other medium that allows for one party to sign off on the terms of the effort.
It is trust that allows a methodology to be lightweight. If that is broken, it can no longer be lightweight and needs to be as heavy as the conflicts that stand in between the communications. That is obviously costly in time, so trying to stay as lightweight as possible is more effective.
It should also be noted that quality comes from not rushing through the work. So, if enhanced quality is a part of the requirements for solving the problem, then it necessarily slows down the effort. Basic quality for a rushed software project is a pretty low bar. Somewhere around expecting monthly big failures, if not weekly. If that is too much, the development effort needs to be slower; more tracking or oversight will only make the issues worse.
Quality control is an essential part of all software development. Since coding is highly mistake-prone, it is critical to ensure that all code is always properly tested before getting into releases. No untested code should ever end up in the operations environment; that is an avoidable and a seriously bad mistake.
The degree and type of testing depend heavily on the required quality of the effort and the time available for building it. Some code needs incredibly intense testing; some just needs a light amount of smoke testing. That is driven more by the problem domain, not the technical one. That is, writing an app for entertainment may not need much testing, but writing one to be included in a medical device, airplane, or any other strict domain needs a lot and many different types of testing.
So, as well as tracking and sign-offs, heavyweight processes often involve adding strong one-to-one relationships between the specified functionality and its testing in order to verify that it was implemented as described. That is, it ensures that the development process did not accidentally or intentionally deviate from what was specified and is known to be needed for the problem. It also ensures that there is no poorly tested functionality stuffed into the codebase.
In many cases, if the software can glitch with low impact, and the timeline is super tight, the testing can be minimal. But if that results in significant runtime problems, then the testing requirements need to increase. As long as there is any sort of upset from bugs escaping, the testing needs to be enhanced until finally the right level is found. That obviously means that putting effort into automating the testing itself is important, since it is more reliable. But it also means that the type of testing chosen is critical too. If partially testing a few of the components isn’t sufficient, and too many problems are occurring, then it needs more system, regression, or even reference testing.
Testing is not an afterthought; it is tied directly to the problem. That is, some problems need better quality, some are more lenient. It is influenced by the development effort as well. Some teams are better at double-checking their work; some are more reckless. The lead developer should know what the development needs and what the team is capable of. The domain expert should set a minimal level of quality that doesn’t contradict their time expectations.
As a big part of this methodology is in providing solutions that satisfy people’s ability to deal with their problems, part of the trust issues between developers and them is the need to act professionally. That is not just in writing code of reasonable quality or putting in the time to understand the domain perspectives, but also in the words and terminology used to describe the process as well. Because of that, for the parts of the process that developers may or may not need, it is important not to use gamified terminology for them. Some of the earlier lightweight methodologies tried to make the names sound fun or even cute, but that only degrades the way others see the profession. The names and terminology used need to look professional from the outside perspective because that is how developers get people to trust them, which allows them to keep the processes between them and stakeholders lightweight, which we know is at least a little less stressful for developers.
In summary, if a development project is adaptive, you should be able to easily identify the domain expert and the lead developer. Both of them should know what they are responsible for, and be good at avoiding stepping over those lines. As the issues have arisen, more heavyweight processes have been added, but the processes are as lightweight as possible. Both the problem descriptions and the code are organized; you can look through the details and see that they have been cataloged and handled appropriately. The code is always tested enough, and there are few operational issues. Whatever higher-weighted processes like some tracking are there, they are well organized and are not choking out the momentum. The train is on the tracks and moving forward at the best pace it can given the life phase of the project.
So for developers, the idea is that you have to know what the long-term solution is that you are building, while you have to keep adapting to the short-term disruptions that you encounter along the way. It’s trying to be proactive, while not letting the reactive issues knock everything too far off course. It's knowing that there is at least one person who really understands the problem you are trying to solve and another who really understands utilizing technology to really solve those problems, and that they rarely are the same person. It’s not over-engineered solutions that veered way off course, nor overly reactive little partial solutions that are piled together into a mess. That the long-term anchor keeps the short-term responses in line to extend out the lifespan of the effort, and that that is the most cost-effective way to proceed.
Realistically, we need methodologies, even if we don't like to be confined by them. Too many loose strings in a project will eventually tangle.
Which is why I liked the emerging lightweight ideas at the turn of the century. But I found that over time their intent turned away from what I thought was reasonable.
So, I like to propose my own methodology. I guess I’ll call it “Adaptive”.
I realize that the word ‘adaptive’ is already taken for a specific type of ‘project management’, but this methodology is somewhat aligned to that, so they are complementary, while still very different. This is about the act of building complex things, not about the show that surrounds it.
The methodology starts as lightweight, but as quality or funding constraints are needed, it gets heavier. So, it’s almost weightless if you need to whip out a demo, but it’s pretty darn heavy if the code will end up in a medical device.
It values quality over quantity. That is, the intent of a software development project is not to create a mass of code; rather, it is to get the best code you can to solve the problem as precisely as you can. If the solution needs 150K to work properly, then the closer you get to that size, with enough quality to keep it useful, the better it is going.
It values organization over speed. No matter what you create, if it ends up in a giant pile of disorganized stuff, it is useless. Organization is, at minimum, having a place for everything and then putting everything in its place. The development and operations of the solution both need to be kept organized. A big part of organization is knowing where to put things, so there needs to be enough effort devoted to that to allow lots of people to know where things go.
It values engineering over evolution. That is, if there is a state of the art for implementing some specific part of the solution, then the people involved will go out and learn more about those prior efforts. They won’t reinvent the wheel just to save a little bit of time. Crude is not good. They know that the knowledge to do the work is out there, and they’ll attempt to find it first. Once they know, they are free to bend it to their will, or extend it, or even go off in a different direction, but first they have to know.
It values fit over ease of effort. If there are two ways to implement something, a simple one and a complex one, the best implementation isn’t driven by the technology or by the developers' preferences; rather, it is driven by which choice makes the solution the best fit for the users. It’s from their perspective, not the people building the solution. The point of software is that it gets used, so degrading its usefulness to save a little time or concentration doesn’t make sense. What is done is always done to make the users' lives better, even if it is hard and painful for the developers.
As for the specifics, since there are two main axes to the effort, there are two main roles for any project.
On the solution side, there is a lead developer. An expert in software development. They are fully responsible for the development of the software. They have a long, extensive background in building stuff that was complicated enough to act as a reasonable solution for similar problems. It’s not about energy or intent, but rather it is about having been through work in the past that is close enough to what needs to be done now, so that they will make the best choices for the project to keep it from derailing.
On the problem side, there is a domain champion. An expert in the domain problems. They are the person who is managing all aspects of the development from the outside and has considerable expertise in the problem itself. In an enterprise, they are at least a PM / BA combination with actual real-world experience in the business. They have spent time in the domain trenches. They have to have fairly deep knowledge in the domain and its practices, since this is where the details get difficult. They handle the external issues.
The domain expert is responsible for and the expert on all aspects of the problem. They know what the related problems are; they understand them. They have some idea of how to correctly deal with aspects of the problem, but they are not experts in solving it. So they know what the problem is, and have some ideas about how they think a computer might help deal with those issues. They control or have raised money to get the problem solved.
The developer is responsible for and in control of all aspects of the solution. The internals of development. If the solution does not correctly solve the problems, it is their fault. They know how to build it, how to deploy it, and how to keep it running over time. They understand how well it actually fits into the problem space. That is, they can’t just build a generic app and claim it ‘could’ solve the problems; they have to build something that ‘does’ solve the problems, so they can’t ignore understanding the problems themselves.
In that sense, the domain expert sets out a series of features they would like to see in the solution, while the software expert implements these as functionality. It’s a partnership. The software expert understands a lot of the nature of the problems that drive the solution, while the domain expert understands the limits and costs of technologies. But the domain expert has the final word on all domain issues, and the software expert has the final word on all technical ones.
The domain expert can say “your proposed solution is not actually solving our problems”, and the software expert can say “at best we can only approximate a solution to that given problem”. The path forward lies between these two points.
As for specific development processes, it is incumbent on the software expert to keep in constant communication with their own development team. How they choose to do that is up to them. Maybe it’s frequent meetings, or one-on-ones, or some combination. These are internal development product conversations; they should be private away from the domain conversations. What’s important is that they are constantly and actively communicating. The development team cannot be left in the dark, and if they are stuck on something, they need to raise it to the lead immediately.
If the internal project communication breaks, the project will derail.
The software expert and the domain expert are always working together. Neither the problem aspects nor the solution aspects have priority over each other. The idea is that they both need to be balanced. Part of the problem is often ‘time’. If a solution arrives too late, that is a big deal. But, unfortunately, since most development work is iterative over years of effort, prioritizing the correct order of the development work is on the software side. This is a contradiction that both sides have to resolve. It is necessary to avoid wasting time doing the wrong work at the wrong time, while still being timely. So, the two experts are in constant negotiation.
Small ambiguities or misunderstood domain and technical issues pop up on a regular basis. It’s normal. The sooner they are pushed over the fence, the better. While this methodology is not reactive, it does prioritize timeliness; but both sides should try to get ahead of the ball wherever possible. That is, the work may suddenly need to change in response to the ground shifting. It is not appropriate to sit and wait on a showstopper, for example, as the effort going on in the meantime may just become makework.
In very small projects, the lead developer might also be the domain expert, but only if they actually have the background to know enough about the domain. That is, they can’t be a faux domain expert; they have to be a real one. They don’t have to know the total full width of the domain, but they do have to know the width of the domain problems they are intending to solve.
The converse is not true. A domain expert with no significant technical background, given the scale of the project, cannot act as the lead developer. Whacking out some Excel spreadsheets or demos does not qualify someone to run a medium-sized development; the consequences will be disastrous.
A project gets started with the intent to solve a problem. Thus, the domain expert is there from day one. Once the project is in motion, however, the technical issues tend to slowly dominate. Most often, it is difficult to prioritize which features get done first, given whatever the limited time constraints.
Once there is an existing codebase, it has a number of deficiencies, and there is also some desire to grow certain parts of the solution. It’s a bit easier. But the effort slows down, as the work itself should get better; more precise.
There is always a long-term plan, as there is a defined problem, and an expectation that a specific collection of features will act as a reasonable solution to solve them. The initial round of development may only be scheduled for six months, but the long-term plan may extend out five or more years. Both should be discussed; both should be on the table. The value in any software solution comes not from its existence, but from its ability to be there, over and over again, as you need it. That is, even if the initial version is crude, if it does have value, it will gradually evolve into something significantly more complex. If it doesn’t have that value, it was probably a waste of time.
That there is both a short-term and a long-term for all projects is important to keeping the effort organized and to ensuring that the work overall is as effective as it can be. Which is why any sort of work compression like ‘reuse’ or ‘abstraction’ is critical. Once some work is completed, you’ll need to leverage it later, not redo it.
Any and all codebases will eventually reach a maturity point where the effort to extend them is so slow and painful that the life of the codebase is effectively frozen. As well, all codebases need constant maintenance, since the underlying technology is always changing at a crazy rate. If the work gets into that ugly ball of mud stage too early, then it will suffer a premature death. It will waste a lot of money; cost way more. Obviously, for many little demos this is not important, but for any medium-sized system or larger that lots of people rely on, it is highly damaging. If a line of business is reliant on the software, it is extraordinarily destructive.
At the core, though, writing software for the sake of writing software may be an interesting performance art, but it is not the intent of any of these larger development efforts. For these, the software is written explicitly as a solution to some real-world problems. So the code that goes into that solution must fully solve the problems and match their complexity. You shouldn’t over-engineer, but you also shouldn’t oversimplify.
Development projects are inherently iterative. That is, it is a never-ending discrete set of development efforts that can go on for years, or even decades. The frequency of demos and releases is driven by an external timetable and is often influenced by issues such as confidence and trust, particularly in the early stages of development. This means that the length of time for each round of development is variable and shifts. On the one hand, if the release periods are too short, there is more work involved because work is done out of sequence and more likely to be disorganized, but if they are too long, the funding or interest may fade. So, there is no perfect immutable length of time for each iteration. For big work, longer is better; for maintaining trust, shorter is better. On top of this, most funding comes with significant strings, so there is some minimal amount of tracking that is also necessary to maintain confidence. This is not extra work; both the technical and domain sides need to be confident that the big expense of development is worth the risk.
But it’s worth noting that excessive tracking will gum up the effort and the morale. Nobody likes being micromanaged, and any attempt to track the time for each and every little task is a classically bad form. The technology lead needs to keep presenting evidence that the work is progressing as expected, and they need to be diligent about raising any potential show stoppers early and often. The domain expert needs significant patience and to understand that even with the best of specifications, development teams, and technologies, there are still a great deal of uncertainties that may cause significant disruptions, particularly if the pace of development has been rushed.
Overly frequent tiny releases are both expensive and disruptive to the users, so they are not a good thing. People don’t need a new version of an app with just 200 lines of new code to fix minor, mostly unencountered problems. Unless it is an emergency, new releases should at minimum provide revised functionality or infrastructure shifts. That is, most of the time, a user should see some obvious changes in each release. If it’s not visible to them, and they have not been affected, it is just a disruption, not a positive.
Some iterations should always be pure cleanup. The bane of software projects is disorganization. As the project grows, its size invalidates earlier organization efforts. So, even regular consistent cleanup attempts will not be enough to keep the code from reaching the end of its life. Thus, organization itself is both the primary feature and main functionality of every software project. If it’s not there, then the value of the effort is extraordinarily low.
Even for the best lead developers and domain experts, there will be plenty of unknown unknowns. Which means that throughout the entire life of the project, there will be new and better information coming in which needs to be acted on. That is, while we know the technology itself is always changing, the understanding of the domain problems is often in flux as well. Managing change is essentially part of managing the development complexity of any software. Expect change, embrace change, and try to spend as little time as possible fighting about when those changes get reflected properly in the software.
As well, everyone involved in any software development project will grow. They will learn more, they will understand more, and many of the details will emerge. Even for late-stage development where the domain is nearly fully covered, the depth of what is understood is bottomless. Working on software is an exercise in constant learning; it never ends.
In some development efforts, there are significant trust issues. When this is the case, then all major decisions need to be papered, and they need sign-offs. If there is trust, particularly between the lead developer and the domain expert, then it can be bypassed. But if neither side insists on sign-offs, then both are equally guilty later when there is a disagreement. If the sign-off isn’t papered, it can be verbal. If it didn’t happen, then one side moved forward without informing the other. So, you can work on a verbal set of features, functionality, specifications, design, etc. only if both sides are in full trust mode. But if it results in a disruptive disagreement, the weight has to change going forward, and the decisions have to be written out in an email, a document, or any other medium that allows for one party to sign off on the terms of the effort.
It is trust that allows a methodology to be lightweight. If that is broken, it can no longer be lightweight and needs to be as heavy as the conflicts that stand in between the communications. That is obviously costly in time, so trying to stay as lightweight as possible is more effective.
It should also be noted that quality comes from not rushing through the work. So, if enhanced quality is a part of the requirements for solving the problem, then it necessarily slows down the effort. Basic quality for a rushed software project is a pretty low bar. Somewhere around expecting monthly big failures, if not weekly. If that is too much, the development effort needs to be slower; more tracking or oversight will only make the issues worse.
Quality control is an essential part of all software development. Since coding is highly mistake-prone, it is critical to ensure that all code is always properly tested before getting into releases. No untested code should ever end up in the operations environment; that is an avoidable and a seriously bad mistake.
The degree and type of testing depend heavily on the required quality of the effort and the time available for building it. Some code needs incredibly intense testing; some just needs a light amount of smoke testing. That is driven more by the problem domain, not the technical one. That is, writing an app for entertainment may not need much testing, but writing one to be included in a medical device, airplane, or any other strict domain needs a lot and many different types of testing.
So, as well as tracking and sign-offs, heavyweight processes often involve adding strong one-to-one relationships between the specified functionality and its testing in order to verify that it was implemented as described. That is, it ensures that the development process did not accidentally or intentionally deviate from what was specified and is known to be needed for the problem. It also ensures that there is no poorly tested functionality stuffed into the codebase.
In many cases, if the software can glitch with low impact, and the timeline is super tight, the testing can be minimal. But if that results in significant runtime problems, then the testing requirements need to increase. As long as there is any sort of upset from bugs escaping, the testing needs to be enhanced until finally the right level is found. That obviously means that putting effort into automating the testing itself is important, since it is more reliable. But it also means that the type of testing chosen is critical too. If partially testing a few of the components isn’t sufficient, and too many problems are occurring, then it needs more system, regression, or even reference testing.
Testing is not an afterthought; it is tied directly to the problem. That is, some problems need better quality, some are more lenient. It is influenced by the development effort as well. Some teams are better at double-checking their work; some are more reckless. The lead developer should know what the development needs and what the team is capable of. The domain expert should set a minimal level of quality that doesn’t contradict their time expectations.
As a big part of this methodology is in providing solutions that satisfy people’s ability to deal with their problems, part of the trust issues between developers and them is the need to act professionally. That is not just in writing code of reasonable quality or putting in the time to understand the domain perspectives, but also in the words and terminology used to describe the process as well. Because of that, for the parts of the process that developers may or may not need, it is important not to use gamified terminology for them. Some of the earlier lightweight methodologies tried to make the names sound fun or even cute, but that only degrades the way others see the profession. The names and terminology used need to look professional from the outside perspective because that is how developers get people to trust them, which allows them to keep the processes between them and stakeholders lightweight, which we know is at least a little less stressful for developers.
In summary, if a development project is adaptive, you should be able to easily identify the domain expert and the lead developer. Both of them should know what they are responsible for, and be good at avoiding stepping over those lines. As the issues have arisen, more heavyweight processes have been added, but the processes are as lightweight as possible. Both the problem descriptions and the code are organized; you can look through the details and see that they have been cataloged and handled appropriately. The code is always tested enough, and there are few operational issues. Whatever higher-weighted processes like some tracking are there, they are well organized and are not choking out the momentum. The train is on the tracks and moving forward at the best pace it can given the life phase of the project.
So for developers, the idea is that you have to know what the long-term solution is that you are building, while you have to keep adapting to the short-term disruptions that you encounter along the way. It’s trying to be proactive, while not letting the reactive issues knock everything too far off course. It's knowing that there is at least one person who really understands the problem you are trying to solve and another who really understands utilizing technology to really solve those problems, and that they rarely are the same person. It’s not over-engineered solutions that veered way off course, nor overly reactive little partial solutions that are piled together into a mess. That the long-term anchor keeps the short-term responses in line to extend out the lifespan of the effort, and that that is the most cost-effective way to proceed.
Thursday, July 23, 2026
Predictions
Being able to predict what’s coming up in the future allows you to make better decisions right now. Better decisions will give you more chances to actually change the current direction rather than just getting caught up in the flow.
The first part of enhancing predictions is to accept that there is only one objective reality. Although it surrounds all of us, we only ever see a tiny sliver of it. Because of that, it is not uncommon for people to believe that their sliver is somehow personalized just for them. It is not. We might sometimes perceive the world as somewhat random, but that’s only because we are caught by surprise.
The next part is to understand that all things follow a trajectory. They will continue on their merry way unless they are interrupted by something unusual. From our perspective, reality is an informal system; all sorts of surprising things can occur. It is constrained, but these constraints are beyond our ability to formalize them. Although this allows for a lot of wiggle room on what might actually happen, it’s still constrained. Some things just can’t happen; they violate some very deep underlying rules about how the physics of the universe will play out.
In that sense, there are plenty of things that are possible, but also highly unlikely. And there are plenty of things that are absolutely impossible, although we might not have full certainty that they are not just unlikely.
So if we observe any phenomena long enough, while we can’t discount that surprising things may occur, if they don’t, then it will follow the trajectory that it has been on already. That is, things will mostly continue as expected based on their past behaviour.
That can be a bit tricky though, if the past behaviour has a rather complicated path through a very long period of time. That’s a lot of complexity to absorb.
Thus, to get a reasonable prediction, you need to observe the actual path through a context that is large enough to include all of the non-surprising but significant influences. Pull back far enough to include all of the dominant variations and influences, watch the movement, and then sketch out the default path forward. If nothing unusual throws it off course, you know where it is headed.
If you are surprised, it could just be an extremely rare event, or that your context was just not large enough to be accurate. Most often, it was too narrow a context.
In that sense, being able to predict lots of things around you is an indication that your knowledge extends far beyond your own sliver. You can learn to predict larger phenomena or farther out in the future as you acquire a deeper understanding of reality. They are correlated.
In the general sense, getting better at predictions will make your actions go smoother. You can leave a little room for the informality, but mostly get the results that you desire. It also helps with insurance. You might not be able to precisely predict infrequent events, but at least you can plan for them in order to mitigate their disruptions. This is a better strategy than just being purely reactive, in that it prepares you to better navigate any obstacles while keeping you aligned to a longer-term goal. When you can anticipate your bad days, you can prepare enough in advance to make them easier to deal with.
The first part of enhancing predictions is to accept that there is only one objective reality. Although it surrounds all of us, we only ever see a tiny sliver of it. Because of that, it is not uncommon for people to believe that their sliver is somehow personalized just for them. It is not. We might sometimes perceive the world as somewhat random, but that’s only because we are caught by surprise.
The next part is to understand that all things follow a trajectory. They will continue on their merry way unless they are interrupted by something unusual. From our perspective, reality is an informal system; all sorts of surprising things can occur. It is constrained, but these constraints are beyond our ability to formalize them. Although this allows for a lot of wiggle room on what might actually happen, it’s still constrained. Some things just can’t happen; they violate some very deep underlying rules about how the physics of the universe will play out.
In that sense, there are plenty of things that are possible, but also highly unlikely. And there are plenty of things that are absolutely impossible, although we might not have full certainty that they are not just unlikely.
So if we observe any phenomena long enough, while we can’t discount that surprising things may occur, if they don’t, then it will follow the trajectory that it has been on already. That is, things will mostly continue as expected based on their past behaviour.
That can be a bit tricky though, if the past behaviour has a rather complicated path through a very long period of time. That’s a lot of complexity to absorb.
Thus, to get a reasonable prediction, you need to observe the actual path through a context that is large enough to include all of the non-surprising but significant influences. Pull back far enough to include all of the dominant variations and influences, watch the movement, and then sketch out the default path forward. If nothing unusual throws it off course, you know where it is headed.
If you are surprised, it could just be an extremely rare event, or that your context was just not large enough to be accurate. Most often, it was too narrow a context.
In that sense, being able to predict lots of things around you is an indication that your knowledge extends far beyond your own sliver. You can learn to predict larger phenomena or farther out in the future as you acquire a deeper understanding of reality. They are correlated.
In the general sense, getting better at predictions will make your actions go smoother. You can leave a little room for the informality, but mostly get the results that you desire. It also helps with insurance. You might not be able to precisely predict infrequent events, but at least you can plan for them in order to mitigate their disruptions. This is a better strategy than just being purely reactive, in that it prepares you to better navigate any obstacles while keeping you aligned to a longer-term goal. When you can anticipate your bad days, you can prepare enough in advance to make them easier to deal with.
Thursday, July 16, 2026
Hedging
A long time ago, I remember someone trying to convince me that sufficient testing would prevent ‘all’ bugs. All you need are tests.
That’s never been the case for me, even after all of these decades.
That’s never been the case for me, even after all of these decades.
At the very best, with a gifted team, we were super careful, had strict standards, and hyper-intense reviews, but we still had one bug escape. Worse, though, was that the effort took considerably longer than most people would even guess. It was the only time that I’ve ever not felt rushed during development. So it’s rare.
Ever since then, the default for me has been to anticipate that we won’t have enough time and that the code will be half-baked at best.
That’s okay if you expect it, but it changes your practices and habits. You learn to hedge any and all bets, always.
For example, you always assume there will be lots of bugs and make sure you’ve scheduled in enough time to find and fix them. If you track it, you can often get a sort of accurate guesstimate on how many to expect or at least an upper and lower limit.
You also want scaffolding. That is code that indirectly asserts, verifies, stops, counts, or in any other way validates the incoming data or calling sequences. You have it turned on during development, on for some testing, and off for production.
Full end-to-end system testing is always the best, but unit testing works for overly complex components. There is never a one-type-of-testing-that-finds-all bullet. Testing is expensive; you have to match the different types based on your time and quality requirements.
You want good logs and lots of auditing. From the logs, you should be able to see any and all internal state changes. While having one is best, they do get added all over the place. Logs are the main way to fully resolve real production issues.
For auditing, you need to know who did what. For some domains, a few weeks would suffice, but for some, you need years or decades worth of it.
You need a fast flag switcher. It lets you swap out functionality on the fly in production. It’s for extreme emergencies only, but time compression makes these far more likely.
You want to table-drive every piece of code that involves high uncertainties. That’s the most flexible arrangement possible, and it tends to be resilient against update mistakes.
You need two release processes. The right way and the fast way. You should never use the fast way unless you have no choice.
You want as much reuse as you have the time to get. It pays for itself a bit later, but by then you’ll really appreciate it. Particularly if you have to whip out some revised functionality, and you can do that nearly instantaneously with battle-hardened code. Way better than doing the bug dance again.
You should always be suspicious of any code that you depend on but don’t understand or didn’t write. It tends to bite you when you least expect it. The more you learn about how it works, the better things will go. If you can do it properly yourself, your life will be easier.
Ever since then, the default for me has been to anticipate that we won’t have enough time and that the code will be half-baked at best.
That’s okay if you expect it, but it changes your practices and habits. You learn to hedge any and all bets, always.
For example, you always assume there will be lots of bugs and make sure you’ve scheduled in enough time to find and fix them. If you track it, you can often get a sort of accurate guesstimate on how many to expect or at least an upper and lower limit.
You also want scaffolding. That is code that indirectly asserts, verifies, stops, counts, or in any other way validates the incoming data or calling sequences. You have it turned on during development, on for some testing, and off for production.
Full end-to-end system testing is always the best, but unit testing works for overly complex components. There is never a one-type-of-testing-that-finds-all bullet. Testing is expensive; you have to match the different types based on your time and quality requirements.
You want good logs and lots of auditing. From the logs, you should be able to see any and all internal state changes. While having one is best, they do get added all over the place. Logs are the main way to fully resolve real production issues.
For auditing, you need to know who did what. For some domains, a few weeks would suffice, but for some, you need years or decades worth of it.
You need a fast flag switcher. It lets you swap out functionality on the fly in production. It’s for extreme emergencies only, but time compression makes these far more likely.
You want to table-drive every piece of code that involves high uncertainties. That’s the most flexible arrangement possible, and it tends to be resilient against update mistakes.
You need two release processes. The right way and the fast way. You should never use the fast way unless you have no choice.
You want as much reuse as you have the time to get. It pays for itself a bit later, but by then you’ll really appreciate it. Particularly if you have to whip out some revised functionality, and you can do that nearly instantaneously with battle-hardened code. Way better than doing the bug dance again.
You should always be suspicious of any code that you depend on but don’t understand or didn’t write. It tends to bite you when you least expect it. The more you learn about how it works, the better things will go. If you can do it properly yourself, your life will be easier.
You want to assume at any time that you only have half of the picture, and that it is about to change. So, hedging some code with expected variability is more than reasonable.
If you really understand why the code solves a specific problem, you can vet it as you write it. If it's all Greek to you, it's more likely that it will need excessive changes in the future. It often gets lost in translation. You can factor it to make it easier to change.
Initially, you might get praise for grinding out tons of static brute force code, but that quickly changes as the problems and the high costs of fixing them emerge. It’s better if they accuse you of being slow, but later the drama just doesn’t happen. They’ll catch up eventually.
At all times, you just know that it's going to get difficult and somebody somewhere will always be upset. You shouldn’t develop a lack of empathy; instead, you should set up the game to quickly respond with increased precision. The second it's discovered to be wrong, you’re right on top of the fix. That gives you a much better reputation than if you shrug angrily and say, “I don’t know”.
If you really understand why the code solves a specific problem, you can vet it as you write it. If it's all Greek to you, it's more likely that it will need excessive changes in the future. It often gets lost in translation. You can factor it to make it easier to change.
Initially, you might get praise for grinding out tons of static brute force code, but that quickly changes as the problems and the high costs of fixing them emerge. It’s better if they accuse you of being slow, but later the drama just doesn’t happen. They’ll catch up eventually.
At all times, you just know that it's going to get difficult and somebody somewhere will always be upset. You shouldn’t develop a lack of empathy; instead, you should set up the game to quickly respond with increased precision. The second it's discovered to be wrong, you’re right on top of the fix. That gives you a much better reputation than if you shrug angrily and say, “I don’t know”.
Thursday, July 9, 2026
Data Stitching
If you collect some data with respect to a specific context, when there are significant changes in that context, the original data no longer fits seamlessly with any newly collected data.
One way to handle this is to change all of the old data, updating it to the new changes. This works, but it could take quite a while in production and can be extremely risky. A change in massive data can take days or even weeks to run. A small mistake could effectively destroy the old data.
The other way to handle this is at ‘presentation’. You split the computations, doing something slightly different for the older data.
All you need is a precise ‘line’ between them to test for. Then, to keep the code sane, you make both sides polymorphic and hide the test. When some code asks for the final data, if that request crosses the line, it gets handled correctly.
The way you don’t want to do this is by hardcoding the test somewhere high. Doing this once may not cause too much damage, but if the context has changed once already, it is highly likely to change again. A few times is really ugly; more than that is a classic bug sinkhole.
The overall idea is to always preserve what you collect and stitch it together with polymorphism. Then you can double-check it for correctness, so you have the best possible options for getting it right, and you don’t leave around landmines for later efforts.
For anyone worried about performance, presentations are almost always fixed-sized small data sets; the extra costs of the tests and the polymorphism are usually hardly noticeable. If there is some other type of dynamic large-scale bulk interface over the data, then it’s an architectural problem, not a data one. That is, if the performance is really noticeable, then you’re probably doing something else in the software that you shouldn’t be doing.
Data stitching is often avoided, but when done reasonably, it ensures that the data appears correctly and that it is straightforward to fix later if it doesn’t. Depending on the data, it often requires a domain expert to verify, which usually happens very late in the effort, so it's best to expect last-minute changes.
One way to handle this is to change all of the old data, updating it to the new changes. This works, but it could take quite a while in production and can be extremely risky. A change in massive data can take days or even weeks to run. A small mistake could effectively destroy the old data.
The other way to handle this is at ‘presentation’. You split the computations, doing something slightly different for the older data.
All you need is a precise ‘line’ between them to test for. Then, to keep the code sane, you make both sides polymorphic and hide the test. When some code asks for the final data, if that request crosses the line, it gets handled correctly.
The way you don’t want to do this is by hardcoding the test somewhere high. Doing this once may not cause too much damage, but if the context has changed once already, it is highly likely to change again. A few times is really ugly; more than that is a classic bug sinkhole.
The overall idea is to always preserve what you collect and stitch it together with polymorphism. Then you can double-check it for correctness, so you have the best possible options for getting it right, and you don’t leave around landmines for later efforts.
For anyone worried about performance, presentations are almost always fixed-sized small data sets; the extra costs of the tests and the polymorphism are usually hardly noticeable. If there is some other type of dynamic large-scale bulk interface over the data, then it’s an architectural problem, not a data one. That is, if the performance is really noticeable, then you’re probably doing something else in the software that you shouldn’t be doing.
Data stitching is often avoided, but when done reasonably, it ensures that the data appears correctly and that it is straightforward to fix later if it doesn’t. Depending on the data, it often requires a domain expert to verify, which usually happens very late in the effort, so it's best to expect last-minute changes.
Thursday, July 2, 2026
Language Ideas
Decades ago, our project had a convention where we never directly used ANSI C primitive types, unless it was trivial in a loop. But for any and all other variables, everything had to be typedef’d explicitly. Everything.
Some people would see that as excessive and way over the top, but it actually turned out to be a very good habit that helps with getting really high quality.
If you went to call something, you’d need an X, and that would force you to go into the code to find out how to create one. If the code is all ints, strings, doubles, etc., then people skip that effort and just find some hacky way to kludge the value they need. But now you can’t.
I’ve often thought that I’d like to see a language that has zero primitive types for variables. None. If you need a variable type, you have to declare it.
But to add to the fun, there aren’t even open types like ‘int’. If you needed an integer, you’d create the type, but as you did, you’d have to explicitly specify the range.
For example:
It would be more fun for strings, as they are not open either:
Each would be constrained by an RE state machine.
What I think would happen, maybe naturally, is that programmers would get bored with creating a million different types and a billion conversions between them. So, instead, they’d start to pack things together into larger structures all the time.
And for any discerning programmers out there who suspect that using just alphabetical characters is not wide enough to handle all of these types of names across the planet, staring at this declaration would trigger a need to further investigate and correct the model.
But the trick would be that in correcting the model, in that one place, it would also be corrected everywhere else. So the benefit would be that all of the validation code, both at the interface and at persistence, that would have needed to have also been updated when the model changes, would actually not need to be changed, since it is all implicit in the language. A recompile would do the trick.
I’d go even further, though. I’d not have any primitive data structures in the language’s library; they’d all be baked into the language too.
For example:
If it’s starting to smell like SQL, I apologize. The type semantics is essentially declarative, but the rest of the language syntax would be imperative, with some extended paradigm on top. Probably more like Golang than any of the other OO or FP variants.
The point, though, is that you would use the type mechanism to build up larger and larger data structures, and all of the base ones, like lists, trees, dags, graphs, and even hypergraphs, would already be there. For fun, you’d have all others like stacks, queues, and pagodas. All with various options, but to use more advanced ones, you would have to explicitly declare that they were not just trivial implementations.
That would ensure that anyone reading would not have to infer anything about the underlying implementations. The crudest thing would always be the default thing. Any special ability, property, enhancement, or optimization would have to be explicitly mentioned. It would minimize confusion.It would aid in avoiding bloat.
Over the decades, the trend in languages was to make them more dynamic and to minimize expressions. The result of adding those freedoms in the hands of a disciplined programmer was great, but the abuses heavily outweighed the elegant examples. Learning from that, I think we should find more ways to restrict the freedoms, but not cripple the expressiveness. That is, you’re not forced to type in reams of boring boilerplate, but you also can’t write code that is cryptic enough to win the obfuscated C code contest either. The code you have to write always makes its intent clear. Readability is part of the language.
Some people would see that as excessive and way over the top, but it actually turned out to be a very good habit that helps with getting really high quality.
If you went to call something, you’d need an X, and that would force you to go into the code to find out how to create one. If the code is all ints, strings, doubles, etc., then people skip that effort and just find some hacky way to kludge the value they need. But now you can’t.
I’ve often thought that I’d like to see a language that has zero primitive types for variables. None. If you need a variable type, you have to declare it.
But to add to the fun, there aren’t even open types like ‘int’. If you needed an integer, you’d create the type, but as you did, you’d have to explicitly specify the range.
For example:
type Counter: integer 0..INTMAX
It would be more fun for strings, as they are not open either:
type Token: string [a-zA-Z]*
Each would be constrained by an RE state machine.
What I think would happen, maybe naturally, is that programmers would get bored with creating a million different types and a billion conversions between them. So, instead, they’d start to pack things together into larger structures all the time.
type UserName:
First: string [a-zA-Z]*
Last: string [a-zA-Z]*
And for any discerning programmers out there who suspect that using just alphabetical characters is not wide enough to handle all of these types of names across the planet, staring at this declaration would trigger a need to further investigate and correct the model.
But the trick would be that in correcting the model, in that one place, it would also be corrected everywhere else. So the benefit would be that all of the validation code, both at the interface and at persistence, that would have needed to have also been updated when the model changes, would actually not need to be changed, since it is all implicit in the language. A recompile would do the trick.
I’d go even further, though. I’d not have any primitive data structures in the language’s library; they’d all be baked into the language too.
For example:
type UserList linkedlist:
Entries: User all
Sorted by User.UserName.Last
If it’s starting to smell like SQL, I apologize. The type semantics is essentially declarative, but the rest of the language syntax would be imperative, with some extended paradigm on top. Probably more like Golang than any of the other OO or FP variants.
The point, though, is that you would use the type mechanism to build up larger and larger data structures, and all of the base ones, like lists, trees, dags, graphs, and even hypergraphs, would already be there. For fun, you’d have all others like stacks, queues, and pagodas. All with various options, but to use more advanced ones, you would have to explicitly declare that they were not just trivial implementations.
That would ensure that anyone reading would not have to infer anything about the underlying implementations. The crudest thing would always be the default thing. Any special ability, property, enhancement, or optimization would have to be explicitly mentioned. It would minimize confusion.It would aid in avoiding bloat.
Over the decades, the trend in languages was to make them more dynamic and to minimize expressions. The result of adding those freedoms in the hands of a disciplined programmer was great, but the abuses heavily outweighed the elegant examples. Learning from that, I think we should find more ways to restrict the freedoms, but not cripple the expressiveness. That is, you’re not forced to type in reams of boring boilerplate, but you also can’t write code that is cryptic enough to win the obfuscated C code contest either. The code you have to write always makes its intent clear. Readability is part of the language.
Thursday, June 25, 2026
The Craft of Programming
Back in the eighties, when I was a student, there were only two choices: you used ‘vi’ or you used ‘emacs’.
I picked vi, my roommate picked emacs. The stuff he did with his editor was way cooler, but over the decades, vi has pretty much been everywhere I have stumbled.
I assumed it would fall out of fashion, but it persisted. Vim and all of those embeddings in the IDEs keep it alive.
I read something about how someone thought that using a complicated editor like vi was unnecessary. They implied that it was a waste of time to learn lots of its features. There were so many other things to learn, why not just use a lame mouse-based editor and move on?
For me, the answer is that it is all about craft.
When you build a large program, it’s a lot more than just throwing together the thousands of instructions necessary to tell the computer what to do. The code is just the output, but if you are hasty and careless on your way there, the whole thing ends up as a house of cards; any little breeze will knock it over. That is, it isn’t just coming up with some code; it is coming up with good, solid code that is readable and will survive all of the craziness that the world throws at it.
I’d rather have 150K lines of something that is super tight, abstract, solid, dependable than have 1M lines of stuff that could be served in an Italian restaurant. A whole lot of bad code is just a few steps closer to an apocalypse than a system.
So, it’s not about how much code you can create or how fast you can create it. Instead, it is about how carefully you embed precision into that code. Not for tomorrow, but for its entire lifespan. You want strong solutions to the real problems you need to solve, and you want them encoded in a way that you can keep them moving forward and leverage them for all sorts of related problems.
In the act of doing that sometimes tedious, thoughtful, and precise work, you want to develop habits that are more likely to produce high-quality output. Quality matters, not quantity.
Getting back to vi, if you are concerned about your code, you care very deeply about how it is ordered in any sort of file. It should not be random; you have come up with some consistent reason why one function is placed ahead of the others. In the daily grind, you have probably violated that order accidentally a few times. In most editors, you can use the mouse to highlight a chunk of code, hit ctrl-X to cut it, then ctrl-V to paste it back where it should go. It works, but it’s a touch haphazard. You could be interrupted, or get confused, or any other disruption that might interfere with the outcome you desire. In vi, you ‘mark’ the start, then ‘mark’ the end, then issue a few commands to move the code into place. It’s a small difference, but it’s transactional integrity. It either happens or it does not. If you get interrupted, it doesn’t matter; nothing is lost, nothing is messed up. If you're unsure about where to place the code, you can scroll around with the keyboard until you are sure. It isn’t any slower than using the mouse, but it is a whole lot more precise.
That’s a trivial example, for sure, but it’s just one of those things that you can do with a good tool when you’ve invested the time to understand it fully. Being a good programmer, a reliable one that is always there in a crisis, is the act of investing a whole lot of time in honing these acts of precision. It’s not that you can whack out a whole lot of code that you barely understand; it is that when you are given a piece of work, you can ensure that that work is done satisfactorily, even in environments that are difficult.
That is the essence of good programmers. They are the ones that everybody leans on, especially when it gets difficult, because they know that they’ll be there and make sure it is at least as good as it can get right now. They are reliable. Not always efficient, never fast, but always making sure that the work going out the door is solid.
We’ve always had the expression “garbage in, garbage out” in reference to the necessity to have good input data, but it also applies to code. If you get a group of people that churn millions of lines of questionable code, it’s just a field of landmines waiting to go off. If you get a group of people that solve complex problems with the least amount of code they can, carefully, then it doesn’t take that long before that quality of work starts to pay for itself. The craft of programming isn’t coding; the craft is in providing solutions, it just so happens that most of them involve code in some way.
I picked vi, my roommate picked emacs. The stuff he did with his editor was way cooler, but over the decades, vi has pretty much been everywhere I have stumbled.
I assumed it would fall out of fashion, but it persisted. Vim and all of those embeddings in the IDEs keep it alive.
I read something about how someone thought that using a complicated editor like vi was unnecessary. They implied that it was a waste of time to learn lots of its features. There were so many other things to learn, why not just use a lame mouse-based editor and move on?
For me, the answer is that it is all about craft.
When you build a large program, it’s a lot more than just throwing together the thousands of instructions necessary to tell the computer what to do. The code is just the output, but if you are hasty and careless on your way there, the whole thing ends up as a house of cards; any little breeze will knock it over. That is, it isn’t just coming up with some code; it is coming up with good, solid code that is readable and will survive all of the craziness that the world throws at it.
I’d rather have 150K lines of something that is super tight, abstract, solid, dependable than have 1M lines of stuff that could be served in an Italian restaurant. A whole lot of bad code is just a few steps closer to an apocalypse than a system.
So, it’s not about how much code you can create or how fast you can create it. Instead, it is about how carefully you embed precision into that code. Not for tomorrow, but for its entire lifespan. You want strong solutions to the real problems you need to solve, and you want them encoded in a way that you can keep them moving forward and leverage them for all sorts of related problems.
In the act of doing that sometimes tedious, thoughtful, and precise work, you want to develop habits that are more likely to produce high-quality output. Quality matters, not quantity.
Getting back to vi, if you are concerned about your code, you care very deeply about how it is ordered in any sort of file. It should not be random; you have come up with some consistent reason why one function is placed ahead of the others. In the daily grind, you have probably violated that order accidentally a few times. In most editors, you can use the mouse to highlight a chunk of code, hit ctrl-X to cut it, then ctrl-V to paste it back where it should go. It works, but it’s a touch haphazard. You could be interrupted, or get confused, or any other disruption that might interfere with the outcome you desire. In vi, you ‘mark’ the start, then ‘mark’ the end, then issue a few commands to move the code into place. It’s a small difference, but it’s transactional integrity. It either happens or it does not. If you get interrupted, it doesn’t matter; nothing is lost, nothing is messed up. If you're unsure about where to place the code, you can scroll around with the keyboard until you are sure. It isn’t any slower than using the mouse, but it is a whole lot more precise.
That’s a trivial example, for sure, but it’s just one of those things that you can do with a good tool when you’ve invested the time to understand it fully. Being a good programmer, a reliable one that is always there in a crisis, is the act of investing a whole lot of time in honing these acts of precision. It’s not that you can whack out a whole lot of code that you barely understand; it is that when you are given a piece of work, you can ensure that that work is done satisfactorily, even in environments that are difficult.
That is the essence of good programmers. They are the ones that everybody leans on, especially when it gets difficult, because they know that they’ll be there and make sure it is at least as good as it can get right now. They are reliable. Not always efficient, never fast, but always making sure that the work going out the door is solid.
We’ve always had the expression “garbage in, garbage out” in reference to the necessity to have good input data, but it also applies to code. If you get a group of people that churn millions of lines of questionable code, it’s just a field of landmines waiting to go off. If you get a group of people that solve complex problems with the least amount of code they can, carefully, then it doesn’t take that long before that quality of work starts to pay for itself. The craft of programming isn’t coding; the craft is in providing solutions, it just so happens that most of them involve code in some way.
Thursday, June 18, 2026
Structureless
The most common mistake I have seen in big ugly balls of mud is to try to capture data without enough structure.
Chopping up some incoming data into a lot of little strongly typed fields is a pain. Sometimes it seems like an unwarranted pain. You may need to get a mailing address. Why not just give the user a big textbox to fill in?
The problem isn’t that the users can’t carefully type in the text with appropriate structure; it’s that sometimes they won’t.
And the code to parse unstructured text is stupidly complicated. They can type anything; you have to be able to apply some type of structure to each and every possible variation. Since there are an infinite number of those, you are going to lose.
You can add a ton of validation, but if it’s not rigorous within a fully interactive interface, then if there is any tiny way to bypass it, it was a total waste of effort.
You could scream at the users and make them format it correctly, but as time wears on, unless you keep screaming, eventually that practice will degrade. It will just delay the inevitable.
Which is to say that in a computer, a big box full of text is absolutely nothing more than a big box full of text. It has no other use or value. That makes it useful for someone putting a “personal note” somewhere in a report, or something like that, but there is nothing beyond that. It’s not really data; it’s just an extra external comment of some type.
If what you intend to do is collect data with a very specific structure, you should never subvert text boxes to do that for you. It’s not a shortcut; you haven’t “figured it out”, you just made a very bad mistake. And sadly, doing it right wasn’t that much more time.
Likewise, if you are using some questionable software, and it has a lot of text boxes, so you come up with a clever idea about how to put structured data into those, just because you can’t or don’t know how to change the program, then it isn’t brilliant. It’s just a sloppy, hacky way of trying to get around some other code. One step worse than duct tape.
Data is not data without structure. Untyped data is not data. A big string of characters is a mess. Mostly.
If it originated under the strict control of some code somewhere, and the pathway was closed and guarded, then sure, two programs can use strings and complicated parsers to pass data back and forth. It starts with structure, is transported in an unstructured container, and then it is restructured again. But if the pathway is open or one end of that game is a human, then a big string of stuff is just potential garbage. At some point, usually in the not-too-distant future, someone will fill that string with a problem, and it will end up wasting a lot of time.
Do note that there are subtle variations on this. A user might use one program to render XML that they upload manually into another. In a case like this, the human isn’t the endpoint; they are just the transfer medium. It's an open pathway, but still just between two programs. And you can close down that pathway by only strictly accepting XML with a very specific schema. Since you can verify that and reject garbage as necessary, it becomes closed.
The foundation of all software systems is to collect data. The full and complete structure of that data, as it relates to the real and digital worlds, is an essential part of that data.
There are extraordinary times when it makes sense to only sub-model some external data and live with the consequences of that choice. But the default should always be that if the program needs some data, it is key to its usage and computations, then the data needs to be fully and correctly modelled. That is, it needs the right structures, for example, not shoving a tree into a list, and it needs all of the individual fields in that data to be very strongly typed. It needs this in order to make the correct choices with its instructions based on what is actually there. It can’t be vague or ambiguous; computers are not smart enough to grok external context. They can only act on the very specific information that they have.
It’s also worth noting that while parsing may look easy, and in some tiny instances, it is not that complicated, it should always be considered a hard problem to properly solve. As such, if you can avoid parsing, or at least push it to a human somewhere, then you will get far fewer bugs and the code will be more likely to behave as expected. If you do have to venture into parsing, then it is one of the coding areas where reading a boatload of stuff in advance will pay off huge dividends. Trial and error with parsing is a massive bug generator.
Chopping up some incoming data into a lot of little strongly typed fields is a pain. Sometimes it seems like an unwarranted pain. You may need to get a mailing address. Why not just give the user a big textbox to fill in?
The problem isn’t that the users can’t carefully type in the text with appropriate structure; it’s that sometimes they won’t.
And the code to parse unstructured text is stupidly complicated. They can type anything; you have to be able to apply some type of structure to each and every possible variation. Since there are an infinite number of those, you are going to lose.
You can add a ton of validation, but if it’s not rigorous within a fully interactive interface, then if there is any tiny way to bypass it, it was a total waste of effort.
You could scream at the users and make them format it correctly, but as time wears on, unless you keep screaming, eventually that practice will degrade. It will just delay the inevitable.
Which is to say that in a computer, a big box full of text is absolutely nothing more than a big box full of text. It has no other use or value. That makes it useful for someone putting a “personal note” somewhere in a report, or something like that, but there is nothing beyond that. It’s not really data; it’s just an extra external comment of some type.
If what you intend to do is collect data with a very specific structure, you should never subvert text boxes to do that for you. It’s not a shortcut; you haven’t “figured it out”, you just made a very bad mistake. And sadly, doing it right wasn’t that much more time.
Likewise, if you are using some questionable software, and it has a lot of text boxes, so you come up with a clever idea about how to put structured data into those, just because you can’t or don’t know how to change the program, then it isn’t brilliant. It’s just a sloppy, hacky way of trying to get around some other code. One step worse than duct tape.
Data is not data without structure. Untyped data is not data. A big string of characters is a mess. Mostly.
If it originated under the strict control of some code somewhere, and the pathway was closed and guarded, then sure, two programs can use strings and complicated parsers to pass data back and forth. It starts with structure, is transported in an unstructured container, and then it is restructured again. But if the pathway is open or one end of that game is a human, then a big string of stuff is just potential garbage. At some point, usually in the not-too-distant future, someone will fill that string with a problem, and it will end up wasting a lot of time.
Do note that there are subtle variations on this. A user might use one program to render XML that they upload manually into another. In a case like this, the human isn’t the endpoint; they are just the transfer medium. It's an open pathway, but still just between two programs. And you can close down that pathway by only strictly accepting XML with a very specific schema. Since you can verify that and reject garbage as necessary, it becomes closed.
The foundation of all software systems is to collect data. The full and complete structure of that data, as it relates to the real and digital worlds, is an essential part of that data.
There are extraordinary times when it makes sense to only sub-model some external data and live with the consequences of that choice. But the default should always be that if the program needs some data, it is key to its usage and computations, then the data needs to be fully and correctly modelled. That is, it needs the right structures, for example, not shoving a tree into a list, and it needs all of the individual fields in that data to be very strongly typed. It needs this in order to make the correct choices with its instructions based on what is actually there. It can’t be vague or ambiguous; computers are not smart enough to grok external context. They can only act on the very specific information that they have.
It’s also worth noting that while parsing may look easy, and in some tiny instances, it is not that complicated, it should always be considered a hard problem to properly solve. As such, if you can avoid parsing, or at least push it to a human somewhere, then you will get far fewer bugs and the code will be more likely to behave as expected. If you do have to venture into parsing, then it is one of the coding areas where reading a boatload of stuff in advance will pay off huge dividends. Trial and error with parsing is a massive bug generator.
Thursday, June 11, 2026
Software Systems
I use the term ‘software system’ loosely. I usually intend it to mean: all of the boundaries for a set of related solutions that have been or will be implemented with software.
In that sense, it is less about the technical parts of the ‘system’ and more about how they all come together to help people.
I do this mostly because I tend to visualize a ‘problem space’ as a flat 2D terrain. It is a convenient oversimplification. It is a big, wide, open, empty field of grass which spans over related problems.
When I am doing greenfield work, I see the start as picking one spot in that field. You start there, building up enough structure to be useful. First, you lay down some common foundations, then you start adding in functionality that implements the features you know will help solve the problem.
As you do this, people will see the effort and start making suggestions. Some will want to go off in one direction, while others will prioritize the opposite way.
The trick to keeping it all as usable as possible is to slowly expand out your borders, but not in too many directions all at once.
Someone once told me that for software, you should never pick a path unless you are willing to walk it. From this perspective, it usually means that you won’t expand into another area in the field haphazardly. If you do choose to go there, it needs to be done correctly. That is, adding a few really good, solid features is way better than adding a million lame ones.
The same is true for the data. If you need new data, you add it carefully, properly structured, or not at all.
Overall, though, you start at one specific spot and keep growing. If it’s a good set of programs and people find it valuable, you’ll probably be at it for years, if not decades. So, it’s really crucial to its lifespan that the work you do in the very early days is as good as it can be. It needs to be neat, tidy, organized, and carefully thought out.
With that in mind, calling the current work and any of the rather obvious future work a ‘system’ works quite well. The system isn’t the code, but rather it is all of the territory that the code is trying to cover at some point. You might build a system for handling the account problems in a large corporation, for example. There might be lots of included pieces, and even some nearly stand-alone sub-systems, but they are all trying to fit together to deal with the same problems.
So, it’s similar to seeing the forest through the trees. The boundaries of the expected work are the system, but the system may not stretch right up to those boundaries yet.
From this view, it makes it easier to understand a bottom-up implementation. You might not know all of the features that people will ask for, but you should have a reasonable sense of the territory you are covering right now. Lots of that territory is similar, so building reusable components and engines will really help in getting more ground covered at a faster rate.
The classic example is reporting. You know people will need it at some point, so instead of just hardcoding a couple of static examples, it would be better to either offload it to somewhere else as exported data, or write some generic engine that is flexible enough to cover its rather massive width. The trick is not to write a lot of code, but rather to leverage any code you do write to cover the largest parts of that territory. In software, a little foresight goes a long way.
Thinking of a software system this way really helps in making a lot of the implementation decisions. If your field doesn’t cover having a million users, then designing an architecture to support that scale doesn’t make any sense.
More importantly, if you are located in one corner of the field, then trying to expand way over to the other side of a nearby hill doesn’t make a lot of sense either. That’s far enough away that it is clearly another system, another project and thus another codebase.
Building even medium-sized software is surprisingly complicated, so finding ways to frame it nicely really helps with making better decisions. Since time is a precious resource, getting to the right code as quickly as possible is important. Seeing it all as a system occupying some territory in an endless field is a good guide.
In that sense, it is less about the technical parts of the ‘system’ and more about how they all come together to help people.
I do this mostly because I tend to visualize a ‘problem space’ as a flat 2D terrain. It is a convenient oversimplification. It is a big, wide, open, empty field of grass which spans over related problems.
When I am doing greenfield work, I see the start as picking one spot in that field. You start there, building up enough structure to be useful. First, you lay down some common foundations, then you start adding in functionality that implements the features you know will help solve the problem.
As you do this, people will see the effort and start making suggestions. Some will want to go off in one direction, while others will prioritize the opposite way.
The trick to keeping it all as usable as possible is to slowly expand out your borders, but not in too many directions all at once.
Someone once told me that for software, you should never pick a path unless you are willing to walk it. From this perspective, it usually means that you won’t expand into another area in the field haphazardly. If you do choose to go there, it needs to be done correctly. That is, adding a few really good, solid features is way better than adding a million lame ones.
The same is true for the data. If you need new data, you add it carefully, properly structured, or not at all.
Overall, though, you start at one specific spot and keep growing. If it’s a good set of programs and people find it valuable, you’ll probably be at it for years, if not decades. So, it’s really crucial to its lifespan that the work you do in the very early days is as good as it can be. It needs to be neat, tidy, organized, and carefully thought out.
With that in mind, calling the current work and any of the rather obvious future work a ‘system’ works quite well. The system isn’t the code, but rather it is all of the territory that the code is trying to cover at some point. You might build a system for handling the account problems in a large corporation, for example. There might be lots of included pieces, and even some nearly stand-alone sub-systems, but they are all trying to fit together to deal with the same problems.
So, it’s similar to seeing the forest through the trees. The boundaries of the expected work are the system, but the system may not stretch right up to those boundaries yet.
From this view, it makes it easier to understand a bottom-up implementation. You might not know all of the features that people will ask for, but you should have a reasonable sense of the territory you are covering right now. Lots of that territory is similar, so building reusable components and engines will really help in getting more ground covered at a faster rate.
The classic example is reporting. You know people will need it at some point, so instead of just hardcoding a couple of static examples, it would be better to either offload it to somewhere else as exported data, or write some generic engine that is flexible enough to cover its rather massive width. The trick is not to write a lot of code, but rather to leverage any code you do write to cover the largest parts of that territory. In software, a little foresight goes a long way.
Thinking of a software system this way really helps in making a lot of the implementation decisions. If your field doesn’t cover having a million users, then designing an architecture to support that scale doesn’t make any sense.
More importantly, if you are located in one corner of the field, then trying to expand way over to the other side of a nearby hill doesn’t make a lot of sense either. That’s far enough away that it is clearly another system, another project and thus another codebase.
Building even medium-sized software is surprisingly complicated, so finding ways to frame it nicely really helps with making better decisions. Since time is a precious resource, getting to the right code as quickly as possible is important. Seeing it all as a system occupying some territory in an endless field is a good guide.
Thursday, June 4, 2026
Round Holes
A classic expression describes shoving a square peg into a round hole. Basically, you’re matching the wrong part with the wrong location.
I see this often in software architecture. Sometimes I like to use the term impedance mismatch. There is a component that someone is suggesting for use in a different variation of its problem space. It fits badly.
Sometimes the issue is access. There are hard limits on the availability of stacks, libraries, frameworks, tools, and services. In some organizations, they need to be vetted and approved first. This can be very slow, but I still think a good idea; using too many technologies is a nightmare.
In some cases, it is knowledge. People tend to gravitate to the things they already know, so they’ll prefer technologies they’ve used in the past, even if it means forcing them into place. That makes little sense if the architect has that preference, but it’s a different development team that does the construction.
Sometimes it is a misunderstanding. The marketing for the component says it will do everything, perfectly, but the reality is that it is far more than a stretch. Hastily cobbled together features to help sales. Still, the people funding the effort got swayed, so now everyone else is forced to jam weak pieces into the wrong place.
The habit I’ve always encouraged is to look carefully and admit that a round hole is, in fact, just a round hole.
I see this often in software architecture. Sometimes I like to use the term impedance mismatch. There is a component that someone is suggesting for use in a different variation of its problem space. It fits badly.
Sometimes the issue is access. There are hard limits on the availability of stacks, libraries, frameworks, tools, and services. In some organizations, they need to be vetted and approved first. This can be very slow, but I still think a good idea; using too many technologies is a nightmare.
In some cases, it is knowledge. People tend to gravitate to the things they already know, so they’ll prefer technologies they’ve used in the past, even if it means forcing them into place. That makes little sense if the architect has that preference, but it’s a different development team that does the construction.
Sometimes it is a misunderstanding. The marketing for the component says it will do everything, perfectly, but the reality is that it is far more than a stretch. Hastily cobbled together features to help sales. Still, the people funding the effort got swayed, so now everyone else is forced to jam weak pieces into the wrong place.
The habit I’ve always encouraged is to look carefully and admit that a round hole is, in fact, just a round hole.
I don’t start with the square pegs; that is my last step. Not surprisingly, this can frustrate people, in that there might not be any round pegs available. They don’t exist or can’t be used. For me, though, I still want to know that the hole is round, even if I can’t fill it correctly right now, or in some cases, ever.
But if you can do that, and imagine for any bunch of holes what would fit rather perfectly, first, before getting lost in the messiness, it will really help both simplify the effort and get it as good as possible.
But if you can do that, and imagine for any bunch of holes what would fit rather perfectly, first, before getting lost in the messiness, it will really help both simplify the effort and get it as good as possible.
Otherwise, you risk unintentionally creating a Rube Goldberg machine. For people unfamiliar with those machines, they are works of art that are deliberately overcomplicated. Just a collection of mismatching pieces made to do something interesting. They make great entertainment, but are not something that you’d want to have to rely on.
I’ve seen that too often in enterprise architecture, systems built out of odd, mismatching components sloppily glued together into a giant house of cards. That, paired with excessive brute force for the glue, tends to generate an endless amount of support and bug fixing, while never really working correctly. The system exists, but it is just off by enough that it would be better if it didn’t. It’s a time sink. Now, instead of a solution, it is an ill-placed speed bump.
Often, to avoid that fate, I want to just look at the way the data needs to flow around at the high, rather abstract level.
I’ve seen that too often in enterprise architecture, systems built out of odd, mismatching components sloppily glued together into a giant house of cards. That, paired with excessive brute force for the glue, tends to generate an endless amount of support and bug fixing, while never really working correctly. The system exists, but it is just off by enough that it would be better if it didn’t. It’s a time sink. Now, instead of a solution, it is an ill-placed speed bump.
Often, to avoid that fate, I want to just look at the way the data needs to flow around at the high, rather abstract level.
You need to get the major entities from other sources, persist it all, and then deliver to interfaces, reporting, other systems, etc. If you understand the amount of information, its timeliness, and frequency, you start to get a sense of the minimum pegs you need below it. If you can grok that, then you can start the torturous phase of trying to see what is actually available and whether or not it is close enough to be workable. But if you go the other way and pick the components first, you’ll quickly get lost in gluing together odd parts for no reason.
It’s the same form of thinking that is needed to get good, simple, clean code, too. You have to see it first from a top-down perspective, before trying to build it up from what’s already available. It’s really the only way to keep from getting lost, but also to leverage reuse, encapsulation, etc. You want to know the scope of the problem first, come up with a near-perfect solution, and then map that impossible solution back to things that are possible. It probably sounds a bit crazy to people who can’t see it that way, but it is a perspective that anyone can learn to leverage. A superpower of sorts.
So we can get there with three easy questions.
It’s the same form of thinking that is needed to get good, simple, clean code, too. You have to see it first from a top-down perspective, before trying to build it up from what’s already available. It’s really the only way to keep from getting lost, but also to leverage reuse, encapsulation, etc. You want to know the scope of the problem first, come up with a near-perfect solution, and then map that impossible solution back to things that are possible. It probably sounds a bit crazy to people who can’t see it that way, but it is a perspective that anyone can learn to leverage. A superpower of sorts.
So we can get there with three easy questions.
- What is the ‘full’ scope of the problem?
- What would solve this perfectly?
- What’s available to approximate that perfect solution?
In an enterprise that might be building up a replacement system for tracking some type of inventory or case management. The primary features are pretty well known; the useful secondary ones are findable with a bit of investigation.
Perfection might be a dynamic data store to accommodate wide but slowly changing shallow data. The users need a nice GUI to get at this and keep control. The incoming data is real-time, vibrates occasionally, so a queue would protect it and help with integrity. The system feeds a few others that specialize in other forms of management. It’s always a smallish number of people. It should all run in a managed environment.
This then is the hole that needs to be filled in with whatever technologies are available now, in the future, or can be suitably crafted in a “reasonable” time.
Contrast that with something where the data rarely changes, there are millions of users constantly accessing it, and they are the primary source of the data. It’s a very different hole that likely needs industrial-strength pegs in order to keep it going. It’s not a system running on one or two boxes, but requires a large cluster of machines all cooperating to cope with its huge and variable load. The scale is so large that there is no overlap with that first medium system, so it’s unlikely that they should share any common technologies. It’s more of a star-shaped hole, needs special stuff to fill it.
The converse is also true, in that any of the technologies suitable for the second design would be grossly over-engineered for the first one. You can’t just cherry-pick a few and shove them into place. One is a 2D circle that needs to be painted, the other is a 3D hole that needs to be filled.
In that sense, you learn as much as you can about the full width of the problem, then let your imagination run wild with getting it perfect. With those boundaries in place, you can start picking the fewest number of pieces that come close to filling it. There will be ugliness and rough edges, but you’ve found them early and minimized them, which is the best you can do if you can’t just build it all from the metal to the top.
Perfection might be a dynamic data store to accommodate wide but slowly changing shallow data. The users need a nice GUI to get at this and keep control. The incoming data is real-time, vibrates occasionally, so a queue would protect it and help with integrity. The system feeds a few others that specialize in other forms of management. It’s always a smallish number of people. It should all run in a managed environment.
This then is the hole that needs to be filled in with whatever technologies are available now, in the future, or can be suitably crafted in a “reasonable” time.
Contrast that with something where the data rarely changes, there are millions of users constantly accessing it, and they are the primary source of the data. It’s a very different hole that likely needs industrial-strength pegs in order to keep it going. It’s not a system running on one or two boxes, but requires a large cluster of machines all cooperating to cope with its huge and variable load. The scale is so large that there is no overlap with that first medium system, so it’s unlikely that they should share any common technologies. It’s more of a star-shaped hole, needs special stuff to fill it.
The converse is also true, in that any of the technologies suitable for the second design would be grossly over-engineered for the first one. You can’t just cherry-pick a few and shove them into place. One is a 2D circle that needs to be painted, the other is a 3D hole that needs to be filled.
In that sense, you learn as much as you can about the full width of the problem, then let your imagination run wild with getting it perfect. With those boundaries in place, you can start picking the fewest number of pieces that come close to filling it. There will be ugliness and rough edges, but you’ve found them early and minimized them, which is the best you can do if you can’t just build it all from the metal to the top.
Thursday, May 28, 2026
Versioning
If you start from the premise that a system is just a series of access points into a vast array of computations, then if you accept that there will always be a huge number of changes to this underlying code, you see why this is messy.
All the computer is really doing is taking a bunch of inputs, grinding through computation, then spitting it out. But we often end up changing these computations, sometimes because we had obvious or subtle bugs, sometimes because we’ve acquired new knowledge about how to do the work better, more accurately, or much faster.
At the high level, people interact with all of these access points, apply some variability to them, and then set the computations in motion. It might take a millisecond, an hour, or even a few days. The interaction might be rapid (real-time-ish), or it might just be infrequent. What is important to these people is that they can trust that the computer does the thing that they expect it to do. Trust is the bedrock.
We can skip over any sort of difference; the output is either a blob of text or a pretty little graphic of some type, it’s all just variations on presentation. The text could be typed and structured, which doesn’t matter either.
In order to trust the system (app, program, plugin, etc.), one key property is that the behaviour has to be ‘stable’. It should not change day to day, hour to hour. If you used it yesterday to do something, then you expect that with the same inputs, it will do pretty much the exact same thing (determinism). An added expectation is that if there were changes, then mostly those changes would be adding more features, not changing the old ones.
This is the core of what we call backward compatibility. Most programmers think of it in terms of APIs they are calling that are stable, but really, it is an overall property of the system itself. It is backward compatible if and only if any interactions, code, or humans are fully preserved after endless updates. If it worked ten years ago, it will work exactly the same today.
There is a loose exception for bug fixes, particularly bugs that have rendered the functionality to be useless. These are obviously not expected to be backward compatible, as the old behaviour does not correctly match expectations, so it needs to be changed to something else.
This relation is expressed nicely in using three-digit version numbers.
The last digit is bumped up for 1 or more bug fixes. Going from n.n.10 to n.n.11 means that at least one bug was fixed, maybe a dozen of them.
The second last digit is listed as a minor enhancement, but honestly, it is really just there for added functionality. Nothing else changed, nothing was deleted. So, if n.10.n is bumped up to n.11.n you expect backward compatibility for all of the existing functionality, and there is now some new functionality included.
That leaves the first digit to clearly state that you have broken backward compatibility. It is essentially a flag. There is some change that is major enough that the user is not necessarily going to be able to expect the code to be deterministic. Something big changed, and it will be noticed.
If people were strict in the usage of version numbers, and if they respected the notion that any 0.n.n version was just an initial demo or test instance, then even if the system was under active development for years, if it was backward compatible, then 1.9343523.14 would be a reasonable version number. 9M times new features were added, but the rest is still intact. Lots of stuff has been added, but all of it is backward compatible. The last round of added features needed 14 tries to get the bugs knocked out. All of these should have been in testing.
As it is with user interfaces, it is true for any pure computational dependencies below. Libraries, frameworks, languages, tools, etc. Strict usage of the version numbers is enough to get a very strong sense of both how the development is going and whether the authors even understand backward compatibility.
Probably the most embarrassing self-inflicted mistake a software developer can make is to push out a release that immediately crashes due to an underlying dependency change. If they were doing things reasonably, this would never occur. At minimum, an embarrassing non-backward-compatible library change would get picked up in testing. Untested code should never, ever get into a release. Subtle changes could slip through, but at the bare minimum, the work should have been smoke tested to catch exactly this sort of mistake. But the stronger habit is to only upgrade questionable libraries at the beginning of a long development cycle, while also doing lots of non-destructive refactoring. That is, before dumping in new stuff, you tidy up the junk from the last release and update some of the libraries. Run it a lot yourself until you are sure it is stable, then you go to town to add new stuff.
If the library is any good, and it has done an excellent job at being backward compatible, this is extremely low risk. You can kinda cheat the game sometimes. But if it is some dodgy little thing written by a couple of people as an advertising attempt, then you would have to wrap it in very expensive testing for each and every little thing you’ve used it for. It’s this that makes most libraries not worth integrating, either because the testing is too much work or the risk is just way too high. Reading the code and applying some of its better ideas is more suitable.
Often, you can get a sense of the quality of the library just by looking at the version numbers. For instance, 24.3.2 is a suspicious number if the work is only a couple of years old. They’re not taking backward compatibility very seriously; they are high risk.
It comes across with some of the larger tech stacks, too. If there is a major version bump that fundamentally breaks all backward compatibility, but someone has the newer and older versions haphazardly laid on top of each other, you pretty much know that the confusion caused by being too loose with the versioning is going to cause a lot of chaos that will either waste a lot of your time or result in embarrassing bugs. If the break was wide enough, the new work should really abandon the ‘brand’ of the old work. They are two different things, even if that means it is harder now for the new version to get a lot of traction. Just because you decided to change it doesn’t mean everyone else in the world should change too. Once you’ve committed to a particular set of computations, you have to stay committed and only grow from there. You can’t just pick up and move to some other spot farther away and claim it is the same work; it is not.
Backward compatibility is hard, really hard, which is why everyone loves to cheat the game so much. But it is an essential property of stability, which is necessary for trust. If you want to do a good job providing some complex computations to others, it is going to be hard. There is no way to avoid it. If you do the hard work, then you can communicate it quite clearly with the version numbers. That will let people know that your work is serious.
All the computer is really doing is taking a bunch of inputs, grinding through computation, then spitting it out. But we often end up changing these computations, sometimes because we had obvious or subtle bugs, sometimes because we’ve acquired new knowledge about how to do the work better, more accurately, or much faster.
At the high level, people interact with all of these access points, apply some variability to them, and then set the computations in motion. It might take a millisecond, an hour, or even a few days. The interaction might be rapid (real-time-ish), or it might just be infrequent. What is important to these people is that they can trust that the computer does the thing that they expect it to do. Trust is the bedrock.
We can skip over any sort of difference; the output is either a blob of text or a pretty little graphic of some type, it’s all just variations on presentation. The text could be typed and structured, which doesn’t matter either.
In order to trust the system (app, program, plugin, etc.), one key property is that the behaviour has to be ‘stable’. It should not change day to day, hour to hour. If you used it yesterday to do something, then you expect that with the same inputs, it will do pretty much the exact same thing (determinism). An added expectation is that if there were changes, then mostly those changes would be adding more features, not changing the old ones.
This is the core of what we call backward compatibility. Most programmers think of it in terms of APIs they are calling that are stable, but really, it is an overall property of the system itself. It is backward compatible if and only if any interactions, code, or humans are fully preserved after endless updates. If it worked ten years ago, it will work exactly the same today.
There is a loose exception for bug fixes, particularly bugs that have rendered the functionality to be useless. These are obviously not expected to be backward compatible, as the old behaviour does not correctly match expectations, so it needs to be changed to something else.
This relation is expressed nicely in using three-digit version numbers.
The last digit is bumped up for 1 or more bug fixes. Going from n.n.10 to n.n.11 means that at least one bug was fixed, maybe a dozen of them.
The second last digit is listed as a minor enhancement, but honestly, it is really just there for added functionality. Nothing else changed, nothing was deleted. So, if n.10.n is bumped up to n.11.n you expect backward compatibility for all of the existing functionality, and there is now some new functionality included.
That leaves the first digit to clearly state that you have broken backward compatibility. It is essentially a flag. There is some change that is major enough that the user is not necessarily going to be able to expect the code to be deterministic. Something big changed, and it will be noticed.
If people were strict in the usage of version numbers, and if they respected the notion that any 0.n.n version was just an initial demo or test instance, then even if the system was under active development for years, if it was backward compatible, then 1.9343523.14 would be a reasonable version number. 9M times new features were added, but the rest is still intact. Lots of stuff has been added, but all of it is backward compatible. The last round of added features needed 14 tries to get the bugs knocked out. All of these should have been in testing.
As it is with user interfaces, it is true for any pure computational dependencies below. Libraries, frameworks, languages, tools, etc. Strict usage of the version numbers is enough to get a very strong sense of both how the development is going and whether the authors even understand backward compatibility.
Probably the most embarrassing self-inflicted mistake a software developer can make is to push out a release that immediately crashes due to an underlying dependency change. If they were doing things reasonably, this would never occur. At minimum, an embarrassing non-backward-compatible library change would get picked up in testing. Untested code should never, ever get into a release. Subtle changes could slip through, but at the bare minimum, the work should have been smoke tested to catch exactly this sort of mistake. But the stronger habit is to only upgrade questionable libraries at the beginning of a long development cycle, while also doing lots of non-destructive refactoring. That is, before dumping in new stuff, you tidy up the junk from the last release and update some of the libraries. Run it a lot yourself until you are sure it is stable, then you go to town to add new stuff.
If the library is any good, and it has done an excellent job at being backward compatible, this is extremely low risk. You can kinda cheat the game sometimes. But if it is some dodgy little thing written by a couple of people as an advertising attempt, then you would have to wrap it in very expensive testing for each and every little thing you’ve used it for. It’s this that makes most libraries not worth integrating, either because the testing is too much work or the risk is just way too high. Reading the code and applying some of its better ideas is more suitable.
Often, you can get a sense of the quality of the library just by looking at the version numbers. For instance, 24.3.2 is a suspicious number if the work is only a couple of years old. They’re not taking backward compatibility very seriously; they are high risk.
It comes across with some of the larger tech stacks, too. If there is a major version bump that fundamentally breaks all backward compatibility, but someone has the newer and older versions haphazardly laid on top of each other, you pretty much know that the confusion caused by being too loose with the versioning is going to cause a lot of chaos that will either waste a lot of your time or result in embarrassing bugs. If the break was wide enough, the new work should really abandon the ‘brand’ of the old work. They are two different things, even if that means it is harder now for the new version to get a lot of traction. Just because you decided to change it doesn’t mean everyone else in the world should change too. Once you’ve committed to a particular set of computations, you have to stay committed and only grow from there. You can’t just pick up and move to some other spot farther away and claim it is the same work; it is not.
Backward compatibility is hard, really hard, which is why everyone loves to cheat the game so much. But it is an essential property of stability, which is necessary for trust. If you want to do a good job providing some complex computations to others, it is going to be hard. There is no way to avoid it. If you do the hard work, then you can communicate it quite clearly with the version numbers. That will let people know that your work is serious.
Thursday, May 21, 2026
Feedback
Recently, my blog has been getting a lot more views. Unfortunately, a lot of the incoming fields for these reads are just tagged with ‘Other’.
That tells me that the traffic is not coming from the older established sites I know, like HackerNews, but is either fake traffic or newer sites that I haven’t seen. It would be nice to know which is correct. Are people actually reading these posts?
So, if you are reading this, I’d really appreciate you taking a moment to comment. Anonymous is fine, and since my comments are screened before they are published, feel free to say ‘do not publish’ if you want. A ping is good; mentioning the source helps.
A long time ago, I briefly dreamed of monetizing my writing, but as I realized that the way to do that is to effectively change what I am saying, I decided not to do that.
I’ve always had to be careful not to upset any of my current employers, but beyond that, I write what I know, either from firsthand experience or from conversations with others.
Because of that, and my limited writing style, it’s never been a popular blog, but I still feel, after decades, that I want to get what I understand down somewhere. Maybe people read it, maybe not. It’s okay.
The software development industry varies hugely, so not surprisingly, plenty of other people have had very different experiences in their careers, but I also do suspect that there is way too much propaganda out there that is deliberately trying to mislead people. It’s an immature, messy and often ugly industry.
With all that in mind, if you could take a moment to say ‘Hi,’ at least I’ll know if you really exist or are just a figment of the web’s imagination.
That tells me that the traffic is not coming from the older established sites I know, like HackerNews, but is either fake traffic or newer sites that I haven’t seen. It would be nice to know which is correct. Are people actually reading these posts?
So, if you are reading this, I’d really appreciate you taking a moment to comment. Anonymous is fine, and since my comments are screened before they are published, feel free to say ‘do not publish’ if you want. A ping is good; mentioning the source helps.
A long time ago, I briefly dreamed of monetizing my writing, but as I realized that the way to do that is to effectively change what I am saying, I decided not to do that.
I’ve always had to be careful not to upset any of my current employers, but beyond that, I write what I know, either from firsthand experience or from conversations with others.
Because of that, and my limited writing style, it’s never been a popular blog, but I still feel, after decades, that I want to get what I understand down somewhere. Maybe people read it, maybe not. It’s okay.
The software development industry varies hugely, so not surprisingly, plenty of other people have had very different experiences in their careers, but I also do suspect that there is way too much propaganda out there that is deliberately trying to mislead people. It’s an immature, messy and often ugly industry.
With all that in mind, if you could take a moment to say ‘Hi,’ at least I’ll know if you really exist or are just a figment of the web’s imagination.
UPDATE: Ok, I got a few responses, which is great. Thanks! Seems like at least some of the traffic is RSS and Atom, which doesn't show up in the stats. It might be those views where I do get a country and browser type, but that still leaves a great deal of traffic as Other. I guess I'll never know if those are real or not.
If anyone has suggestions about future topics, that would be great too. I feel like I am getting too repetitive in my old age :-)
Thursday, May 14, 2026
Security
Programmers hate adding security to their systems.
First, it is a huge amount of work, and second, since it is so often left to the end, it is very ugly and disruptive work. A patchwork of hacks.
But it’s misdirected. Without enough security, the system they built is useless, well, worse than useless. If people use it, it could severely screw them over. Nobody would intentionally use something that helps criminals more than it helps them. Even if it is in a walled garden, you can never really be sure that someone isn’t motivated to take a peek.
It’s worth noting that I am not a security expert, and although I’ve had to deal with it a lot in my career, my practice might not be as strong as the experts would like. That being said, I’ll continue.
There are only a few things you need to worry about in security. First is actually identifying any ‘users’. You always have to know who they are and have enough confidence in that decision that you don’t make a mistake.
Then the other part of it is that you want to protect both the data and the code from anyone who isn’t supposed to see or activate it. It’s not enough to protect just the data or just the code. You need both.
In that sense, security isn’t that hard if it is your concern from day one. There are a bunch of entry points that people will use to get to the features and functionality. First, you identify them, then you check to see if they can access the given functionality. If they can, then lower, you check to see if they can access all of the data input into that functionality. If they can, then they can see the output. Simple 🙂
Here’s where the trouble starts. First, there should be no anonymous endpoints. But people love adding them, but they open the door to leaks or denial of service attacks. If you have none, though, all of that goes away. If you can’t quickly identify someone right at the top, punt them immediately, send a log to some administrator. They might have to block the incoming address or put up some firewalls to stop botnets and other nasty things. You always need to flag a punted user as a serious problem.
Second is databases. For capitalist reasons, they charge by users, so the system users are not the same as the database users. That sucks, and it has always sucked. Life would be pretty easy if a person’s identity propagated all the way down to the metal. It should.
If there was a necessity for a group or functional account shared by a bunch of people, then the group is the identity, and that identity goes all the way down.
If your database or its license makes that impossible, then you need to wrap it. You need to wrap it thoroughly enough that pretty much nobody can get to it in any way without first passing an identity check. So, not just in the backend code, but also on the machine in the scripts, with the OS, etc. Everywhere.
Wrap the database. It won’t make it convenient, since that is the opposite of secure, but you need to do it.
Now, at the top, after you have checked identity, you take a quick look at whatever functionality is called. Are they allowed to use it? In some extreme cases, that is a messy lookup table, and it needs to be managed by data admins. It’s annoying, but really, in a large organization, that really should be a distinct piece that is shared by a whole bunch of systems. You just check with it, user X wants to call foo, is that okay?
If that’s good, then as the code executes and hits the wrapped database, the second check will trigger on the data. If it’s good, then it is all done. If you always reuse both the high and lower levels, then the security will be everywhere, and you don’t have to lie awake at night worrying about it failing.
The only other part is that if a user ever sends you ‘code’, you laugh and reject it. If you want some cool dynamic execution feature, great, but there have to be two paths, not one. The code comes in from somewhere else, having been fully and completely vetted, and then the user later asks for it to execute dynamically. That keeps it really simple, and sets you up with some external means for this uber dangerous code to be properly managed, vetted, and approved. That in itself is a huge task; you can’t just ignore it and hope for the best. Dynamic code can never be ‘open’ dynamic code; it has to be closed and come from a reliable source that actually has to be more reliable than just reliable.
So, in the end, if you wrap the database, always identify everyone, manage a lookup table or two, and punt anything that could ever be possibly executed by any downstream party or library, then you’re done. All of this code is reusable; you just need to do it once, at the beginning of the project, then leverage it for success and glory.
First, it is a huge amount of work, and second, since it is so often left to the end, it is very ugly and disruptive work. A patchwork of hacks.
But it’s misdirected. Without enough security, the system they built is useless, well, worse than useless. If people use it, it could severely screw them over. Nobody would intentionally use something that helps criminals more than it helps them. Even if it is in a walled garden, you can never really be sure that someone isn’t motivated to take a peek.
It’s worth noting that I am not a security expert, and although I’ve had to deal with it a lot in my career, my practice might not be as strong as the experts would like. That being said, I’ll continue.
There are only a few things you need to worry about in security. First is actually identifying any ‘users’. You always have to know who they are and have enough confidence in that decision that you don’t make a mistake.
Then the other part of it is that you want to protect both the data and the code from anyone who isn’t supposed to see or activate it. It’s not enough to protect just the data or just the code. You need both.
In that sense, security isn’t that hard if it is your concern from day one. There are a bunch of entry points that people will use to get to the features and functionality. First, you identify them, then you check to see if they can access the given functionality. If they can, then lower, you check to see if they can access all of the data input into that functionality. If they can, then they can see the output. Simple 🙂
Here’s where the trouble starts. First, there should be no anonymous endpoints. But people love adding them, but they open the door to leaks or denial of service attacks. If you have none, though, all of that goes away. If you can’t quickly identify someone right at the top, punt them immediately, send a log to some administrator. They might have to block the incoming address or put up some firewalls to stop botnets and other nasty things. You always need to flag a punted user as a serious problem.
Second is databases. For capitalist reasons, they charge by users, so the system users are not the same as the database users. That sucks, and it has always sucked. Life would be pretty easy if a person’s identity propagated all the way down to the metal. It should.
If there was a necessity for a group or functional account shared by a bunch of people, then the group is the identity, and that identity goes all the way down.
If your database or its license makes that impossible, then you need to wrap it. You need to wrap it thoroughly enough that pretty much nobody can get to it in any way without first passing an identity check. So, not just in the backend code, but also on the machine in the scripts, with the OS, etc. Everywhere.
Wrap the database. It won’t make it convenient, since that is the opposite of secure, but you need to do it.
Now, at the top, after you have checked identity, you take a quick look at whatever functionality is called. Are they allowed to use it? In some extreme cases, that is a messy lookup table, and it needs to be managed by data admins. It’s annoying, but really, in a large organization, that really should be a distinct piece that is shared by a whole bunch of systems. You just check with it, user X wants to call foo, is that okay?
If that’s good, then as the code executes and hits the wrapped database, the second check will trigger on the data. If it’s good, then it is all done. If you always reuse both the high and lower levels, then the security will be everywhere, and you don’t have to lie awake at night worrying about it failing.
The only other part is that if a user ever sends you ‘code’, you laugh and reject it. If you want some cool dynamic execution feature, great, but there have to be two paths, not one. The code comes in from somewhere else, having been fully and completely vetted, and then the user later asks for it to execute dynamically. That keeps it really simple, and sets you up with some external means for this uber dangerous code to be properly managed, vetted, and approved. That in itself is a huge task; you can’t just ignore it and hope for the best. Dynamic code can never be ‘open’ dynamic code; it has to be closed and come from a reliable source that actually has to be more reliable than just reliable.
So, in the end, if you wrap the database, always identify everyone, manage a lookup table or two, and punt anything that could ever be possibly executed by any downstream party or library, then you’re done. All of this code is reusable; you just need to do it once, at the beginning of the project, then leverage it for success and glory.
Thursday, May 7, 2026
Goodness
It really feels like the world has sunk to its lowest point in my lifetime. And it does not seem likely to improve anytime soon. We’re sliding downhill.
When I first started playing with computers, way back in the 80s, I felt like they had huge potential to help humanity. To lift us up, but it seems like they did the opposite. First, they trapped us; now they are forcing us to regress.
It’s not the computers themselves, but the types of monsters that latch onto them in order to make money, grab power, and manipulate people. Sadly, they find it too easy to use software to do this.
It seems like software developers made a rather tragic mistake in not preventing this earlier. We were just too eager to build stuff; we didn’t ask enough questions.
But the good news is that we can still do something about it now. We can build all new stuff that is empathy-driven and meant to really help people, not just pick their pockets or force their behaviour.
The trick is not to get hyper-focused on the technology itself; it doesn’t really matter. Instead, we put ourselves into the shoes of the users. Software without empathy is just a weapon waiting to be exploited.
The problem has always been that empathy-driven software is extraordinarily hard to write. It’s not just getting the technology to dance, or flooding it with domain data; you have to integrate all that very carefully into the full context of a user's life in order to shave off any of the hard spikes. It’s not just code and data; it is code and data that deliberately help people. It all emanates from their perspective, not from the builders or the operators.
To get us going, I’d suggest that everyone just start throwing any “non-monetizable” ideas they have out there. Pick a problem you know, write up a dream solution, and publish it. It doesn’t have to be practical; it doesn’t even have to be possible. It’s not about technology, but about seeing the world from the user’s perspective and making suggestions about how to really improve it. Too often, we first focus on technology, and then we try to shove it back into the solution space. That doesn’t work very well.
Once we have ideas, we can figure out how to implement these as solutions in ways that can’t be subverted by monsters. That, of course, is the difficult part. Serious software is still very expensive to build and run, and the cost of getting it funded has a lot of painful strings that we’ve seen over and over again are used to pull the efforts off in very bad directions.
If we can figure that out, then we just need to find a way to swap these technologies with the mess we’ve got right now.
I’ve occasionally dumped out some raw ideas:
They were mostly unfundable, and since I had needed to pay the bills, they were beyond my ability to take further. But I’ve always thought it would be cool if someone was inspired to do something similar, so I wrote them up.
Other areas that desperately need our attention:
Source of Truth. I appreciate and admire Wikipedia, but I really want something more structured, like an ontology built on graphs or even hypergraphs, that contains all of human knowledge or at least as much as we can capture right now. It would assign a probability to any “knowledge”. For instance, a known mathematical proof would be 100% correct, but most other things we think are true are at best 99ish. And the myths and falsehoods are really low, maybe even 0. If there were multiple competing opinions, they would all exist in the data, but with some percentage of likely truth (as of today). It would be worldwide and not controllable by any country or dictator. Untouchable by monsters. A perfect use case for decentralization.
Privacy. We need to protect any facts about individuals, but still provide some (difficult) means of external verification. This would extend to group conversations as well. Some part of it would only allow retrospective external access if and only if the case made it to a territorial court accepted by all of the individuals. That is, they can’t spy on you, but if you did something bad in some jurisdiction that you have accepted, the information could be retrieved if there were actual court proceedings. It’s the notion that they have to do the policing legwork to catch you, but once you are in trouble, the whole truth will come out.
Time/Complexity Simulations. Being able to list out the consequences of a given decision over a complex circumstance. Lots of moving parts. You could throw together some approximate complexity for something, then play with any possible decisions to see how they fare in the long run. We need this, as too many people can’t see beyond extremely short horizons. Even if it is crude, it would help people think about more than just tomorrow, or next month, or next quarter. If you could come back with a chance that there is a 48.2% that “doing that” would turn the profits negative in the next seven years, it would be harder for someone to just forge ahead blindly. Or that there really are “century” events that we do need to protect against, like pandemics.
Consolidation. It sucks having to rely on dozens of different, widely inconsistent apps. Their collective value is eroded by the combined cognitive load. I’d want something simpler than a spreadsheet that brings together the common data and can trigger code in all sorts of remote places. A customized gateway that makes it easy to leverage the power of a computer, but just for you. The trick is to breach the complexity limits that so often hold us back. The abstraction that holds it together can’t be too abstract but still needs to be powerful enough that it is all-encompassing. If I could configure it for all of the repeatable parts of my life, like a crazy, distributed, super-integrated to-do list, with behaviours and data shareable for a wide range of scenarios, it would be my first point of contact on all my computers. It would have some deep way of reorganizing itself as I keep adding more to it. The key, though, is that it isn’t a remote service; you don’t rent it. You own it, it is yours, it can be seamlessly upgraded over the decades of your life, and it is fully private. The costs are trivial, but it will consume your time. There are parts you can share if you want, but there would never be a way to make money off your contributions; all you get for your efforts is a better life.
Guardrails. Lots of awful stuff happens on the web. Why? Why can’t we keep the good qualities of the Internet, without continuously opening doors for the bad ones? My guess is that capitalism drives an unquenchable thirst for monetization, so making that safe is just too costly. Eats into the profits. So we get half-baked stuff that eventually the monsters figure out how to leverage. From that perspective, it seems like we could put up some types of walls and fences that would protect this weak code from being exploited. Protect private data from going anywhere. You shouldn’t be subject to an attack unless you explicitly lowered your guard. It shouldn’t be possible to trick you into lowering your guard. All of the angst from this not being true today piles on the friction that devalues the capabilities of the computers. Finding a way to stop that is huge.
I’m sure there are a million more issues and ideas out there. Now is the time to flood the world with them, and then maybe we can figure out how to bring the best of them to life. If you do this, odds are you won’t get much credit, and it definitely won’t make you rich, but it is still a good thing to do, so it is worth the effort.
What we ultimately want is for computers to make our lives easier and more meaningful. To take away some of the drudgeries and difficulties of reality, but not numb us into a coma or stupor. Sure, we’ll still turn to the machines for assistance, but we won’t get caught in negative incentive loops like doom scrolling. We will live life in reality, not digitally.
To get this, we need to stop the people who are financially motivated from bending all of the technologies against us. They only see the bad potential, realize their use in carving off profits, and then find ways to slip these into our lives. They’re tearing us apart so they can own mansions, sports cars, jets, and yachts. We have to stop allowing this.
When I first started playing with computers, way back in the 80s, I felt like they had huge potential to help humanity. To lift us up, but it seems like they did the opposite. First, they trapped us; now they are forcing us to regress.
It’s not the computers themselves, but the types of monsters that latch onto them in order to make money, grab power, and manipulate people. Sadly, they find it too easy to use software to do this.
It seems like software developers made a rather tragic mistake in not preventing this earlier. We were just too eager to build stuff; we didn’t ask enough questions.
But the good news is that we can still do something about it now. We can build all new stuff that is empathy-driven and meant to really help people, not just pick their pockets or force their behaviour.
The trick is not to get hyper-focused on the technology itself; it doesn’t really matter. Instead, we put ourselves into the shoes of the users. Software without empathy is just a weapon waiting to be exploited.
The problem has always been that empathy-driven software is extraordinarily hard to write. It’s not just getting the technology to dance, or flooding it with domain data; you have to integrate all that very carefully into the full context of a user's life in order to shave off any of the hard spikes. It’s not just code and data; it is code and data that deliberately help people. It all emanates from their perspective, not from the builders or the operators.
To get us going, I’d suggest that everyone just start throwing any “non-monetizable” ideas they have out there. Pick a problem you know, write up a dream solution, and publish it. It doesn’t have to be practical; it doesn’t even have to be possible. It’s not about technology, but about seeing the world from the user’s perspective and making suggestions about how to really improve it. Too often, we first focus on technology, and then we try to shove it back into the solution space. That doesn’t work very well.
Once we have ideas, we can figure out how to implement these as solutions in ways that can’t be subverted by monsters. That, of course, is the difficult part. Serious software is still very expensive to build and run, and the cost of getting it funded has a lot of painful strings that we’ve seen over and over again are used to pull the efforts off in very bad directions.
If we can figure that out, then we just need to find a way to swap these technologies with the mess we’ve got right now.
I’ve occasionally dumped out some raw ideas:
https://theprogrammersparadox.blogspot.com/2024/05/identitynet.html
https://theprogrammersparadox.blogspot.com/2015/08/digital-discussions.html
https://theprogrammersparadox.blogspot.com/2009/04/end-of-coding-as-we-know-it.html
They were mostly unfundable, and since I had needed to pay the bills, they were beyond my ability to take further. But I’ve always thought it would be cool if someone was inspired to do something similar, so I wrote them up.
Other areas that desperately need our attention:
Source of Truth. I appreciate and admire Wikipedia, but I really want something more structured, like an ontology built on graphs or even hypergraphs, that contains all of human knowledge or at least as much as we can capture right now. It would assign a probability to any “knowledge”. For instance, a known mathematical proof would be 100% correct, but most other things we think are true are at best 99ish. And the myths and falsehoods are really low, maybe even 0. If there were multiple competing opinions, they would all exist in the data, but with some percentage of likely truth (as of today). It would be worldwide and not controllable by any country or dictator. Untouchable by monsters. A perfect use case for decentralization.
Privacy. We need to protect any facts about individuals, but still provide some (difficult) means of external verification. This would extend to group conversations as well. Some part of it would only allow retrospective external access if and only if the case made it to a territorial court accepted by all of the individuals. That is, they can’t spy on you, but if you did something bad in some jurisdiction that you have accepted, the information could be retrieved if there were actual court proceedings. It’s the notion that they have to do the policing legwork to catch you, but once you are in trouble, the whole truth will come out.
Time/Complexity Simulations. Being able to list out the consequences of a given decision over a complex circumstance. Lots of moving parts. You could throw together some approximate complexity for something, then play with any possible decisions to see how they fare in the long run. We need this, as too many people can’t see beyond extremely short horizons. Even if it is crude, it would help people think about more than just tomorrow, or next month, or next quarter. If you could come back with a chance that there is a 48.2% that “doing that” would turn the profits negative in the next seven years, it would be harder for someone to just forge ahead blindly. Or that there really are “century” events that we do need to protect against, like pandemics.
Consolidation. It sucks having to rely on dozens of different, widely inconsistent apps. Their collective value is eroded by the combined cognitive load. I’d want something simpler than a spreadsheet that brings together the common data and can trigger code in all sorts of remote places. A customized gateway that makes it easy to leverage the power of a computer, but just for you. The trick is to breach the complexity limits that so often hold us back. The abstraction that holds it together can’t be too abstract but still needs to be powerful enough that it is all-encompassing. If I could configure it for all of the repeatable parts of my life, like a crazy, distributed, super-integrated to-do list, with behaviours and data shareable for a wide range of scenarios, it would be my first point of contact on all my computers. It would have some deep way of reorganizing itself as I keep adding more to it. The key, though, is that it isn’t a remote service; you don’t rent it. You own it, it is yours, it can be seamlessly upgraded over the decades of your life, and it is fully private. The costs are trivial, but it will consume your time. There are parts you can share if you want, but there would never be a way to make money off your contributions; all you get for your efforts is a better life.
Guardrails. Lots of awful stuff happens on the web. Why? Why can’t we keep the good qualities of the Internet, without continuously opening doors for the bad ones? My guess is that capitalism drives an unquenchable thirst for monetization, so making that safe is just too costly. Eats into the profits. So we get half-baked stuff that eventually the monsters figure out how to leverage. From that perspective, it seems like we could put up some types of walls and fences that would protect this weak code from being exploited. Protect private data from going anywhere. You shouldn’t be subject to an attack unless you explicitly lowered your guard. It shouldn’t be possible to trick you into lowering your guard. All of the angst from this not being true today piles on the friction that devalues the capabilities of the computers. Finding a way to stop that is huge.
I’m sure there are a million more issues and ideas out there. Now is the time to flood the world with them, and then maybe we can figure out how to bring the best of them to life. If you do this, odds are you won’t get much credit, and it definitely won’t make you rich, but it is still a good thing to do, so it is worth the effort.
What we ultimately want is for computers to make our lives easier and more meaningful. To take away some of the drudgeries and difficulties of reality, but not numb us into a coma or stupor. Sure, we’ll still turn to the machines for assistance, but we won’t get caught in negative incentive loops like doom scrolling. We will live life in reality, not digitally.
To get this, we need to stop the people who are financially motivated from bending all of the technologies against us. They only see the bad potential, realize their use in carving off profits, and then find ways to slip these into our lives. They’re tearing us apart so they can own mansions, sports cars, jets, and yachts. We have to stop allowing this.
Thursday, April 30, 2026
Shortcuts and Makework
On the face of it, shortcuts and makework may seem like they are opposites.
A shortcut is a faster way to do something that effectively pushes out the consequences down the road. You take the quick and easy way now, only to pay for it later.
Makework, on the other hand, is anything that you are made to do that does not directly or indirectly contribute to the work at hand. For instance, you fill out a complicated form with copious details that is ultimately ignored forever.
Makework is usually some people trying to control or throttle others, often an abuse of power, or a justification of their value.
In bureaucratic organizations, the centralized control over poorly understood aspects of the company is usually thick with makework. There are plenty of administrators trying to control things that they do not understand. Thus, the rules and processes get weird and form the basis for lots of politics.
But the two are oddly related. Where you see one, you usually see the other.
The root cause is time. There is a project that has a tight timeline. But the people working keep losing huge blocks of time to makework. However, as makework is an integral part of the organization, blaming it for being late is not allowed. So, in order to try to catch up, they resort to a lot of shortcuts. The long-term consequences don’t matter if, in the short term, you will get in trouble for being late. The context of the project forces mistakes and panic.
It gets triggered the other way, too. Some people just take shortcuts out of habit; the project looks initially crazy successful. But as the long-term consequences come due, it collapses. In the downfall, lots of unrelated people jump in to “help”, like bureaucrats and generic management. Since they don’t understand and they don’t know why a once successful effort suddenly flipped, they propose a lot of work that they believe will fix the problem. More tracking, more documentation, more sign-offs, more meetings, etc. But all of this is effectively makework, and the real problem of replacing the shortcuts causing all of the grief gets ignored. The project ends up under the microscope, which amplifies its problems and does not correct them.
Mostly, though, the best approach is to be rigorously practical. Minimize both shortcuts and makework. Carefully assess any and all effort with respect to both of those categories. If it smells like one or the other, don’t do it. Get the core work done as best as possible.
The other part is to tackle the hardest parts first, don’t leave them for later, and don’t rush through them. While that gives the appearance of being late right from the get-go, it provides two valuable properties.
First, if you get stuck, you can raise a late flag early, rather than later, which tends to mitigate some of the bureaucrats coming out of the woodwork and drowning you with makework.
Second is that a shortcut on the hard stuff is way more destructive than a shortcut on the easy stuff. If time forces you to take shortcuts, then the ones with minimal consequences are far better. They are less costly to repair later. If the foundations are solid, you have a better shot of recovery when late. In many organizations, you are already late long before you even realize that you have work to do. It’s normal, so you need to adopt habits that mitigate it.
The biggest problem, though, is that coming up with shortcuts or makework is often a lot easier than doing things properly. It’s the easy path for both the workers and management. But it is an unsuccessful path too. It distracts from the things that really need to get done.
Ultimately, there is some work that needs to be finished with at least enough quality to keep the detractors at bay. Do that work, don’t get lost by trying to avoid it.
A shortcut is a faster way to do something that effectively pushes out the consequences down the road. You take the quick and easy way now, only to pay for it later.
Makework, on the other hand, is anything that you are made to do that does not directly or indirectly contribute to the work at hand. For instance, you fill out a complicated form with copious details that is ultimately ignored forever.
Makework is usually some people trying to control or throttle others, often an abuse of power, or a justification of their value.
In bureaucratic organizations, the centralized control over poorly understood aspects of the company is usually thick with makework. There are plenty of administrators trying to control things that they do not understand. Thus, the rules and processes get weird and form the basis for lots of politics.
But the two are oddly related. Where you see one, you usually see the other.
The root cause is time. There is a project that has a tight timeline. But the people working keep losing huge blocks of time to makework. However, as makework is an integral part of the organization, blaming it for being late is not allowed. So, in order to try to catch up, they resort to a lot of shortcuts. The long-term consequences don’t matter if, in the short term, you will get in trouble for being late. The context of the project forces mistakes and panic.
It gets triggered the other way, too. Some people just take shortcuts out of habit; the project looks initially crazy successful. But as the long-term consequences come due, it collapses. In the downfall, lots of unrelated people jump in to “help”, like bureaucrats and generic management. Since they don’t understand and they don’t know why a once successful effort suddenly flipped, they propose a lot of work that they believe will fix the problem. More tracking, more documentation, more sign-offs, more meetings, etc. But all of this is effectively makework, and the real problem of replacing the shortcuts causing all of the grief gets ignored. The project ends up under the microscope, which amplifies its problems and does not correct them.
Mostly, though, the best approach is to be rigorously practical. Minimize both shortcuts and makework. Carefully assess any and all effort with respect to both of those categories. If it smells like one or the other, don’t do it. Get the core work done as best as possible.
The other part is to tackle the hardest parts first, don’t leave them for later, and don’t rush through them. While that gives the appearance of being late right from the get-go, it provides two valuable properties.
First, if you get stuck, you can raise a late flag early, rather than later, which tends to mitigate some of the bureaucrats coming out of the woodwork and drowning you with makework.
Second is that a shortcut on the hard stuff is way more destructive than a shortcut on the easy stuff. If time forces you to take shortcuts, then the ones with minimal consequences are far better. They are less costly to repair later. If the foundations are solid, you have a better shot of recovery when late. In many organizations, you are already late long before you even realize that you have work to do. It’s normal, so you need to adopt habits that mitigate it.
The biggest problem, though, is that coming up with shortcuts or makework is often a lot easier than doing things properly. It’s the easy path for both the workers and management. But it is an unsuccessful path too. It distracts from the things that really need to get done.
Ultimately, there is some work that needs to be finished with at least enough quality to keep the detractors at bay. Do that work, don’t get lost by trying to avoid it.
Thursday, April 23, 2026
Users
A common misconception in software development comes from not understanding users.
For any piece of software, there is a bunch of primary users who are using it to solve their problems. This ranges from commercial product usages all the way to large enterprise usages.
If the software is large and has been evolving for quite some time, these usages are often partitioned into different subgroups. Some use one feature set, others use a different one. Some users are tied directly to their group, and others will overlap between a bunch of groups.
There is usually a second set of users, whose primary tasks are system management, usually related to the data in the software, but sometimes configuration, access, or feature capabilities. Occasionally referred to as data administrators.
They most often sit outside of the primary users and do not use the software to solve those problems; they are just involved in making sure the software itself is capable of allowing the other users to get their work done.
Totally ignored, there is actually a third set of users. These users take the software, install it on the fundamental hardware, and interact with it when there are problems. Sometimes they don’t even know what the software does, but they are still responsible for providing the platform for the software to exist and fulfil its role.
For any piece of software, there is a bunch of primary users who are using it to solve their problems. This ranges from commercial product usages all the way to large enterprise usages.
If the software is large and has been evolving for quite some time, these usages are often partitioned into different subgroups. Some use one feature set, others use a different one. Some users are tied directly to their group, and others will overlap between a bunch of groups.
There is usually a second set of users, whose primary tasks are system management, usually related to the data in the software, but sometimes configuration, access, or feature capabilities. Occasionally referred to as data administrators.
They most often sit outside of the primary users and do not use the software to solve those problems; they are just involved in making sure the software itself is capable of allowing the other users to get their work done.
Totally ignored, there is actually a third set of users. These users take the software, install it on the fundamental hardware, and interact with it when there are problems. Sometimes they don’t even know what the software does, but they are still responsible for providing the platform for the software to exist and fulfil its role.
It used to be that this was classified as an Operations Department, but over the decades, there have often been a lot of Software Developers directly involved as well. These are users, too. They should have their own ops, test, or development accounts; they have complex access issues, and sometimes they have to be able to get into the software to determine that it is working or malfunctioning in a specific way, but most times should not be able to see what the data administrators can see.
Traditionally, people have not designated operations as users, which is a common mistake. They do “use” the software, and getting their work done is also dependent on it. It’s just that they are not focused on using the specific features or managing the data.
If you take a step back, then user requirements, and even user stories, should cover all three groups, not just the first or second one.
But it’s also true that if an enterprise has hundreds of software packages running, from an operations perspective, all of them share a large number of common requirements. They all need to be installed and upgraded, they all need to be monitored, and they all need some form of smoke tests for emergencies. An operations dept could put out a list of mandatory common user requirements long before any specific software project was a twinkle in someone’s eye.
What’s also true is that for the most part, these types of requirements do not change significantly with most tech stacks. The specifics may change a bit, but the base requirement is locked in stone.
This is a rather classic mistake that happens with new tech stacks or operations environments. Because they are new, everyone thinks they can start over again from scratch and ignore any previous round of complexities. However, once things get going, those same complexities come back to haunt the effort, and because they were ignored, they get handled extra poorly.
So, we see people put up software systems without any adequate monitoring, for example, and are surprised when the users complain about the system being down. Pushing the monitoring back onto the first and second user groups is common now, but it still makes the effort look rather amateur. Operations should be the first to know about a crash; they just can’t detect more subtle bugs buried deep under big features.
The users of software are anybody who interacts directly with that software, in any way. Non-users, while they still may be “stakeholders” in the effort, will never run the software, test it, log into it, or trigger some of the features.
Someone may be responsible for making sure the software project gets done on time, but if they do not interact with the software, they are not a user.
User requirements should have special priority above almost all other aspects of the work. They would only take a back seat when there are overriding cost or time constraints. But it really should be written down somewhere that the users did not get the feature or functionality they needed due to the enforcement of these constraints. At minimum, that builds up a wonderful list of future enhancements that should be considered as early in the effort as possible. The key point, though, is knowing what the user’s need in order to actually solve their problems is different than the specifics of a technical solution implementation.
Traditionally, people have not designated operations as users, which is a common mistake. They do “use” the software, and getting their work done is also dependent on it. It’s just that they are not focused on using the specific features or managing the data.
If you take a step back, then user requirements, and even user stories, should cover all three groups, not just the first or second one.
But it’s also true that if an enterprise has hundreds of software packages running, from an operations perspective, all of them share a large number of common requirements. They all need to be installed and upgraded, they all need to be monitored, and they all need some form of smoke tests for emergencies. An operations dept could put out a list of mandatory common user requirements long before any specific software project was a twinkle in someone’s eye.
What’s also true is that for the most part, these types of requirements do not change significantly with most tech stacks. The specifics may change a bit, but the base requirement is locked in stone.
This is a rather classic mistake that happens with new tech stacks or operations environments. Because they are new, everyone thinks they can start over again from scratch and ignore any previous round of complexities. However, once things get going, those same complexities come back to haunt the effort, and because they were ignored, they get handled extra poorly.
So, we see people put up software systems without any adequate monitoring, for example, and are surprised when the users complain about the system being down. Pushing the monitoring back onto the first and second user groups is common now, but it still makes the effort look rather amateur. Operations should be the first to know about a crash; they just can’t detect more subtle bugs buried deep under big features.
The users of software are anybody who interacts directly with that software, in any way. Non-users, while they still may be “stakeholders” in the effort, will never run the software, test it, log into it, or trigger some of the features.
Someone may be responsible for making sure the software project gets done on time, but if they do not interact with the software, they are not a user.
User requirements should have special priority above almost all other aspects of the work. They would only take a back seat when there are overriding cost or time constraints. But it really should be written down somewhere that the users did not get the feature or functionality they needed due to the enforcement of these constraints. At minimum, that builds up a wonderful list of future enhancements that should be considered as early in the effort as possible. The key point, though, is knowing what the user’s need in order to actually solve their problems is different than the specifics of a technical solution implementation.
Thursday, April 16, 2026
Strange Loops
I remember when I was in school; we got a difficult programming assignment that I struggled with. The algorithm, if I remember correctly, was to fill an oddly shaped virtual swimming pool at different levels, and then calculate the volume of water. The bottom of the pool was wavy curves.
The most natural way to write the code to simulate adding little bits of water at a time to the pool was with recursion. No doubt, you could unroll that into a loop and maybe a stack or two, but we were supposed to use recursion; it was the key to the assignment.
I had never encountered recursion before, and I found it mystifying. A function calls itself? That just seemed rather odd and crazy. Totally non-intuitive. I coded it up, but it never really worked properly. I got poor marks on that assignment.
Many months later, the light went on. It came out of nowhere, and suddenly, not only did recursion make sense, but it also seemed trivial. Over the decades, I’ve crafted some pretty intense recursive algorithms for all sorts of complex problems. It now comes intuitively, and I find it far easier to express the answer recursively than to unroll it.
Recursion is, I think, a simple version of what Douglas Hofstadter refers to in GEB as a ‘strange loop’; a paradox is a more advanced variety. For me, it is a conceptual understanding that sorts of acts like a cliff. When you are at the bottom, it makes no sense; it seems like a wall, but if you manage to climb up, it becomes obvious.
There are all sorts of strange loops in programming. Problems where the obvious first try is guaranteed to fail, yet there is another simpler way to encode the logic that always works.
A good example is parsing. The world is littered with what I call string-split parsers. They are the most intuitive way to decompose some complex data. You just start chopping the data into pieces, then you look at those pieces and react. For very simple data that works fine, but if you fall into programming languages, or pretty much anywhere where there are some inherent ambiguities, it will fail miserably.
But all you really need to do to climb this cliff is read the Green Dragon book. It gives you all of the practical knowledge you need to implement a parser, but also to understand any of the complicated parser generators, like Antlr or Yacc.
I guess the cool part is that once you have encountered and conquered a particular strange loop, that knowledge is so fundamental that it transcends tech stacks. If you can write a solid parser in C, you can write it in any language. If you understand how to write parsers, you can learn any programming language way faster. And nothing about that knowledge will radically change over the course of your career. You’ll jump around from different domains and stacks, but you always find that the same strange loops are waiting underneath.
A slight change over the decades was that more and more of the systems programming aspects of building software ended up in libraries or components. While that means that you’ll have fewer opportunities to implement strange loops yourself for actual production systems, you really still do need to understand how they work inside the components to leverage them properly. Being able to pound out a parser and an AST does help you understand some of the weirdness of SQL, for example. You intuitively get what a query plan is, and with a bit of understanding of relational algebra and indexing, how it is applied to the tables to satisfy your request. You’ll probably never have enough time in your life to write your own full database, but at least now you can leverage existing ones really well.
I’m not sure it’s technically a strange loop, but there is a group of mathematical solutions that I’ve often encountered that I refer to as ‘basis swaps’ that are similar. Essentially, you have a problem with respect to one basis, but in order to solve it, you need to find an isomorphism to some other basis, swap it, partially solve it there, then swap all or part of it back to the original basis. This happens in linear programming, exponential curve fitting, and it seemed to be the basis of Andrew Wiles solution for Fermat’s last theorem. But I’ve also played around with this for purely technical formal systems, such as device-independent document rendering. I guess ASTs and cross-compilers are there too, as are language-specific VMs like the JVM.
What I’ve seen in practice is that some programmers, when confronted with strange-loop type problems, go looking for shortcuts instead of diving in and trying to understand the actual problem. I do understand this. There is so much to learn in basic software development that you really don’t want to have to keep going off and reading huge textbooks. But I also know that trying to cheat a solution to a strange loop is a massive time waste. You’re always just another bug away from making it work correctly, but it will never work correctly. The best choice if you don’t have the time to do it properly is always not to do it. Instead, find someone else who knows or use something else where it already exists and is of pretty decent quality.
Mostly, there are very few strange loops in most applications programming, although there are knowledge swamps like schema normalization that cause similar grief. Once you stumble into system programming, even if it is just a simple cache, a wee bit of locking, or the necessity of transactional integrity, you run into all sorts of sticky problems that really do require existing knowledge to resolve them permanently.
Strange loops are worth learning. They don’t change with the trends or stacks, and they’ll enable you to be able to write, use, or leverage any of the software components or tools floating around out there. Sure, they slow you down a bit when you first encounter them, but if you bother to jump those hurdles, you’re lightning fast when you get older.
The most natural way to write the code to simulate adding little bits of water at a time to the pool was with recursion. No doubt, you could unroll that into a loop and maybe a stack or two, but we were supposed to use recursion; it was the key to the assignment.
I had never encountered recursion before, and I found it mystifying. A function calls itself? That just seemed rather odd and crazy. Totally non-intuitive. I coded it up, but it never really worked properly. I got poor marks on that assignment.
Many months later, the light went on. It came out of nowhere, and suddenly, not only did recursion make sense, but it also seemed trivial. Over the decades, I’ve crafted some pretty intense recursive algorithms for all sorts of complex problems. It now comes intuitively, and I find it far easier to express the answer recursively than to unroll it.
Recursion is, I think, a simple version of what Douglas Hofstadter refers to in GEB as a ‘strange loop’; a paradox is a more advanced variety. For me, it is a conceptual understanding that sorts of acts like a cliff. When you are at the bottom, it makes no sense; it seems like a wall, but if you manage to climb up, it becomes obvious.
There are all sorts of strange loops in programming. Problems where the obvious first try is guaranteed to fail, yet there is another simpler way to encode the logic that always works.
A good example is parsing. The world is littered with what I call string-split parsers. They are the most intuitive way to decompose some complex data. You just start chopping the data into pieces, then you look at those pieces and react. For very simple data that works fine, but if you fall into programming languages, or pretty much anywhere where there are some inherent ambiguities, it will fail miserably.
But all you really need to do to climb this cliff is read the Green Dragon book. It gives you all of the practical knowledge you need to implement a parser, but also to understand any of the complicated parser generators, like Antlr or Yacc.
I guess the cool part is that once you have encountered and conquered a particular strange loop, that knowledge is so fundamental that it transcends tech stacks. If you can write a solid parser in C, you can write it in any language. If you understand how to write parsers, you can learn any programming language way faster. And nothing about that knowledge will radically change over the course of your career. You’ll jump around from different domains and stacks, but you always find that the same strange loops are waiting underneath.
A slight change over the decades was that more and more of the systems programming aspects of building software ended up in libraries or components. While that means that you’ll have fewer opportunities to implement strange loops yourself for actual production systems, you really still do need to understand how they work inside the components to leverage them properly. Being able to pound out a parser and an AST does help you understand some of the weirdness of SQL, for example. You intuitively get what a query plan is, and with a bit of understanding of relational algebra and indexing, how it is applied to the tables to satisfy your request. You’ll probably never have enough time in your life to write your own full database, but at least now you can leverage existing ones really well.
I’m not sure it’s technically a strange loop, but there is a group of mathematical solutions that I’ve often encountered that I refer to as ‘basis swaps’ that are similar. Essentially, you have a problem with respect to one basis, but in order to solve it, you need to find an isomorphism to some other basis, swap it, partially solve it there, then swap all or part of it back to the original basis. This happens in linear programming, exponential curve fitting, and it seemed to be the basis of Andrew Wiles solution for Fermat’s last theorem. But I’ve also played around with this for purely technical formal systems, such as device-independent document rendering. I guess ASTs and cross-compilers are there too, as are language-specific VMs like the JVM.
What I’ve seen in practice is that some programmers, when confronted with strange-loop type problems, go looking for shortcuts instead of diving in and trying to understand the actual problem. I do understand this. There is so much to learn in basic software development that you really don’t want to have to keep going off and reading huge textbooks. But I also know that trying to cheat a solution to a strange loop is a massive time waste. You’re always just another bug away from making it work correctly, but it will never work correctly. The best choice if you don’t have the time to do it properly is always not to do it. Instead, find someone else who knows or use something else where it already exists and is of pretty decent quality.
Mostly, there are very few strange loops in most applications programming, although there are knowledge swamps like schema normalization that cause similar grief. Once you stumble into system programming, even if it is just a simple cache, a wee bit of locking, or the necessity of transactional integrity, you run into all sorts of sticky problems that really do require existing knowledge to resolve them permanently.
Strange loops are worth learning. They don’t change with the trends or stacks, and they’ll enable you to be able to write, use, or leverage any of the software components or tools floating around out there. Sure, they slow you down a bit when you first encounter them, but if you bother to jump those hurdles, you’re lightning fast when you get older.
Subscribe to:
Posts (Atom)