Thursday, November 19, 2026

Analysis

Analysis for software projects is the act of gathering, curating, and collating enough detailed information about a set of given problems that people would like to use software to solve. If it is complete enough, then it can be used as the sole reference for people creating designs for these solutions. It has all of the answers to all of the questions that a designer will have.

All software systems are anchored by the data that they collect.

The usage of these systems is confined within the boundaries of the data that they have access to. In that sense, this data sets the foundation on which any other features or functionality are built.

If there is a problem for which one wants to use software to solve it, then obviously the first and foremost thing that needs to be done is to figure out what data is necessary.

Data comes in two basic forms: a) you collect it directly from people, b) you cross-reference it from somewhere else.

So analysis goes through each and every datum and tries very hard to list it out with proper names, some understanding of expected frequency, location, properties, etc.

The more you know about the data, the smoother the development process will be.

For some data, it is not worth collecting literally; you can get it from relatively easy computations based on other raw data. So, the analysis needs to specify any ‘derived’ data necessary and how to compute it from the underlying raw data.

Since the software is a ‘solution’, it is important to understand how people will use it in order to solve their problems. Most of the time it is not ‘log in and click a button’. It is a considerably more complex ‘workflow’ where the users leverage specific functionality in a bunch of features, over a multi-step process in order to make decisions or create specific artifacts that they share.

The analysis needs to list out all of the known workflows, in as much detail as possible. When there are misfits with using the software, they begin and end in these workflows. That is, if it is only possible to do half the work in the system to accomplish something, then the system isn’t a particularly tight fit. If you can do it all and there is just one button to press, it is really tight, but if there is a lot of functionality in the system, the problem switches to being able to find that button, not pressing it.

Presenting data for decisions is straightforward, but then minimizing the navigation after that decision is critical too. If someone decides that the data indicates a specific action, it would be best if the button to trigger that action was presented at the same time, or at least very close by. So, presentations need to be linked back to their lists of expected actions. That is, you need to know why the data needed to be presented in order to wire up the navigation for the decisions.

Creating new artifacts is similar. They are essentially manually triggered derived data, but the tools to create and edit them can be very complex. The tools need to be arranged in a way that matches the expected usage for creating stuff. And more importantly, the tools should leverage as much of the existing data as they can; the user should not have to retype things, just rearrange them.

For cross-referenced data, there are usually performance vs. quality decisions that need to be made. If you grab the data early and keep it around, there is a chance it can get stale before you update it. That would be worse than if the process of using or presenting the data is really slow. So, there are accuracy expectations for all cross-reference data that need to be taken into account.

The analysis of a set of problems should give enough details about people dealing with those problems that a solid design can be produced. Analysis is nearly timeless. If the business does not change, the “business logic” within does not change either, even if the system gets reimplemented with many different technologies over the years. So, the descriptions, references, tables, etc. in the document should not reference, rely on, or be tied to any technology. In order to deal with a problem, you need to collect data and work with it in the following ways. It’s the ‘what’ portion of the issues, where the design document is the ‘how’.

For example, you might need to know a derived metric like the ‘yield’ of a given bond in the financial markets. There are many different ways to calculate that, all based on the bond’s prospectus. The yield is a real number, which for many technologies is represented using floating-point variables, although there are lots of other ways of representing it.

Analysis should really only be done once for a new line of business, then updated over the years and decades as the underlying domain changes. In this sense, it is independent of any sort of software implementation, and the work to keep it up to date should be disconnected from any IT initiatives. A business should be constantly working through its own analysis in order to keep its own line of business operating as well as possible. If that were done, then software projects would start at ‘design’ instead of ‘analysis’, which would likely take less time and improve their quality.

Thursday, November 12, 2026

Unbounded Loops

[ Editor's note: I wrote this because I forgot, again :-( ]

I’ve written this before, but it is always worth repeating.

Unbounded loops are bad. If there is a bounded loop that can be used instead, use that instead.

Code really has two major flow-of-control constructs. One is conditional execution. You execute the code “sometimes”. We see that as ‘if’ statements, but there are all sorts of variations on that theme.

The other is loops. Loops come in two flavours, bounded and unbounded. A bounded loop is one that has the number of iterations effectively set in advance. A good example is a ‘for’ statement, with a variable called ‘i’. It starts at 0 and finishes at some ‘size’. The 0..size is the bounds for that loop. It will “always” stay within them.

Sometimes we don’t have bounds. A REPL is a good example. As long as the user keeps typing in text, the loop will continue to run. At some point, the user types in something special that triggers the code to ‘break’ the loop. Could be ‘exit’ or ‘quit’ or ‘EOF’ or something. It could just be a dynamic terminal triggered by the end of work. There are plenty of other places, like communication protocols, where there are often unbounded loops. But all of them are special.

Sometimes, people encode their loops as unbounded, even though they are essentially bounded. If encoding it as a bounded loop is only slightly less convenient, then it should be encoded as a bounded loop. Being clever with loops is really bad.

The reason for this is that it is self-describing. If you hit an unbounded loop, you might have to search around through a whack load of spaghetti to find its bounds. That’s a huge waste of time.

Another good reason is computational complexity. If you are coding loops, you really, really, really need to know how long they are going to take. If it loops on 3 or 4 things, you can quickly ignore it, but if it might loop on a million things, you can’t just chuck code like that out into production and hope for the best. One day it will fail, and people will be angry about it. If it can loop ‘that hard,’ then at bare minimum you’ll need some sort of testing to be in place so that you have the certainty that it won’t be you causing a denial of service attack on the system.

So, if you quickly read the code, and you can instantaneously pick up all of the bounds, then often in your head you can have a reasonable degree of certainty that when this code gets out there in the world, its behaviour will not sufficiently degrade from the behaviour you’ve been seeing while you worked on it in development.

That certainty is everything.

You have become an expert in programming because the code you write does only and exactly what it should do relative to your expectations. Obviously, that doesn’t mean it is “right”; it just means that when it runs, you will not be surprised, and you really don’t want surprises, do you?

Thursday, November 5, 2026

Difficult Problems

In software, these days, if there is a fleet of existing interoperating systems, it is most likely that when they were developed, they focused on solving the majority of the easy problems.

Often, it is also the case that, in doing that specific work, they made any related difficult problems far more difficult.

What this usually means is that if things are not working smoothly, then there are probably no longer any easy solutions that would fix the mess. The development has painted itself into a corner.

Anything that would make significant progress in correcting some or all of these deficiencies is going to be extremely difficult now.

Why?

The obvious point is that the current solution is fragmented; e.g. there is more than one interconnected system. The reason they are fragmented is that people kept tearing off smaller parts and solving those. That behaviour is a side effect of people zeroing in on the easy stuff and then trying to solve only their bit in isolation. Thus you get a bunch of systems where interconnection is an afterthought.

If you have a space where there are a lot of these smaller independent fragments, not only do they, as separate pieces, cause artificial complexity, but also their focus, as tunnelled from the larger whole, forces them to skip over difficulties. That is, they are so focused on just getting their thing done that they do not care about the whole thing, so if they hit something difficult while doing their thing, they punt that out of their space. This leaves the space between all these systems filled with all of these punted difficulties. When this has been the approach for a while, most of the easy stuff is gone, and what remains is darn scary. The overall complexity of the Frankenstein mess is orders of magnitude higher than any sort of unified, difficult solution.

What to do?

Really, you have to go back over the mess and identify the difficulties. Pick a big one, start there, and grow it out to cover back over the easy stuff. That is, if hyper-focusing on solving simple things got you into this mess, then only hyper-focusing on difficult things will get you out of it.

Most difficult problems in software aren’t really that difficult; they are just difficult in a very restricted context. If you pull up and out a bit, there is probably a known way to get just that problem solved correctly. It definitely won’t be a cakewalk, but it is still likely solvable.

Given that, you take this difficult problem, a known thorn in everyone’s side, and you write up both a specification of the problem itself and a design for something that would fix it. With those, you see what other existing parts of other systems they now cover. Then, if you are sure that would really fix the difficult issues, you work out a slow, phased transition to get that difficult core into place, and gradually extend it out to everything else. If you keep at this, eventually everything else disappears, and what you have left is one single integrated, fully functional, “difficult” solution that works smoothly.

I’ll say it again. Simple solutions do not solve difficult problems; they only fragment them into bigger messes. If the problem is difficult, the solution needs to be at least as difficult. There is no getting around this. You cannot avoid difficult problems; you can only ignore them for a while. I’ll call this the first law of solutions.

Thursday, October 29, 2026

Little Things

There are a bunch of little things -- habits -- that I’ve learned over the years that tend to make coding smoother and the results much better.

The primary one is to do as much as possible in advance of the actual coding. Make sure the environment is set up and working, including disks, peripherals, and furniture. Lay a good, solid foundation for things like logging, errors, config, secrets, and security. Get all of that out of the way first, not later as it is needed. When you sit down to write the core code, the last thing you want is distractions, so preparing your working environment to be able to get into that state of higher focus and stay there for longer is a key to being able to do a lot more things a whole lot faster.

The second one is to leave no stone unturned. That is, if you have questions about what or why you are coding, go answer those questions now. The answers don’t get better if you wait, and the lack of them always threatens to turn your labours into pure make-work. If you don’t know, then find out. Don’t assume, ignore, or procrastinate. Find out now.

Another one is to use way too many functions. Way too many. I use functions as a way to name clumps of code. It helps a lot. Even though there are a lot of functions, they shouldn’t end up with crazy or obfuscated names. Generally, they are just some verb as it operates on some input nouns. One word is preferable; two is acceptable. More than that is suspicious. Anything cryptic is offensive.

Functions, instead of comments, as a way to document what you are doing makes it way easier to review the code later and convince yourself that it really is doing what you thought it was doing. Also, they act as the primitives of decomposition. You break down the problem as far as you can put it into a bunch of small functions, but no further. An added bonus is that they make it easier to refactor. They are a powerful construct; use them wisely.

On that note, splitting out the pre-conditions and the core logic is big too. The first part of any function is there just to make sure the second part cannot get messed up. So, get all the asserts, error handling, little calcs, etc. out of the way at the top. Put every piece of data you will need later into its own unique variable; do not be shy. Similarly, use the language to clump stuff that must stay together all together. Clumping stuff saves on a lot of stupid typos.

Don’t waste time on makework, but also don’t be afraid to radically refactor some large chunk of code or a bad feature if it turns out that you were mistaken. If the logic is upside down, for example, and you persist rather than byte the bullet and correct it right now, it is just a time vacuum.

Make everything readable; spend effort on those tiny details. Partly because it shows you care, but also because those pesky things will hide lurking problems. Expect to have to revisit your own code in a few months, when you are too busy with other things; try to make that moment a lot easier.

Be terrified of names. They shouldn’t be robotic; there is no mindless convention for creating them. You really have to go out and find out what the parts are actually called, both technically and in the domain. Misnaming something is a magnet for bugs. If the poor name propagates all over the place, it can cause grief for decades. Give it the right name; put a lot of thought and effort into searching out that right name. Don’t be creative; that is usually just a failed shortcut.

Labour over your loops. Conditional logic is messy, but looping can be bloat. Don’t loop if you don’t have to; don’t bury the loops even if you think it's pretty. Split them out; you should be able to justify each and every one of them. Get rid of them whenever you can. Looping is where every performance problem begins and ends. Loops are not your friends; they are just the cost of doing business. Respect their power to make your evening awful.

Pull back. If it feels like you shouldn’t be spending 50 hours grinding out boilerplate, it is because you don’t need to be spending 50 hours grinding out boilerplate. There is a better way; you just have to calm down and find it. It’s exactly these types of moments where you learn to properly generalize or abstract stuff. The code would have been so similar that it is painful to type it out, but there is a way to get the same results, just smarter. Some techniques are easy, some take years to master, but all of them will keep you out of trouble. So, if it feels like you shouldn’t continue down this ugly brute force path, then don’t. Let that instinct push you into doing a search for a better way. It is out there.

Don’t be clever, be smart. I know’s tempting to find some clever little bit of syntactic sugar that can save a few characters here or there, but instead of being smart, it’s just a landmine. You or someone else will accidentally stumble over it and blow off their leg, usually in the middle of a production disaster or when there are -200 hours left on the clock. It almost always pays to be explicit, just not when it is tedious boilerplate. But otherwise, type it out in full. Eat that little expense, because later you will appreciate that you did.

The most important thing to understand is that although the code is not the system, the code itself is super important because it is your strongest way of controlling exactly what the system does, and you always need to control that to match everyone else’s expectations. It’s universally true that as the size of the code grows, the style, formatting, organization, consistency, and details get more and more important. A tiny demo can be messy, but a massive mission-critical system cannot. To be effective, you must be certain, and to be certain, you have to learn to be precise. That way, when everyone’s back is against the wall and you have to dive in and save the ship, you can go in and do exactly that, with no negative side effects. Perfect code is the dream, but the ability to perfect code quickly and reliably is good too. Don’t set yourself up to fail.

Thursday, October 22, 2026

Tactics vs Strategy

I guess I didn’t understand it when I was younger. Back then, doing things was just doing things. You decide you were going to do something, then you go about doing it. Simple.

But over the years, I was told otherwise. For instance, if you decide you are going to do something big that is relatively far out into the future, that is quite different than making an impulsive decision right now. E.g., deciding to go to University versus deciding to go out on the town tonight.

Two decisions, but very, very different.

When you set a goalpost for yourself far out into the future, that is basically a strategy. Since it’s so far away and often quite vague, it is nothing more than a direction you want to head. It’s essentially proactive; you’re looking trying to get ahead of the game.

When you make a decision right now, based on the immediate circumstances, that is tactical. It is reactive; you were forced into making that decision, and even if you choose not to decide, it is still a choice.

We see that in software development all of the time. Someone decided to build a system; that is the direction and the strategy. Someone else insists that a quirky feature get shoved in at the last minute; doing that work to appease them is tactical.

A development project without a direction is essentially a random walk. As people make suggestions about what to add to the codebase, if there is no way to evaluate them, then everything gets added. Eventually, the different random features conflict with each other. It is a mess. For example, you’re building a code editor, but someone wants it to play audio files. Why?

In that sense, having a direction carefully bounds the work, making it far easier to distinguish between good suggestions and bad ones. Someone once said something about how having a long-term plan allows you to make better short-term choices; that seems to be universally true.

Software development, however, is a little more complex. It is often a multidimensional tug of war, with lots of ropes and all sorts of people trying to constantly pull the work in only their direction. Which means that in order to not upset this balance, you frequently make tactical compromises.

For instance, there is some really good persistence technology that you know will work as expected, but for the initial release you bend to a request to dump in some trendy, rather flaky, tech instead. You know it won’t work as expected, but you lack enough control over the situation in order to stave off the make-work of using an inappropriate component.

What that implies is that many, if not most, of the tactical choices you end up making in order to keep everyone happy and keep the funding coming in are headed straight into technical debt. They are things that you must do now, but will also have to fix later, if the codebase makes it that far. If they are not fixed later, then it will hamper the lifespan of the code, cutting its utility short.

And this is the game in steering large development projects. You need to carefully set that strategy while getting bounced around by tactical issues. For those immediate problems, you either make good decisions in line with your strategy, or you make bad ones that you will remember to revisit and correct later. If you step back and see software development in that light, it gets a little easier to make better choices.

For instance, you’re tasked with building something significant, so you know where you are headed now. But the environment has lots of constraints and politics, so you know you need to weave around all of this. You don’t lose sight of that goal while you are forced to zigzag all over the place.

As the work progresses, you learn how to batch up some of the tech debt cleanup directly with new feature development. So, there was some older part of the system that you were forced to do poorly in the early days, but now, if you extend the scope a little on some new functionality, it will end up covering that mistake. From the earlier example, you put in some less-than-ideal persistence technology, but now you have upcoming work that needs a much larger schema. So, you wrap that new work along with switching the database technology as well. Kill two birds with one stone.

I used to describe this as alternating between short-term and long-term choices, but it is a lot more nuanced. You’re really keeping track of the bad tactical decisions and attempting to fix them along with newer but related work.

That works well when you do a non-destructive refactoring first, for instance, on some code before you extend it. The first work is pure cleanup, correcting earlier mistakes, while the second is moving forward with enhancements. But the second part is now way easier, because the pain was removed with the first effort. What should you refactor? The painful parts. Putting that into perspective, you did a strategic refactoring to undo poor tactical choices in order to smooth out some more tactical work.

Note that this does not only apply to the code. Basically, it is everything else too.

To get going quickly, you choose not to dive too deeply into understanding the domain for the project. That was a tactical choice. But if you add lots of domain knowledge to your resume, a lot more jobs open up later in your career, which is a strategic consequence. Technical knowledge is good, but lots of it goes stale quickly, whereas domain knowledge is more valuable in the long run. You can keep returning to it over the decades.

To expedite a project, people did an ultra-light job of dealing with the analysis. Again, a tactical choice. Later, though, the missing and incorrect information drives a lot of ‘unexpected’ changes, which delay the release, which was a more strategic goal. Fighting or arguing over the mistakes, another tactical choice, further pushes out the dates, making the project look strategically weak and more likely to get cancelled.

Or maybe there is demand to get the code out on a specific date. Again, very tactical. To make that window, lots of work gets cut, like carefully checking and editing the code, and creating a lot of tests. So, both the code quality and the QA fall short of what was actually needed, strategically. But the tactical reaction to the tsunami of operational bugs is to rush through fixing them. That burns everyone out, and the bugs just cascade or move around, since the quality of the fixes is no difference than the original coding; rushed. The strategic choice is to revisit that poor work, but raise the bar this time. Enhance the quality; do much better testing. The tactical twist is to head towards the buggiest code first, but strategically everything is suspect.

If you frame everything this way, it really helps. Without some strategy, you won’t go anywhere, just around in circles. So the goal is always to get yourself closer to the end, but the path there is often quite twisty. If you have to do something, you can decide whether it helps strategically, is useless, or will need to be revisited later. If you just make everyone happy, but get zero movement toward the goal, you lose. If you do too few strategic things, you lose. If it becomes a mess because of too many poor tactical choices, you lose. If you go nowhere because of too much make-work, you lose. You need to do as much of the stuff as you can in order to keep up the momentum and not be a mess.

It’s also worth noting that the goal should always be, at least, solving the user’s problems. If it’s just adding some feature, or hitting a release date, or playing with some technology, those are far too short-term to help. The goal has to be far enough away that it drives most of the tactical choices in a direction. It’s also worth noting that you probably won't be there if and when the project crosses the goal line, and that is fine. Your contribution was to head it off in a good direction; if it gets derailed long after you have left, that is unfortunate, but not your fault. The corollary to that is that if you join an existing project, then the most important thing to learn is the current strategy, and the worst tactical choice you can make is to ignore it or bend it way off into another direction. Strategy changes do happen, but they are slow-moving beasts, since they effectively reweight all of the previous tactical choices. You can’t start writing an accounting system, then switch to it being a video game, or vice versa. The first part is turned into make-work by the shift to the second goal.

In most cases, picking a strategy isn’t hard. But picking it and communicating it to everyone is a key part of leadership, which can be tricky. On the technical side, it is the dev lead's responsibility. If the team is headed one way and a programmer is going off in a different direction, their work now effectively has no strategy; it’s just random. So, it’s critical that everyone adopts the same strategy in order to work together. This means that if you are young or new to a team, even if you have ideas or preferences, you have to set those aside in order to effectively contribute. So, mostly, except for very early greenfield work, we don’t have to come up with a strategy; rather, we have to sort through the noise and figure out what it is. In many cases, it is actually ancient, but that is a good thing rather than a bad one.

Thursday, October 15, 2026

Synchronization

If you are building a software system, then collecting data directly from the users isn’t too hard.

You need an interface, usually a GUI, that lets them add or edit stuff. 

If you modelled the data reasonably, the screen presentations are mappable back to the database format, so you just need a bidirectional way to go between them.

If you need to decorate that interface with some relatively static data, it doesn’t get much harder. You manually get the data from somewhere, keeping track of how often it gets updated. If changes are occasional or a little more frequent, you can put that data into a config file, then maybe try to add in some update automation or screens to make updating it easier.

The game gets a lot harder, though, when you split the program in two with a client/server type arrangement. If the communication is super reliable, it's just a little extra error handling to support rainy days. If the communication is not reliable, then there are a series of hard trade-offs you need to make when handling failures.

In some cases, if the front can’t reach the back, it can effectively freeze. But in some cases, it needs to continue on as normal, then carefully resync later when the backend is available again. What it shouldn’t do is show the user old, stale data; if changes in the backend went unnoticed, that just causes chaos. It also shouldn’t clobber new data that was changed in the backend without being updated.

This gets super complicated if there are lots of users and they are changing stuff all of the time. And bumps up to a huge problem if there are lots of users and lots of backends. Keeping that type of arrangement in sync is right at the edge of our knowledge.

Internally, you want to try and wrap it somehow. That is, you have user data, static data, and remote data. You put that in a layer underneath any of the computations and presentations.

The different types of data are polymorphic, but you wire in the worst-case error handling at the top for everything. You do that because data accessibility problems are essentially interface problems, not data ones. That is, the interface asks for some data, but it is told to wait. Or maybe it is told to drop what it asked for earlier and update it with something newer. The problem isn’t reading data; rather, it is keeping a small window of data in the client synchronized with some other source of data, which means that the wiring has to include any and all of the means to detect that it is out of sync and also to fix the circumstances, as well as support two-directional communication.

So we can play with the idea that the client sends an event to ‘ask’ for data; the server sends an event to ‘show’ a specific version of that data.

If you’re going to go right over the edge, then you might wire up a client as sets of widgets that send ‘ask’ events for specific data entries. Later they get ‘show’ events and respond. At some point, they can also send ‘update’ events for changes to that data. They do this on all ‘sets’ of interdependent data rather than individual fields.

Then, we get a scenario where two clients send a bunch of ‘ask’ events and get back ‘show’ events. The users change stuff; the first one to hit Save gets updated, but then the other client receives an ‘update’ event. For each widget on their screen with modified data, it can show both the new value and the changed value on the screen and ask the second user to pick, or back off.

We get another scenario where the user grabbed the data via a bunch of ‘asks’ followed by ‘shows’. Then the server went down. So on Save, they get some visible indication that the ‘update’ was sent, but the code never gets the ‘show’ back again. The user is notified that their changes are pending and not saved. Again, it can ask the user to address the situation. It can even provide a means to temporarily hold their changes until later.

In this way, we can reframe the problem not in terms of logic to write, but rather in terms of just keeping a small pool of data synchronized with a much larger pool. This type of abstraction is data-driven and is far easier to understand than lots of twisted nested error handling dumped all over the code. You can explore the essence of a problem and find simple machinery in order to mitigate it.

We can even extend this idea to really difficult scenarios. The program displays integrated data from a huge number of different sources. The widgets only display derived data; underneath this, the derivation does a series of ‘asks’, then does recomputes based on the incoming ‘shows’. It pumps back a ‘show’ to the above widget. For some sources, using the very last ‘show’ that is cached is allowable; for some, it is not. Daily calcs, for example, can’t use day-old stuff, but weekly calcs can use stuff up to six days old. We wire in that data retention config at the source, and then everybody on top can only do the right thing. It can’t be ignored; it can’t be overridden. If the data far below, buried in some deep derived data, won’t update, then nobody on top will either.

That type of strict handling is necessary because different people will each handle the same problem in a different way. That freedom begets chaos, which ultimately confuses the users, diminishing the value of the code, not enhancing it. It would be better to be consistently wrong than to have a random mix of correctness. At least then it could be fixed quickly in just one place.

The idea, though, is that we can step back to look at the way the data flows around, or doesn’t. If we change our perspective away from coding and towards just keeping different things in sync with each other, it becomes a lot easier to solve this as a general problem, rather than keep specifically solving it over and over again for each frustrating little instance. Good abstractions aren’t special; they are just stronger ways to twist your perspective a little when a bunch of related problems are causing similar trouble, in order to do a better job solving them all at once. We just need to dig down to the fundamental problems, not the symptoms on top.

Thursday, October 8, 2026

Preservation

Wealthy people have a very bad tendency to rewrite history for their own benefit.

That is a detriment to our species. History should not be written by the conquerors; it harms us all.

Since we’ve suffered from near-total media capture, all our channels have become untrustworthy. That needs to change.

First, is that we need to promote unaffiliated voices whose primary concern is the truth, rather than profits, influence, or power. There are plenty of people out there who say things that are very close to reality. They are uncorrupted by other desires.

Many are not professional writers; they are just doing their best to work on their own simple blogs. They are clear, with little intent, rational thinking, and the fragments of reality that they glimpse are closer to the truth.

There should be a place where ordinary, everyday people can see these works and know that they are genuine. They have been vetted and preserved.

But we need more than that. Far more.

These writings need commentary. They will not be perfect and are even subject to change over time, as more of reality presents itself.

So the original postings should be preserved, and the authors should be allowed to submit edits, which appear alongside them. History gets recorded in black text on a white background.

Any sort of statements that may be in doubt are rendered with yellow backgrounds. It’s not necessarily wrong, but it’s not necessarily right either. As well, any text that is now known to be wrong is rendered with a red background. It should serve as a warning that the author did accidentally drift away from reality, from the truth, but it does not cast judgment on why they did it.

As links, there should be all sorts of commentary about the text. You can pick a yellow sentence and explore why it is not white. That commentary too is colored based on how it is received. There will be commentary on the commentary, turtles all the way down. A nested world whose complexity reflects our own.

In that way, you can read the original works, but can follow through on any sort of issues related to how well that work actually matches reality.

Obviously, to get this requires a large network of reviewers that are willing to dig into what is said and question it objectively. Within this group, there will be a lot of discussions, which are subject to the same treatment.

As time progresses, our colouring of certain works may change. Those changes should be examinable, and the reader should be able to question them as well. Things that we used to believe were true can sometimes be put in a more evolved light as we learn more. Things that we assumed may be wrong. Reality is immutable, but our perception of it is dynamic.

In that sense, the posts are the tip of the iceberg for much larger discussions and arguments underneath. If we get a community of people dedicated to finding and preserving the truth, this repository will be a trustworthy place to not only get to know reality better, but also to monitor the changes in the way we perceive it as a species.

On top of the words from individuals, we can add significant references from other places, where restrictive laws like copyright allow. If media or institutions donate works, they would be added here as well, and as much as we can get from the public space.

There should be no limits as to the content. It is whatever people feel is important enough to talk about at the time of writing, as it will represent our focus as societies and as a species.

The contributors will be voted in to be included. If you know of people who routinely say well-crafted things that are trustworthy, you can nominate them. If they agree to contribute, the words and images from their blogs will be copied by others into this site. It will archive them for posterity.

Some people will have a small number of works included; some may have their lifetime works there.

It’s worth noting that by opening commenting and fact-checking, some people may find it to be rather brutal. But the goal is not ad hominem attacks; rather, it is to take the words themselves and assess them for being truthful. As such, the posts will not be mostly red. This site is not a place to shame people writing for ulterior motives, just one to preserve the works that we mostly agree are truthful.

The site and repository should be funded with zero strings attached. It can be individuals, governments, or even companies, but it cannot be subject to any sort of restrictions other than that the money is applied to the setup and running of the site. The community, hopefully large and objective, will decide what to pursue. Someone who funded this work can never be allowed to use that as a means of having some works altered or deleted. Once something is preserved, it must stay that way. Changing it would betray the whole purpose of the effort.

Independent government funding, at a higher level like the UN, would be best. Maybe an international institute to gather together the funding that is itself hands-off from the site and the content. As a source of truth, this knowledge is ripe for corruption, so the highest priority would be preservation of what already exists, then ensuring access to the people. Adding to it is important, but only as a third priority.

Getting back to the intent, we have a period in our history where ordinary people can satiate their curiosity about our reality and are able to put some of what they learned into the public domain. This was a rare moment when information was not controlled by wealthy or powerful people. We’ve seen this ability get created, and now we see endless attempts to stop this from continuing. But because it had value, and it led to at least some small enlightenments in our world, it is a very worthy exercise to continue with. Lies only benefit the elite; the truth benefits everyone else. Our world gets better when it is no longer hostage to the whims of madmen.

Thursday, October 1, 2026

Bloat

Bloat, the burning of unnecessary computer resources, is a symptom of a lack of knowledge.

Most people are not deliberately trying to make their code resource-inefficient. It’s just that they have a job to do, and they know just enough to get it barely done. Extras like bloat, security, and extendability are off the table. To handle them properly requires knowledge they do not have, nor the time to acquire it.

There are infinite ways to accomplish any task with a computer. But these days they are all wrapped in inconsistent primitives or calls which are often quite messy. If you understand what’s really below that level, you can figure out how to utilize it to do only exactly what you want. But if you don’t really understand how they work, you're left chaining them together and fiddling until the output approximates your goals.

This has been understood since the 60s, often called the software crisis. The higher we build the house of cards, the more disconnected each new generation gets from what's holding it all up.

The rise of don’t look before you leap philosophies of development fuelled the problem. Coders are in such a panic rush that they just have to grasp at what is easiest, but combined with tunnel vision, that is also usually what is pretty wasteful too.

If you’ve got some fiddling to do with a string, then most people pick the most popular primitives. When they realize that the combination does a bit of extra work, they discount it as a micro-optimization. It’s not; it's a lack of knowledge driving the deoptimization or bloat, as we like to call it.

If they knew more about how it worked underneath, they’d pick a different set and augment that with better code of their own. The output would be the same, but the wasted steps would be gone. They weren’t needed.

That’s a trivial example, but we see this with fiddling, calling libraries and APIs, allocations, polling, queuing and caching, pretty much all over the place. There is some reasonable way to encode the steps, but getting there requires understanding what the steps actually do. Not a guess or a vague understanding, but not enough knowledge to be able to implement them yourself.

Q&A sites fuel this, because they give people a magic sequence, thus offering them the option to just blindly use it.

If you knew better, most people would choose to write the code properly; it doesn’t take more time. If they write bloated stuff, it is because they didn’t know better.

When computers were really slow, bloat was obvious and often unworkable. As Moore’s law kicked in, bloat became just another consequence of the race to get more code out there. But as Moore’s law fades, we'll be forced to return to those early days when people coded with more precision. To do that, they’ll need depth. You can’t correctly pick the best underlying steps if you have no idea what’s actually underneath. Figuring that out will keep you ahead of the pack.

Thursday, September 24, 2026

Webapps

In the early days of the Web, people struggled to make websites more dynamic. The original web technologies were centred around static presentations; they were pretty good at this.

As the trends matured, more and more technologies became available to make the sites dynamic, but also to use them to essentially wrap other programs and systems. The web interface was born; people gave these the cool name of webapps.

In those days, most serious developers continued to produce native GUIs. The web technologies seemed hokey and crude. Many were just mindless fronts that called the real stuff in the back. Light clients calling APIs.

The problem with native apps was portability. There were more operating systems back then, and it was crazy expensive to rewrite the same program for each one. The technologies for writing stuff once and getting it to run everywhere were a great idea, but the implementations were generally too limited to be useful.

What the web promised, though, was a guaranteed way to avoid portability problems. These promises, however, were quickly disrupted by the browser wars and by the emergence of mobile devices.

That led to a huge wave of ‘frameworks’ that all promised portability across all of these different platforms and form factors. Initially, they helped; webapps grew a little more sophisticated. But somewhere along the way, they all got pretty convoluted. Suddenly crafting a reasonable webapp was a whole lot more effort than just crafting native apps. That bump in complexity was rewarded with a visible drop in quality. Webapps started to become extremely buggy.

The browsers themselves grew in sophistication, but their integration was compromised by huge security failures. The web became a free-for-all for scams, driving a lot of people into silos.

They are still just a limited window into some features, but most of the ways to extend them are too fiddly to be practical.

So we’re left with most interfaces starting on browsers, then getting mobile cousins, then maybe better native versions. Oddly ironic since, except for form factor, they're all a bunch of interactions on a whack load of widgets. The foundations for graphical user interfaces haven’t changed for decades, just after the client/server split. All of these modern technologies trace a close lineage to their earlier generations.

If we were going to rethink this, it would be to go way back to the portability days. We’d still like to write one set of code that covers all three locations, pushing each right to its limits. That is, you’d grab a webapp you like, run it natively, and it would save your files locally. If you run it on a phone, it would give you the option of local or hosted.

If you can wire in optional platform capabilities and some dynamic form factor support, then you really could return to a point where writing interfaces wasn’t the bulk of the development effort. If it was near trivial to dump out 80% of the boring screens in a few days, then you could spend more time deciding which widget arrangements were best suited for which tasks, rather than expensive widget/presentation wiring and refactoring.

Webapps suck. They almost didn’t, but then fate intervened. We don’t need more siloed clumps of monetizable half-baked features; we have enough already. We need better adaptive and integrated tools that allow us to spend less time on computers, not more. The answer to this is not probabilistic personalized interface generation; more opaque, crappy code will only make things worse. It is to rework our foundations and get back to some of the great ideas of the past that we skipped over too quickly.

Thursday, September 17, 2026

Oversimplication

For a long time, the software industry turned against sophisticated power techniques like abstraction.

Partly because getting a little abstract is difficult for some people, but also because it slows down the number of lines they can grind. As an added bonus, it is a bit harder to debug.

People felt that rapidly churning out super simple, near-trivial, brute force code was the best approach.

Strangely, though, a great deal of the actual foundation of software rests on some pretty complex abstractions. Little of what is now possible with computers would be there or work properly if it didn’t. A lot of that code came from an age when people still cared about getting it correct. It’s not ‘legacy’, it’s ‘classic’.

As I watched these oversimplification trends take hold and gain strength, I was getting increasingly worried about the frailty of all of the software that we depend on. Everything is so buggy these days that it’s a wonder that anything works. A few decades back, people would dispute that claim and say it was crazy, but now it’s way worse and so obvious the tables have turned.

There are plenty of known problems out there where any and all attempts to brute-force some crude logic will fail. The scary part is that it may not ‘completely fail’. It may just look like it’s working, work some of the time, but then fail right in the middle of consequential moments.

Software has many essentially ‘physical’ boundaries. You can’t do anything to get past them; they are hard and fast limits. It’s also not particularly malleable, and certainly gets rapidly less malleable as the codebase grows. The crap at the bottom gets frozen there as more stuff gets piled on top. A digital form of gravity, I guess.

Any sort of ‘just jump in and pound it out’ philosophy is a recipe for failure. By the time you’ve clued into your most obvious mistakes, and there will be obvious mistakes, it is too late to go back and fix them. Then building on top of that shaky mess is doomed.

If the problem you are trying to solve is complicated, then any viable real solution to that problem is at least equally complicated. We’ll call this the equality rule.

Techniques like abstraction and generalization are the only means to take that unmanageable complexity and bring it down to a reasonable level where you can implement it correctly. All your other attempts to ‘hack through’ the issues are just a waste of time.

But some people are often so focused on trying to create the simplest solution possible that they end up ignoring the problem they are trying to solve. So, it’s disconnected. The solution is wonderful, but useless.

Instead, it’s way better to go grab a sheet of actual paper and start writing out any proposed solution first. Sketch some mechanics. Think of it as modelling clay, like what they use for car design. Then take that proposed solution and go back to properly explore the problem. If they don’t match up, go back to the design board. A sketch in time saves buckets of wasted coding. It’s orders of magnitude faster than coding up a bad solution, throwing it away, coding up another, throwing that away, ad nauseam.

Grungy coding habits like that are oddly okay for demos and quick prototypes, but it only works because the code is tiny. Scale screws with everything, and any code that is good enough will always grow. Maybe we should call that the destiny rule. If the code kinda works, then it is fated for continued expansion until it doesn’t work anymore.

The oddest part, though, is that the motivation to keep it simple is great. The thing should be as simple as possible; it’s just that any distance way over or way under that bar is exponentially worse. You need to hit the bar, and that bar is positioned by the equality rule.

If there is any sort of method to the madness, it is to start simple. But know immediately that it is too oversimplified. Bad design. Keep going back to the problem to figure out why it won’t work in specific situations. Put those complexities carefully back into the design. Iterate, over and over again.

If you think that might take too long, and you really don’t have decades to catch up with the state of the art, the shortcut is not to ignore the problem, but rather to jump out and do some reading on the state of the art. Leverage what other people have already figured out. Read a bit, learn a lot. Revise what you actually think simple really means. Know that if a kink in the problem space stumbles too close to a boundary, then find the best fit possible within your timeframe to actually address it, or at least carve out some space for it to be corrected later.

What you don’t want to do is hobble something together out of components you barely understand. While that keeps your code simple, what rests below explodes with overcomplexity.

It’s worth noting that the crazy balancing act necessary for modern software design is not something you can learn instantaneously. Because of this, the fastest way to gain knowledge is always mentoring. Get on a good project, work with good teams, and get someone who knows a bit to explain their rationale. Don’t be skeptical, at least not at first. Everyone is wrong, but some people are less wrong, thus closer to the knowledge you are seeking.

While it's tempting to believe that there is always a simple software solution for everything, it’s been proven incorrect so many times that it has become boring and forgotten, again and again. If you want to build software that really solves people’s problems for them, then that software has to really solve people’s problems for them. If their problems have become crazy complicated, then it’s obvious that anything simple will not even come close to fitting properly. I really should call ‘the equality rule’ a law or something...

Thursday, September 10, 2026

Ordering

With complex work, the order in which you do it hugely affects the quality, speed, and accuracy.

If there are ten subtasks, you do not really have the option to do them in any order. You need to start with some tasks first, before you do the others. They are dependent.

That’s pretty obvious if you are building a house. You can’t just do the roof first if you don’t have the walls up. And you can’t get the walls up until you have completed the foundations.

The crazy part about software is that although it is exactly like that too, most people incorrectly believe that because it is “digital”, it is somehow different. Somehow it magically escapes any dependencies, so starting in any order is fine. It is not.

As a consequence, you get situations that are equivalent to the stakeholders getting obsessed over the paint colour for a penthouse, when the foundations for the apartment building haven’t even been finalized in design yet, totally out of order.

\While we would not expect non-technical people to understand or be concerned about ordering the work, it is critical for developers to do this correctly. Maybe the second most important thing after organization.

The stakeholders only see the outside of the top; the rest of the box is invisible to them. They always assume it is trivial.

But internally, software is closest to a house of cards. It is far from trivial, where everything rests on what is below it. That orientation sets its dependencies. An interface that calls a backend, for example, is way up there. The persistence that the backend uses is far below. With that in mind, you can assess the “height” of all components.

Some architectures may flip certain parts of the game, but there is still some sort of relative height there. Figure out those heights, and you get a quick rough sketch of the ordering.

The biggest notion is that everything is built on something else. Those times when a few independent components can be built in parallel, in isolation from each other, are rare anomalies. Take advantage when you find them, but don’t expect to find them and definitely don’t assume they exist. Go the other way: assume there are far more dependencies for anything than you are currently aware of, since that is most likely to be true.

Set a height, plan as best as you can, start at the bottom, then work upwards.

When you get it right, development will go smoothly along.

When you get it wrong, there will be lots of frustration and drama. Simple changes will become huge, oversized issues. The schedule will keep getting pushed back. Way too much stress, and the quality will bottom out, causing downstream operational problems which just keep it all boiling.

Not always, but frequently, if there is a lot of drama in a development project, a key ingredient is bad ordering. That is an issue that developers can and should control. It is one of the places where they are directly responsible for the outcomes.

It’s a grey zone, though, in that plenty of management tries to control it too. It is contested ground, mostly because of timelines.

What experience teaches is that the less you say about the tasks and their order, the more control you will have over them, which you need. It’s better to frame any project scheduling around outside events, not internal ones. That’s why along the way I learned to separate ‘features’ from ‘functionality’, where one is a set of things the user needs, while the other is the specific way each one is implemented. That disconnect is not there to over-complicate things, but instead to disconnect the requirements from the work in order to retain control over the ordering of the work.

When and where some non-technical management has stuck their fingers too deeply into the pie, because they don’t care about dependencies, their ordering choices have been disastrous. Erratic and prone to burning crazy amounts of time on wasted efforts that were started far too early. And although they are the problem, they keep complaining that it's developers that are too slow, which is often beyond frustrating.

Ordering matters. It matters a lot. Once you’ve worked on a well-ordered project, it's difficult to deal with a chaotic one again. The shortest path to getting the work done is oddly very strict. A few other paths are similar, but most others are far worse. Stay on a good path; keep people from distracting you.

Thursday, September 3, 2026

Yak Shaving

The loose definition of yak shaving is that, on your way to completing the main thing, you recurse down so far, removing other little blockers, that ultimately you end up shaving a yak.

We see that in programming all of the time. You start adding a button to a screen, but you have to rearrange the other widgets. The backend stuff isn’t quite right; you have to fix some bugs, then add in a new endpoint. But that causes configuration issues, which need more time. Then you have to alter the database schema, so it's worse. But getting to the database means reconfiguring and installing new software on your workstation. What you thought was a simple button turned out to be changing stuff all over the place.

Yak shaving is actually a symptom of disorganization and/or ordering problems.

If you have a well-set-up workspace, where all of your tools are cleaned, organized, and ready to go, then you can just proceed with the main work. If something else distracts you, it is only because you didn't take care of it earlier.

So it's only smooth because you spent the time to get your workspace clean and organized. If it is a mess, then you keep stumbling into other problems that are distracting you.

Instead of being frustrated by yak shaving, you need to learn from it to prevent future problems. Pay close attention to it.

First, get your workspace in order. Do you have all the tools you need? Are they configured properly?

Clean and organize your files. Your desktop and folders shouldn’t look like a bomb went off. Your repo should be just what is needed, and everything should be in the right place.

For the work itself, always start down at the database. Go up from the data to the interface, not the other way around. In a house of cards, making the lowest changes first is the safest and best way to proceed.

When you do end up yak shaving, add it to a personal to-do list. Keep a separate one for any organizational or cleanup tasks. On off days, or just after a release, do these tasks to pace yourself. Most are slow and mindless, just cleanup work that needs to be done.

The trick is to be hyper-aware of yak shaving, and to try to converge on not having to do too much of it in the future. If you always spend a little ‘proactive’ time tidying your environment, gradually it will pay huge dividends.

But even if you are crazy rushed, just building up a big list of improvements helps, since programming is always hurry-up-and-wait. There will be off days. Plus, there are days where you’re just exhausted anyway, good for getting little proactive tasks done.

It’s worth noting that excessive yak shaving is often why estimations are so hard to get right. You know how long it will take for the main task, but you cannot predict how many holes you will fall down before you can get it done. Learn to avoid yak shaving, and then estimates become a whole lot easier and more accurate.

Some people will point out that, for example, they might not know what tool they need until they are deep in the mud. Sometimes that is the case. You’re right in the middle of the effort, hit a really bad bug, and then have to add in some very specialized monitoring or performance tools in order to figure it out. But that is often a problem with you not understanding how an underlying dependency or technology really works. Since you are venturing into new territory, it’s always better to do a bit of learning first before you dive in. If you did that, you would have at least been aware of the low-level toolset and issues before you started.

There was a belief that every bit of code written is so new that it will always be unexpected. The logic was that if it wasn’t really new, it would already exist, so you could use that instead. That was fueled by a younger generation of programmers who essentially ignored the fact that most of the code written out there is routine. It’s just more of the same, very similar to the stuff that’s been around for decades. The trends and dependencies kept changing, but the essence of the code itself actually got easier. More of it has become glue, just picking up data from one site or dependency and chucking it into another. So you always have to keep learning new, eclectic and weird interfaces, but the glue code itself is just stock.

In that sense, as an applications programmer, most of the code you will write will be similar, even in different languages and frameworks. Just widgets to persistence and back again. Lots of little fiddles. Plus a few ETLs.

For systems programming, if you do some low-level abstractions, they are more likely variations on the existing state of the art. Some tweaks and optimizations may differ, but the ideas have been around for decades.

The trend for distributed programming did increase, but the knowledge to really build reliable stuff disappeared. Lots of remote calls these days, but no error handling or transactional integrity. So, more problems, but caused by frequent misunderstandings.

Yak shaving is trying to cook a meal in a kitchen hit by a tornado and is now on fire. If you put out the fire and clean the kitchen up, then cooking the meal will be a whole lot easier. Oddly, keeping things tidy is a lot faster and far more efficient than letting it become a mess just because you’re too rushed.

It will never be perfect; you still have to get the work done, but setting the goal to keep everything as clean and organized as possible is a powerful habit. You can get real sustainable speed when your own environment is not the main issue that is slowing you down.

Thursday, August 27, 2026

Setbacks

In the mid-eighties, I was very excited about the potential of computers. Little personal computers were disrupting those massive, expensive beasts lurking in basements. It seemed like computers could integrate into our daily lives, change the world, and make it better.

Computer Science and Informatics were having a heyday; innovation was overflowing. There seemed to be monthly progress.

But over the decades, that progress was plagued by some pretty bad setbacks.

The first one I really saw was the rise of Microsoft.

They had been the David that took down the IBM Goliath, but the moment that they got traction, they got nasty. Over and over again we’d watch as some small group of people would labour to prove some new innovative technology was viable. But just before they could stabilize, Microsoft would swoop in with an army of kids and eat their business.

I was horrified, often quite angry. Instead of software progressing, any new buds were quickly snuffed out by products of far lesser quality, but way more senseless features. Microsoft then applied the rest of its growing monopoly to crush these small competitors.

Just before this era, the heart of innovation had shifted from academics into a small group of energetic people trying to get established. They had good intentions. Mostly these were wiped out. Innovation quickly died as a result; it was too risky, and after all that pain you’d just lose anyway.

That only changed when the next setback took hold.

The Internet had been around for a while, but the Web on top exploded into a huge bubble.

On the plus side, it brought most consumers to software, but it also embraced chaos. If Microsoft killed innovation, the Web killed quality. Whacking out crude software at high speeds was favoured. The ideas shifted from helping people to just exploiting them. The product was no longer software, but a cut of the money, control over people and surveillance. Software was created as fast and crude.

In a quick tumble, we switched from empathy to pure snake oil. If you could trick a million or more people into your broken product, you were famous.

Almost as a counter to the corruption of the Web, mobile apps nearly reversed that trend. For a while, but then everyone else took a page from the Microsoft warbook and quickly bought or crushed anything even mildly innovative or decent quality.

As the mobile dust settled, another setback had been quietly brewing since the dotcom bomb took hold.

Forget quality and innovation, Agile preached mindless coding at high speed.

If you just grind out enough code this week for whatever reactive drama was dominating, the belief was that your chaos would somehow converge into something useful. That lack of direction and foresight was misguided. You can’t randomly build an apartment building; it would obviously collapse, so why would people think that might work for any large technology structure?

Instead of learning from the mistakes of the past, the software industry doubled down on its madness.

Innovative sites vanished, replaced by grungy interfaces that only captured sellable data. People seem to just adjust to a steady flow of disruptive bugs. Using computers got insanely complicated, buttressed by a flood of questionable how-to pages.

Instead of software becoming more intuitive, people just followed opaque, incoherent recipes as if they were magic spells. This produced a mountain of incomprehensible code piled high on mountains of incomprehensible code. It kinda worked, but only if you stayed very close to the magic recipes. Deviate even a little bit at your own risk. Underlying knowledge of how things worked got decimated.

For a while there, it seemed like we’d finally get enough breathing room to be able to fix the industry’s problems. Maybe this time we’d be able to figure out how to build stuff that isn’t crap, but then LLMs appeared, and it started all over again.

Now we live in an era where we are totally dependent on the messes we’ve created but no longer understand. No longer just a house of cards, we suffer from skyscrapers of cards.

And a new generation of people thinks, again, that doubling down on our lack of understanding and handling that over to random traversals of illegally compiled huge lumps of questionable data will fix our problems. But it’s really just another repeat of the same mistakes.

We keep getting distracted from trying to actually figure out how to engineer massive systems that are both reliable and trustworthy. So long as we keep avoiding that, we’ll keep tumbling over the cliffs.

Thursday, August 20, 2026

Flow Design

One of the pitfalls in software design is for people to only concentrate on the narrow case where everything works as desired, but ignore all of the other cases where it has gone partially or totally wrong.

Data flow analysis can help avoid this.

The idea is that given some complex scenario for software mechanics, we can fully iterate all of the issues in advance, so we know what we need to solve.

To do this, we can examine the flow of the data.

In a simple example, the data is something a user knows. They enter that into a screen. On completion, the data is sent to the back end, which then persists it.

So the flow is:

User -> screen -> request -> backend -> persistence -> response -> user

Two key problems:
  • Data is invalid
  • One of the two backend resources is down

If the data is invalid, it is a computational problem. But it shows up in three places:
  • screen
  • request
  • persistence

We’d like all three to be identical. If they do differ, then the messages need to be specific about at which level.

We’d like to give the user a chance to correct the problem without having to start all over again. They really appreciate not having their time wasted.

There are at least three computers involved in this flow. If the first one is down, the flow is blocked, so we don’t need to worry about it.

If either of the other two is down, most of the time we want to wait for a little bit, just in case they become available again, so we sleep for a bit and retry.

If it's a long disruption at either computer, we want to let the user know. Since they may communicate it to support personnel, we want to be clear about which of the two resources is down. So there are two similar error messages for the user, but distinguishable for diagnostic purposes.

If the attempt to persist is successful, we also want to let the user know. But the backend is an intermediary, so it could fail just after the persistence worked and accidentally hide the success message. It was persisted, but the user gets informed about a failure instead. We need to avoid that somehow.

So, there are six arrows, four of which are problems that should be solved in the software.

One way of getting around this is to collapse the arrows as much as possible. Distributed programming is fraught with complex error handling and transaction integrity problems. Avoiding it is the best way of solving it.

Keep in mind that this is a tiny, common, simple example. But it does include all of the spinning pieces and issues. If you code all of the issues to work correctly, then you can extend that approach to much more complex scenarios. If you get this working, then filling in the missing deterministic computations is easier.

Thursday, August 13, 2026

Uncertaincy

While each line of code is pretty simple, and a few of them are not much worse, assembling big collections of code that behaves as expected is crazy difficult.

I used to think it was just scale that was the problem. Remembering a list of five things isn’t too bad; remembering a list of fifty things is beyond most people’s ability.

That can be solved with organization. Write them down, make it easily accessible. But even when the work is organized, it is still courting disaster.

I realized that any means of organizing stuff falls apart with growth. You neatly organize fifty things; it doesn’t work anymore long before you get up to five hundred. Constantly reorganizing stuff is always necessary, but people try to avoid it.

Still, even in the best huge codebase that’s been meticulously cleaned, it is still difficult. I think now that all of these smaller problems are tied together by one larger one: uncertainty. No matter how experienced you are as a programmer, lots of aspects of the work are uncertain.

The foundations and environment are too complex to fully understand. The technologies all have issues. They are wrapped up in histories and complex dynamics. The domain is messy, and understanding is constrained by the slow reveal. Each little step only shows the next step down.

Together, the norm is that you don’t know way more than you actually know. It’s worse when you are a junior, but it never, ever goes away. The explosive growth of the software industry guarantees it.

The difference between success and failure often seems to be how the developers deal with uncertainty. Do they accept it, or just pretend like it doesn’t exist?

For me, I know that I don’t know. But there are a bunch of things that I currently don’t know, but I really, really need to know them. These are the focal points. I can not complete and release the code until I know these things. Every other problem is immaterial. Resolve the key uncertainties and the rest will fall into place.

These uncertainties are often the properties implied by the way the solution is specified.

A trivial example is any sort of feature that lets a user save something and get it back later.

That obviously implies persistence, since there is no way to guarantee that an instance of the program, a thread, or a session or whatever is guaranteed to be live throughout that arbitrary time period. You have to take the data ‘now’, then return it ‘later’.

But it can get worse. If the data spans different resources during persistence, it requires transaction integrity. If it is dependent on external data, it involves synchronization. If it's a lot of data, it eats through storage and requires optimizations. To implement one simple feature may require resolving dozens of uncertainties or worse.

So it comes down to knowing that there are a massive number of things you don’t know or don’t really understand, then deciding which of those you must understand in order to get out the next release. Once you fill in those details, the others either reveal themselves or end up as some risk that you’ve decided is acceptable.

We’re forced to gamble sometimes; the alternative is a potentially bottomless pit of personalities and politics. Some details you can’t pull from the environment; some decisions get hobbled by personalities. For these uncertainties, you randomly pick an answer, shove it out into the world to see what happens, then pay attention to the feedback in order to really resolve it. That is the shortest path.

Dealing with the tornado of uncertainties whirling around you is what makes software development difficult. If you try to pretend that you are not caught up in this, the denial will only make it worse. If you accept that, unfortunately, this is the way it is right now, then at least you can adopt strategies that are more likely to produce better outcomes. You might not know, but that is fine; you still have to get the work done as well as possible.

Thursday, August 6, 2026

Properties

Code is a set of instructions for a computer to follow.

But code also has additional properties, beyond its runtime behaviour. There are lots of them.

A simple one is readability. Some code is cryptic; it takes a lot of investigation to figure out what it will do. Some code is self-explanatory. It’s written in a clear style so that most programmers can quickly figure out what it will do. It has extra comments added to fill in any missing explanations.

Another one is idempotent. The first time you call that code, something changes. All subsequent calls with the same inputs won't change it again. It’s a useful property to protect against unintentional repeated calls.

Another one is stateless. There are no internal variables that persist between calls. All the outputs are the results of pure computations on the inputs.

Optimization is also a property. Given the same inputs, some variations on the code will produce the expected outputs with less resources. Common resources include memory, CPU, GPU, disk, and network. This code usually exploits deeper dependencies within the data or the environment to get to the results faster.

Secure is a property too. Some code can be triggered by anyone; some code only by a strictly controlled small set of fully identified people.

Generalization is another interesting property. Within the code is considerable flexibility to accept a full range of different inputs and to produce a larger range of outputs. Instead of having dozens or hundreds of versions of nearly identical code, there is one version with a more complex flow that works its way through the different sets of computations.

Its cousin is abstraction. The code isn’t specific in its instructions. It pulls back from the actual logic to solve the problem at a higher conceptual level. Then there is often a way to bind that output back to the specifics, so the results end up the same. Its strength is that it is far less code than just grinding it out by brute force.

Safety is a property too. The author expected problems like bugs or bad data and added additional logic to detect, prevent, or correct the issues. The code has been explicitly constructed to prevent these bad outcomes. This is often called self-defensive as well.

Extensible is another less common one. The code is structured in a way to help ensure that future changes are less cognitively demanding. This often employs organizational techniques like symmetry or table driving. Adding new logic is usually just duplicating a few lines and making fairly safe little changes. Future changes take a few minutes.

While these properties don’t change the outputs of the code when it runs, they do strengthen its usability. They make deploying and using the code far less risky. They head off common problems, and make the development and operations of the code less stressful.

Most of them don’t require a lot of extra work, and all of them save a lot of time later in diagnosing and dealing with the usual problems. The fewer of these properties the code has, the more fragile it is when used. It’s important to add these in where they will enhance the development and usage of the code. Just calculating outputs with overly fragile code is not fully solving the problems.

Thursday, July 30, 2026

Methodology

My very first experiences with different methodologies for software development date back to the early nineties. I’ve lived through some heavyweight ones over the decades, including those in healthcare. 

Realistically, we need methodologies, even if we don't like to be confined by them. Too many loose strings in a project will eventually tangle.

Which is why I liked the emerging lightweight ideas at the turn of the century. But I found that over time their intent turned away from what I thought was reasonable.

So, I like to propose my own methodology. I guess I’ll call it “Adaptive”.

I realize that the word ‘adaptive’ is already taken for a specific type of ‘project management’, but this methodology is somewhat aligned to that, so they are complementary, while still very different. This is about the act of building complex things, not about the show that surrounds it.

The methodology starts as lightweight, but as quality or funding constraints are needed, it gets heavier. So, it’s almost weightless if you need to whip out a demo, but it’s pretty darn heavy if the code will end up in a medical device.

It values quality over quantity. That is, the intent of a software development project is not to create a mass of code; rather, it is to get the best code you can to solve the problem as precisely as you can. If the solution needs 150K to work properly, then the closer you get to that size, with enough quality to keep it useful, the better it is going.

It values organization over speed. No matter what you create, if it ends up in a giant pile of disorganized stuff, it is useless. Organization is, at minimum, having a place for everything and then putting everything in its place. The development and operations of the solution both need to be kept organized. A big part of organization is knowing where to put things, so there needs to be enough effort devoted to that to allow lots of people to know where things go.

It values engineering over evolution. That is, if there is a state of the art for implementing some specific part of the solution, then the people involved will go out and learn more about those prior efforts. They won’t reinvent the wheel just to save a little bit of time. Crude is not good. They know that the knowledge to do the work is out there, and they’ll attempt to find it first. Once they know, they are free to bend it to their will, or extend it, or even go off in a different direction, but first they have to know.

It values fit over ease of effort. If there are two ways to implement something, a simple one and a complex one, the best implementation isn’t driven by the technology or by the developers' preferences; rather, it is driven by which choice makes the solution the best fit for the users. It’s from their perspective, not the people building the solution. The point of software is that it gets used, so degrading its usefulness to save a little time or concentration doesn’t make sense. What is done is always done to make the users' lives better, even if it is hard and painful for the developers.

As for the specifics, since there are two main axes to the effort, there are two main roles for any project.

On the solution side, there is a lead developer. An expert in software development. They are fully responsible for the development of the software. They have a long, extensive background in building stuff that was complicated enough to act as a reasonable solution for similar problems. It’s not about energy or intent, but rather it is about having been through work in the past that is close enough to what needs to be done now, so that they will make the best choices for the project to keep it from derailing.

On the problem side, there is a domain champion. An expert in the domain problems. They are the person who is managing all aspects of the development from the outside and has considerable expertise in the problem itself. In an enterprise, they are at least a PM / BA combination with actual real-world experience in the business. They have spent time in the domain trenches. They have to have fairly deep knowledge in the domain and its practices, since this is where the details get difficult. They handle the external issues.

The domain expert is responsible for and the expert on all aspects of the problem. They know what the related problems are; they understand them. They have some idea of how to correctly deal with aspects of the problem, but they are not experts in solving it. So they know what the problem is, and have some ideas about how they think a computer might help deal with those issues. They control or have raised money to get the problem solved.

The developer is responsible for and in control of all aspects of the solution. The internals of development. If the solution does not correctly solve the problems, it is their fault. They know how to build it, how to deploy it, and how to keep it running over time. They understand how well it actually fits into the problem space. That is, they can’t just build a generic app and claim it ‘could’ solve the problems; they have to build something that ‘does’ solve the problems, so they can’t ignore understanding the problems themselves.

In that sense, the domain expert sets out a series of features they would like to see in the solution, while the software expert implements these as functionality. It’s a partnership. The software expert understands a lot of the nature of the problems that drive the solution, while the domain expert understands the limits and costs of technologies. But the domain expert has the final word on all domain issues, and the software expert has the final word on all technical ones.

The domain expert can say “your proposed solution is not actually solving our problems”, and the software expert can say “at best we can only approximate a solution to that given problem”. The path forward lies between these two points.

As for specific development processes, it is incumbent on the software expert to keep in constant communication with their own development team. How they choose to do that is up to them. Maybe it’s frequent meetings, or one-on-ones, or some combination. These are internal development product conversations; they should be private away from the domain conversations. What’s important is that they are constantly and actively communicating. The development team cannot be left in the dark, and if they are stuck on something, they need to raise it to the lead immediately.

If the internal project communication breaks, the project will derail.

The software expert and the domain expert are always working together. Neither the problem aspects nor the solution aspects have priority over each other. The idea is that they both need to be balanced. Part of the problem is often ‘time’. If a solution arrives too late, that is a big deal. But, unfortunately, since most development work is iterative over years of effort, prioritizing the correct order of the development work is on the software side. This is a contradiction that both sides have to resolve. It is necessary to avoid wasting time doing the wrong work at the wrong time, while still being timely. So, the two experts are in constant negotiation.

Small ambiguities or misunderstood domain and technical issues pop up on a regular basis. It’s normal. The sooner they are pushed over the fence, the better. While this methodology is not reactive, it does prioritize timeliness; but both sides should try to get ahead of the ball wherever possible. That is, the work may suddenly need to change in response to the ground shifting. It is not appropriate to sit and wait on a showstopper, for example, as the effort going on in the meantime may just become makework.

In very small projects, the lead developer might also be the domain expert, but only if they actually have the background to know enough about the domain. That is, they can’t be a faux domain expert; they have to be a real one. They don’t have to know the total full width of the domain, but they do have to know the width of the domain problems they are intending to solve.

The converse is not true. A domain expert with no significant technical background, given the scale of the project, cannot act as the lead developer. Whacking out some Excel spreadsheets or demos does not qualify someone to run a medium-sized development; the consequences will be disastrous.

A project gets started with the intent to solve a problem. Thus, the domain expert is there from day one. Once the project is in motion, however, the technical issues tend to slowly dominate. Most often, it is difficult to prioritize which features get done first, given whatever the limited time constraints.

Once there is an existing codebase, it has a number of deficiencies, and there is also some desire to grow certain parts of the solution. It’s a bit easier. But the effort slows down, as the work itself should get better; more precise.

There is always a long-term plan, as there is a defined problem, and an expectation that a specific collection of features will act as a reasonable solution to solve them. The initial round of development may only be scheduled for six months, but the long-term plan may extend out five or more years. Both should be discussed; both should be on the table. The value in any software solution comes not from its existence, but from its ability to be there, over and over again, as you need it. That is, even if the initial version is crude, if it does have value, it will gradually evolve into something significantly more complex. If it doesn’t have that value, it was probably a waste of time.

That there is both a short-term and a long-term for all projects is important to keeping the effort organized and to ensuring that the work overall is as effective as it can be. Which is why any sort of work compression like ‘reuse’ or ‘abstraction’ is critical. Once some work is completed, you’ll need to leverage it later, not redo it.

Any and all codebases will eventually reach a maturity point where the effort to extend them is so slow and painful that the life of the codebase is effectively frozen. As well, all codebases need constant maintenance, since the underlying technology is always changing at a crazy rate. If the work gets into that ugly ball of mud stage too early, then it will suffer a premature death. It will waste a lot of money; cost way more. Obviously, for many little demos this is not important, but for any medium-sized system or larger that lots of people rely on, it is highly damaging. If a line of business is reliant on the software, it is extraordinarily destructive.

At the core, though, writing software for the sake of writing software may be an interesting performance art, but it is not the intent of any of these larger development efforts. For these, the software is written explicitly as a solution to some real-world problems. So the code that goes into that solution must fully solve the problems and match their complexity. You shouldn’t over-engineer, but you also shouldn’t oversimplify.

Development projects are inherently iterative. That is, it is a never-ending discrete set of development efforts that can go on for years, or even decades. The frequency of demos and releases is driven by an external timetable and is often influenced by issues such as confidence and trust, particularly in the early stages of development. This means that the length of time for each round of development is variable and shifts. On the one hand, if the release periods are too short, there is more work involved because work is done out of sequence and more likely to be disorganized, but if they are too long, the funding or interest may fade. So, there is no perfect immutable length of time for each iteration. For big work, longer is better; for maintaining trust, shorter is better. On top of this, most funding comes with significant strings, so there is some minimal amount of tracking that is also necessary to maintain confidence. This is not extra work; both the technical and domain sides need to be confident that the big expense of development is worth the risk.

But it’s worth noting that excessive tracking will gum up the effort and the morale. Nobody likes being micromanaged, and any attempt to track the time for each and every little task is a classically bad form. The technology lead needs to keep presenting evidence that the work is progressing as expected, and they need to be diligent about raising any potential show stoppers early and often. The domain expert needs significant patience and to understand that even with the best of specifications, development teams, and technologies, there are still a great deal of uncertainties that may cause significant disruptions, particularly if the pace of development has been rushed.

Overly frequent tiny releases are both expensive and disruptive to the users, so they are not a good thing. People don’t need a new version of an app with just 200 lines of new code to fix minor, mostly unencountered problems. Unless it is an emergency, new releases should at minimum provide revised functionality or infrastructure shifts. That is, most of the time, a user should see some obvious changes in each release. If it’s not visible to them, and they have not been affected, it is just a disruption, not a positive.

Some iterations should always be pure cleanup. The bane of software projects is disorganization. As the project grows, its size invalidates earlier organization efforts. So, even regular consistent cleanup attempts will not be enough to keep the code from reaching the end of its life. Thus, organization itself is both the primary feature and main functionality of every software project. If it’s not there, then the value of the effort is extraordinarily low.

Even for the best lead developers and domain experts, there will be plenty of unknown unknowns. Which means that throughout the entire life of the project, there will be new and better information coming in which needs to be acted on. That is, while we know the technology itself is always changing, the understanding of the domain problems is often in flux as well. Managing change is essentially part of managing the development complexity of any software. Expect change, embrace change, and try to spend as little time as possible fighting about when those changes get reflected properly in the software.

As well, everyone involved in any software development project will grow. They will learn more, they will understand more, and many of the details will emerge. Even for late-stage development where the domain is nearly fully covered, the depth of what is understood is bottomless. Working on software is an exercise in constant learning; it never ends.

In some development efforts, there are significant trust issues. When this is the case, then all major decisions need to be papered, and they need sign-offs. If there is trust, particularly between the lead developer and the domain expert, then it can be bypassed. But if neither side insists on sign-offs, then both are equally guilty later when there is a disagreement. If the sign-off isn’t papered, it can be verbal. If it didn’t happen, then one side moved forward without informing the other. So, you can work on a verbal set of features, functionality, specifications, design, etc. only if both sides are in full trust mode. But if it results in a disruptive disagreement, the weight has to change going forward, and the decisions have to be written out in an email, a document, or any other medium that allows for one party to sign off on the terms of the effort.

It is trust that allows a methodology to be lightweight. If that is broken, it can no longer be lightweight and needs to be as heavy as the conflicts that stand in between the communications. That is obviously costly in time, so trying to stay as lightweight as possible is more effective.

It should also be noted that quality comes from not rushing through the work. So, if enhanced quality is a part of the requirements for solving the problem, then it necessarily slows down the effort. Basic quality for a rushed software project is a pretty low bar. Somewhere around expecting monthly big failures, if not weekly. If that is too much, the development effort needs to be slower; more tracking or oversight will only make the issues worse.

Quality control is an essential part of all software development. Since coding is highly mistake-prone, it is critical to ensure that all code is always properly tested before getting into releases. No untested code should ever end up in the operations environment; that is an avoidable and a seriously bad mistake.

The degree and type of testing depend heavily on the required quality of the effort and the time available for building it. Some code needs incredibly intense testing; some just needs a light amount of smoke testing. That is driven more by the problem domain, not the technical one. That is, writing an app for entertainment may not need much testing, but writing one to be included in a medical device, airplane, or any other strict domain needs a lot and many different types of testing.

So, as well as tracking and sign-offs, heavyweight processes often involve adding strong one-to-one relationships between the specified functionality and its testing in order to verify that it was implemented as described. That is, it ensures that the development process did not accidentally or intentionally deviate from what was specified and is known to be needed for the problem. It also ensures that there is no poorly tested functionality stuffed into the codebase.

In many cases, if the software can glitch with low impact, and the timeline is super tight, the testing can be minimal. But if that results in significant runtime problems, then the testing requirements need to increase. As long as there is any sort of upset from bugs escaping, the testing needs to be enhanced until finally the right level is found. That obviously means that putting effort into automating the testing itself is important, since it is more reliable. But it also means that the type of testing chosen is critical too. If partially testing a few of the components isn’t sufficient, and too many problems are occurring, then it needs more system, regression, or even reference testing.

Testing is not an afterthought; it is tied directly to the problem. That is, some problems need better quality, some are more lenient. It is influenced by the development effort as well. Some teams are better at double-checking their work; some are more reckless. The lead developer should know what the development needs and what the team is capable of. The domain expert should set a minimal level of quality that doesn’t contradict their time expectations.

As a big part of this methodology is in providing solutions that satisfy people’s ability to deal with their problems, part of the trust issues between developers and them is the need to act professionally. That is not just in writing code of reasonable quality or putting in the time to understand the domain perspectives, but also in the words and terminology used to describe the process as well. Because of that, for the parts of the process that developers may or may not need, it is important not to use gamified terminology for them. Some of the earlier lightweight methodologies tried to make the names sound fun or even cute, but that only degrades the way others see the profession. The names and terminology used need to look professional from the outside perspective because that is how developers get people to trust them, which allows them to keep the processes between them and stakeholders lightweight, which we know is at least a little less stressful for developers.

In summary, if a development project is adaptive, you should be able to easily identify the domain expert and the lead developer. Both of them should know what they are responsible for, and be good at avoiding stepping over those lines. As the issues have arisen, more heavyweight processes have been added, but the processes are as lightweight as possible. Both the problem descriptions and the code are organized; you can look through the details and see that they have been cataloged and handled appropriately. The code is always tested enough, and there are few operational issues. Whatever higher-weighted processes like some tracking are there, they are well organized and are not choking out the momentum. The train is on the tracks and moving forward at the best pace it can given the life phase of the project.

So for developers, the idea is that you have to know what the long-term solution is that you are building, while you have to keep adapting to the short-term disruptions that you encounter along the way. It’s trying to be proactive, while not letting the reactive issues knock everything too far off course. It's knowing that there is at least one person who really understands the problem you are trying to solve and another who really understands utilizing technology to really solve those problems, and that they rarely are the same person. It’s not over-engineered solutions that veered way off course, nor overly reactive little partial solutions that are piled together into a mess. That the long-term anchor keeps the short-term responses in line to extend out the lifespan of the effort, and that that is the most cost-effective way to proceed.