Being able to predict what’s coming up in the future allows you to make better decisions right now. Better decisions will give you more chances to actually change the current direction rather than just getting caught up in the flow.
The first part of enhancing predictions is to accept that there is only one objective reality. Although it surrounds all of us, we only ever see a tiny sliver of it. Because of that, it is not uncommon for people to believe that their sliver is somehow personalized just for them. It is not. We might sometimes perceive the world as somewhat random, but that’s only because we are caught by surprise.
The next part is to understand that all things follow a trajectory. They will continue on their merry way unless they are interrupted by something unusual. From our perspective, reality is an informal system; all sorts of surprising things can occur. It is constrained, but these constraints are beyond our ability to formalize them. Although this allows for a lot of wiggle room on what might actually happen, it’s still constrained. Some things just can’t happen; they violate some very deep underlying rules about how the physics of the universe will play out.
In that sense, there are plenty of things that are possible, but also highly unlikely. And there are plenty of things that are absolutely impossible, although we might not have full certainty that they are not just unlikely.
So if we observe any phenomena long enough, while we can’t discount that surprising things may occur, if they don’t, then it will follow the trajectory that it has been on already. That is, things will mostly continue as expected based on their past behaviour.
That can be a bit tricky though, if the past behaviour has a rather complicated path through a very long period of time. That’s a lot of complexity to absorb.
Thus, to get a reasonable prediction, you need to observe the actual path through a context that is large enough to include all of the non-surprising but significant influences. Pull back far enough to include all of the dominant variations and influences, watch the movement, and then sketch out the default path forward. If nothing unusual throws it off course, you know where it is headed.
If you are surprised, it could just be an extremely rare event, or that your context was just not large enough to be accurate. Most often, it was too narrow a context.
In that sense, being able to predict lots of things around you is an indication that your knowledge extends far beyond your own sliver. You can learn to predict larger phenomena or farther out in the future as you acquire a deeper understanding of reality. They are correlated.
In the general sense, getting better at predictions will make your actions go smoother. You can leave a little room for the informality, but mostly get the results that you desire. It also helps with insurance. You might not be able to precisely predict infrequent events, but at least you can plan for them in order to mitigate their disruptions. This is a better strategy than just being purely reactive, in that it prepares you to better navigate any obstacles while keeping you aligned to a longer-term goal. When you can anticipate your bad days, you can prepare enough in advance to make them easier to deal with.
Thursday, July 23, 2026
Thursday, July 16, 2026
Hedging
A long time ago, I remember someone trying to convince me that sufficient testing would prevent ‘all’ bugs. All you need are tests.
That’s never been the case for me, even after all of these decades.
That’s never been the case for me, even after all of these decades.
At the very best, with a gifted team, we were super careful, had strict standards, and hyper-intense reviews, but we still had one bug escape. Worse, though, was that the effort took considerably longer than most people would even guess. It was the only time that I’ve ever not felt rushed during development. So it’s rare.
Ever since then, the default for me has been to anticipate that we won’t have enough time and that the code will be half-baked at best.
That’s okay if you expect it, but it changes your practices and habits. You learn to hedge any and all bets, always.
For example, you always assume there will be lots of bugs and make sure you’ve scheduled in enough time to find and fix them. If you track it, you can often get a sort of accurate guesstimate on how many to expect or at least an upper and lower limit.
You also want scaffolding. That is code that indirectly asserts, verifies, stops, counts, or in any other way validates the incoming data or calling sequences. You have it turned on during development, on for some testing, and off for production.
Full end-to-end system testing is always the best, but unit testing works for overly complex components. There is never a one-type-of-testing-that-finds-all bullet. Testing is expensive; you have to match the different types based on your time and quality requirements.
You want good logs and lots of auditing. From the logs, you should be able to see any and all internal state changes. While having one is best, they do get added all over the place. Logs are the main way to fully resolve real production issues.
For auditing, you need to know who did what. For some domains, a few weeks would suffice, but for some, you need years or decades worth of it.
You need a fast flag switcher. It lets you swap out functionality on the fly in production. It’s for extreme emergencies only, but time compression makes these far more likely.
You want to table-drive every piece of code that involves high uncertainties. That’s the most flexible arrangement possible, and it tends to be resilient against update mistakes.
You need two release processes. The right way and the fast way. You should never use the fast way unless you have no choice.
You want as much reuse as you have the time to get. It pays for itself a bit later, but by then you’ll really appreciate it. Particularly if you have to whip out some revised functionality, and you can do that nearly instantaneously with battle-hardened code. Way better than doing the bug dance again.
You should always be suspicious of any code that you depend on but don’t understand or didn’t write. It tends to bite you when you least expect it. The more you learn about how it works, the better things will go. If you can do it properly yourself, your life will be easier.
Ever since then, the default for me has been to anticipate that we won’t have enough time and that the code will be half-baked at best.
That’s okay if you expect it, but it changes your practices and habits. You learn to hedge any and all bets, always.
For example, you always assume there will be lots of bugs and make sure you’ve scheduled in enough time to find and fix them. If you track it, you can often get a sort of accurate guesstimate on how many to expect or at least an upper and lower limit.
You also want scaffolding. That is code that indirectly asserts, verifies, stops, counts, or in any other way validates the incoming data or calling sequences. You have it turned on during development, on for some testing, and off for production.
Full end-to-end system testing is always the best, but unit testing works for overly complex components. There is never a one-type-of-testing-that-finds-all bullet. Testing is expensive; you have to match the different types based on your time and quality requirements.
You want good logs and lots of auditing. From the logs, you should be able to see any and all internal state changes. While having one is best, they do get added all over the place. Logs are the main way to fully resolve real production issues.
For auditing, you need to know who did what. For some domains, a few weeks would suffice, but for some, you need years or decades worth of it.
You need a fast flag switcher. It lets you swap out functionality on the fly in production. It’s for extreme emergencies only, but time compression makes these far more likely.
You want to table-drive every piece of code that involves high uncertainties. That’s the most flexible arrangement possible, and it tends to be resilient against update mistakes.
You need two release processes. The right way and the fast way. You should never use the fast way unless you have no choice.
You want as much reuse as you have the time to get. It pays for itself a bit later, but by then you’ll really appreciate it. Particularly if you have to whip out some revised functionality, and you can do that nearly instantaneously with battle-hardened code. Way better than doing the bug dance again.
You should always be suspicious of any code that you depend on but don’t understand or didn’t write. It tends to bite you when you least expect it. The more you learn about how it works, the better things will go. If you can do it properly yourself, your life will be easier.
You want to assume at any time that you only have half of the picture, and that it is about to change. So, hedging some code with expected variability is more than reasonable.
If you really understand why the code solves a specific problem, you can vet it as you write it. If it's all Greek to you, it's more likely that it will need excessive changes in the future. It often gets lost in translation. You can factor it to make it easier to change.
Initially, you might get praise for grinding out tons of static brute force code, but that quickly changes as the problems and the high costs of fixing them emerge. It’s better if they accuse you of being slow, but later the drama just doesn’t happen. They’ll catch up eventually.
At all times, you just know that it's going to get difficult and somebody somewhere will always be upset. You shouldn’t develop a lack of empathy; instead, you should set up the game to quickly respond with increased precision. The second it's discovered to be wrong, you’re right on top of the fix. That gives you a much better reputation than if you shrug angrily and say, “I don’t know”.
If you really understand why the code solves a specific problem, you can vet it as you write it. If it's all Greek to you, it's more likely that it will need excessive changes in the future. It often gets lost in translation. You can factor it to make it easier to change.
Initially, you might get praise for grinding out tons of static brute force code, but that quickly changes as the problems and the high costs of fixing them emerge. It’s better if they accuse you of being slow, but later the drama just doesn’t happen. They’ll catch up eventually.
At all times, you just know that it's going to get difficult and somebody somewhere will always be upset. You shouldn’t develop a lack of empathy; instead, you should set up the game to quickly respond with increased precision. The second it's discovered to be wrong, you’re right on top of the fix. That gives you a much better reputation than if you shrug angrily and say, “I don’t know”.
Thursday, July 9, 2026
Data Stitching
If you collect some data with respect to a specific context, when there are significant changes in that context, the original data no longer fits seamlessly with any newly collected data.
One way to handle this is to change all of the old data, updating it to the new changes. This works, but it could take quite a while in production and can be extremely risky. A change in massive data can take days or even weeks to run. A small mistake could effectively destroy the old data.
The other way to handle this is at ‘presentation’. You split the computations, doing something slightly different for the older data.
All you need is a precise ‘line’ between them to test for. Then, to keep the code sane, you make both sides polymorphic and hide the test. When some code asks for the final data, if that request crosses the line, it gets handled correctly.
The way you don’t want to do this is by hardcoding the test somewhere high. Doing this once may not cause too much damage, but if the context has changed once already, it is highly likely to change again. A few times is really ugly; more than that is a classic bug sinkhole.
The overall idea is to always preserve what you collect and stitch it together with polymorphism. Then you can double-check it for correctness, so you have the best possible options for getting it right, and you don’t leave around landmines for later efforts.
For anyone worried about performance, presentations are almost always fixed-sized small data sets; the extra costs of the tests and the polymorphism are usually hardly noticeable. If there is some other type of dynamic large-scale bulk interface over the data, then it’s an architectural problem, not a data one. That is, if the performance is really noticeable, then you’re probably doing something else in the software that you shouldn’t be doing.
Data stitching is often avoided, but when done reasonably, it ensures that the data appears correctly and that it is straightforward to fix later if it doesn’t. Depending on the data, it often requires a domain expert to verify, which usually happens very late in the effort, so it's best to expect last-minute changes.
One way to handle this is to change all of the old data, updating it to the new changes. This works, but it could take quite a while in production and can be extremely risky. A change in massive data can take days or even weeks to run. A small mistake could effectively destroy the old data.
The other way to handle this is at ‘presentation’. You split the computations, doing something slightly different for the older data.
All you need is a precise ‘line’ between them to test for. Then, to keep the code sane, you make both sides polymorphic and hide the test. When some code asks for the final data, if that request crosses the line, it gets handled correctly.
The way you don’t want to do this is by hardcoding the test somewhere high. Doing this once may not cause too much damage, but if the context has changed once already, it is highly likely to change again. A few times is really ugly; more than that is a classic bug sinkhole.
The overall idea is to always preserve what you collect and stitch it together with polymorphism. Then you can double-check it for correctness, so you have the best possible options for getting it right, and you don’t leave around landmines for later efforts.
For anyone worried about performance, presentations are almost always fixed-sized small data sets; the extra costs of the tests and the polymorphism are usually hardly noticeable. If there is some other type of dynamic large-scale bulk interface over the data, then it’s an architectural problem, not a data one. That is, if the performance is really noticeable, then you’re probably doing something else in the software that you shouldn’t be doing.
Data stitching is often avoided, but when done reasonably, it ensures that the data appears correctly and that it is straightforward to fix later if it doesn’t. Depending on the data, it often requires a domain expert to verify, which usually happens very late in the effort, so it's best to expect last-minute changes.
Thursday, July 2, 2026
Language Ideas
Decades ago, our project had a convention where we never directly used ANSI C primitive types, unless it was trivial in a loop. But for any and all other variables, everything had to be typedef’d explicitly. Everything.
Some people would see that as excessive and way over the top, but it actually turned out to be a very good habit that helps with getting really high quality.
If you went to call something, you’d need an X, and that would force you to go into the code to find out how to create one. If the code is all ints, strings, doubles, etc., then people skip that effort and just find some hacky way to kludge the value they need. But now you can’t.
I’ve often thought that I’d like to see a language that has zero primitive types for variables. None. If you need a variable type, you have to declare it.
But to add to the fun, there aren’t even open types like ‘int’. If you needed an integer, you’d create the type, but as you did, you’d have to explicitly specify the range.
For example:
It would be more fun for strings, as they are not open either:
Each would be constrained by an RE state machine.
What I think would happen, maybe naturally, is that programmers would get bored with creating a million different types and a billion conversions between them. So, instead, they’d start to pack things together into larger structures all the time.
And for any discerning programmers out there who suspect that using just alphabetical characters is not wide enough to handle all of these types of names across the planet, staring at this declaration would trigger a need to further investigate and correct the model.
But the trick would be that in correcting the model, in that one place, it would also be corrected everywhere else. So the benefit would be that all of the validation code, both at the interface and at persistence, that would have needed to have also been updated when the model changes, would actually not need to be changed, since it is all implicit in the language. A recompile would do the trick.
I’d go even further, though. I’d not have any primitive data structures in the language’s library; they’d all be baked into the language too.
For example:
If it’s starting to smell like SQL, I apologize. The type semantics is essentially declarative, but the rest of the language syntax would be imperative, with some extended paradigm on top. Probably more like Golang than any of the other OO or FP variants.
The point, though, is that you would use the type mechanism to build up larger and larger data structures, and all of the base ones, like lists, trees, dags, graphs, and even hypergraphs, would already be there. For fun, you’d have all others like stacks, queues, and pagodas. All with various options, but to use more advanced ones, you would have to explicitly declare that they were not just trivial implementations.
That would ensure that anyone reading would not have to infer anything about the underlying implementations. The crudest thing would always be the default thing. Any special ability, property, enhancement, or optimization would have to be explicitly mentioned. It would minimize confusion.It would aid in avoiding bloat.
Over the decades, the trend in languages was to make them more dynamic and to minimize expressions. The result of adding those freedoms in the hands of a disciplined programmer was great, but the abuses heavily outweighed the elegant examples. Learning from that, I think we should find more ways to restrict the freedoms, but not cripple the expressiveness. That is, you’re not forced to type in reams of boring boilerplate, but you also can’t write code that is cryptic enough to win the obfuscated C code contest either. The code you have to write always makes its intent clear. Readability is part of the language.
Some people would see that as excessive and way over the top, but it actually turned out to be a very good habit that helps with getting really high quality.
If you went to call something, you’d need an X, and that would force you to go into the code to find out how to create one. If the code is all ints, strings, doubles, etc., then people skip that effort and just find some hacky way to kludge the value they need. But now you can’t.
I’ve often thought that I’d like to see a language that has zero primitive types for variables. None. If you need a variable type, you have to declare it.
But to add to the fun, there aren’t even open types like ‘int’. If you needed an integer, you’d create the type, but as you did, you’d have to explicitly specify the range.
For example:
type Counter: integer 0..INTMAX
It would be more fun for strings, as they are not open either:
type Token: string [a-zA-Z]*
Each would be constrained by an RE state machine.
What I think would happen, maybe naturally, is that programmers would get bored with creating a million different types and a billion conversions between them. So, instead, they’d start to pack things together into larger structures all the time.
type UserName:
First: string [a-zA-Z]*
Last: string [a-zA-Z]*
And for any discerning programmers out there who suspect that using just alphabetical characters is not wide enough to handle all of these types of names across the planet, staring at this declaration would trigger a need to further investigate and correct the model.
But the trick would be that in correcting the model, in that one place, it would also be corrected everywhere else. So the benefit would be that all of the validation code, both at the interface and at persistence, that would have needed to have also been updated when the model changes, would actually not need to be changed, since it is all implicit in the language. A recompile would do the trick.
I’d go even further, though. I’d not have any primitive data structures in the language’s library; they’d all be baked into the language too.
For example:
type UserList linkedlist:
Entries: User all
Sorted by User.UserName.Last
If it’s starting to smell like SQL, I apologize. The type semantics is essentially declarative, but the rest of the language syntax would be imperative, with some extended paradigm on top. Probably more like Golang than any of the other OO or FP variants.
The point, though, is that you would use the type mechanism to build up larger and larger data structures, and all of the base ones, like lists, trees, dags, graphs, and even hypergraphs, would already be there. For fun, you’d have all others like stacks, queues, and pagodas. All with various options, but to use more advanced ones, you would have to explicitly declare that they were not just trivial implementations.
That would ensure that anyone reading would not have to infer anything about the underlying implementations. The crudest thing would always be the default thing. Any special ability, property, enhancement, or optimization would have to be explicitly mentioned. It would minimize confusion.It would aid in avoiding bloat.
Over the decades, the trend in languages was to make them more dynamic and to minimize expressions. The result of adding those freedoms in the hands of a disciplined programmer was great, but the abuses heavily outweighed the elegant examples. Learning from that, I think we should find more ways to restrict the freedoms, but not cripple the expressiveness. That is, you’re not forced to type in reams of boring boilerplate, but you also can’t write code that is cryptic enough to win the obfuscated C code contest either. The code you have to write always makes its intent clear. Readability is part of the language.
Thursday, June 25, 2026
The Craft of Programming
Back in the eighties, when I was a student, there were only two choices: you used ‘vi’ or you used ‘emacs’.
I picked vi, my roommate picked emacs. The stuff he did with his editor was way cooler, but over the decades, vi has pretty much been everywhere I have stumbled.
I assumed it would fall out of fashion, but it persisted. Vim and all of those embeddings in the IDEs keep it alive.
I read something about how someone thought that using a complicated editor like vi was unnecessary. They implied that it was a waste of time to learn lots of its features. There were so many other things to learn, why not just use a lame mouse-based editor and move on?
For me, the answer is that it is all about craft.
When you build a large program, it’s a lot more than just throwing together the thousands of instructions necessary to tell the computer what to do. The code is just the output, but if you are hasty and careless on your way there, the whole thing ends up as a house of cards; any little breeze will knock it over. That is, it isn’t just coming up with some code; it is coming up with good, solid code that is readable and will survive all of the craziness that the world throws at it.
I’d rather have 150K lines of something that is super tight, abstract, solid, dependable than have 1M lines of stuff that could be served in an Italian restaurant. A whole lot of bad code is just a few steps closer to an apocalypse than a system.
So, it’s not about how much code you can create or how fast you can create it. Instead, it is about how carefully you embed precision into that code. Not for tomorrow, but for its entire lifespan. You want strong solutions to the real problems you need to solve, and you want them encoded in a way that you can keep them moving forward and leverage them for all sorts of related problems.
In the act of doing that sometimes tedious, thoughtful, and precise work, you want to develop habits that are more likely to produce high-quality output. Quality matters, not quantity.
Getting back to vi, if you are concerned about your code, you care very deeply about how it is ordered in any sort of file. It should not be random; you have come up with some consistent reason why one function is placed ahead of the others. In the daily grind, you have probably violated that order accidentally a few times. In most editors, you can use the mouse to highlight a chunk of code, hit ctrl-X to cut it, then ctrl-V to paste it back where it should go. It works, but it’s a touch haphazard. You could be interrupted, or get confused, or any other disruption that might interfere with the outcome you desire. In vi, you ‘mark’ the start, then ‘mark’ the end, then issue a few commands to move the code into place. It’s a small difference, but it’s transactional integrity. It either happens or it does not. If you get interrupted, it doesn’t matter; nothing is lost, nothing is messed up. If you're unsure about where to place the code, you can scroll around with the keyboard until you are sure. It isn’t any slower than using the mouse, but it is a whole lot more precise.
That’s a trivial example, for sure, but it’s just one of those things that you can do with a good tool when you’ve invested the time to understand it fully. Being a good programmer, a reliable one that is always there in a crisis, is the act of investing a whole lot of time in honing these acts of precision. It’s not that you can whack out a whole lot of code that you barely understand; it is that when you are given a piece of work, you can ensure that that work is done satisfactorily, even in environments that are difficult.
That is the essence of good programmers. They are the ones that everybody leans on, especially when it gets difficult, because they know that they’ll be there and make sure it is at least as good as it can get right now. They are reliable. Not always efficient, never fast, but always making sure that the work going out the door is solid.
We’ve always had the expression “garbage in, garbage out” in reference to the necessity to have good input data, but it also applies to code. If you get a group of people that churn millions of lines of questionable code, it’s just a field of landmines waiting to go off. If you get a group of people that solve complex problems with the least amount of code they can, carefully, then it doesn’t take that long before that quality of work starts to pay for itself. The craft of programming isn’t coding; the craft is in providing solutions, it just so happens that most of them involve code in some way.
I picked vi, my roommate picked emacs. The stuff he did with his editor was way cooler, but over the decades, vi has pretty much been everywhere I have stumbled.
I assumed it would fall out of fashion, but it persisted. Vim and all of those embeddings in the IDEs keep it alive.
I read something about how someone thought that using a complicated editor like vi was unnecessary. They implied that it was a waste of time to learn lots of its features. There were so many other things to learn, why not just use a lame mouse-based editor and move on?
For me, the answer is that it is all about craft.
When you build a large program, it’s a lot more than just throwing together the thousands of instructions necessary to tell the computer what to do. The code is just the output, but if you are hasty and careless on your way there, the whole thing ends up as a house of cards; any little breeze will knock it over. That is, it isn’t just coming up with some code; it is coming up with good, solid code that is readable and will survive all of the craziness that the world throws at it.
I’d rather have 150K lines of something that is super tight, abstract, solid, dependable than have 1M lines of stuff that could be served in an Italian restaurant. A whole lot of bad code is just a few steps closer to an apocalypse than a system.
So, it’s not about how much code you can create or how fast you can create it. Instead, it is about how carefully you embed precision into that code. Not for tomorrow, but for its entire lifespan. You want strong solutions to the real problems you need to solve, and you want them encoded in a way that you can keep them moving forward and leverage them for all sorts of related problems.
In the act of doing that sometimes tedious, thoughtful, and precise work, you want to develop habits that are more likely to produce high-quality output. Quality matters, not quantity.
Getting back to vi, if you are concerned about your code, you care very deeply about how it is ordered in any sort of file. It should not be random; you have come up with some consistent reason why one function is placed ahead of the others. In the daily grind, you have probably violated that order accidentally a few times. In most editors, you can use the mouse to highlight a chunk of code, hit ctrl-X to cut it, then ctrl-V to paste it back where it should go. It works, but it’s a touch haphazard. You could be interrupted, or get confused, or any other disruption that might interfere with the outcome you desire. In vi, you ‘mark’ the start, then ‘mark’ the end, then issue a few commands to move the code into place. It’s a small difference, but it’s transactional integrity. It either happens or it does not. If you get interrupted, it doesn’t matter; nothing is lost, nothing is messed up. If you're unsure about where to place the code, you can scroll around with the keyboard until you are sure. It isn’t any slower than using the mouse, but it is a whole lot more precise.
That’s a trivial example, for sure, but it’s just one of those things that you can do with a good tool when you’ve invested the time to understand it fully. Being a good programmer, a reliable one that is always there in a crisis, is the act of investing a whole lot of time in honing these acts of precision. It’s not that you can whack out a whole lot of code that you barely understand; it is that when you are given a piece of work, you can ensure that that work is done satisfactorily, even in environments that are difficult.
That is the essence of good programmers. They are the ones that everybody leans on, especially when it gets difficult, because they know that they’ll be there and make sure it is at least as good as it can get right now. They are reliable. Not always efficient, never fast, but always making sure that the work going out the door is solid.
We’ve always had the expression “garbage in, garbage out” in reference to the necessity to have good input data, but it also applies to code. If you get a group of people that churn millions of lines of questionable code, it’s just a field of landmines waiting to go off. If you get a group of people that solve complex problems with the least amount of code they can, carefully, then it doesn’t take that long before that quality of work starts to pay for itself. The craft of programming isn’t coding; the craft is in providing solutions, it just so happens that most of them involve code in some way.
Subscribe to:
Posts (Atom)