Sunday, June 6, 2021

Data Flow

I like to write this post every few years. People generally skim over it, and then ignore what I am saying. So I’ll probably end up writing it a few dozen more times, then give up completely.

There is one major trick to keeping everything sane while building a big system:


Forget about the code. 


Just ignore it. 


The problem isn’t the code, it isn’t ever the code.


It’s the data. 


If the data flows around the organization in a clean organized matter, then the code is just the secondary issue about the small translations needed as it moves about.


That is, if the data is good, then any problems with the code are both obvious and easily fixable.


If the data is bad or incomplete, the entire stability and trustworthiness of the system is busted. Collecting megabytes of useless data is an epic waste of time. There is no foundation for the work.


Also, stepping back and viewing the entire charade as just data flowing around from place to place is extraordinarily simpler than trying to grok millions of lines of code. Mostly, we collect some data from the outside and combine it with data collected from people in the middle of their problems. That’s it. 


You should never be writing code to retroactively repair data. That is if the data doesn’t exist, or it is ambiguous, or it’s stored in a broken format, that is the real problem. Patching that with flaky code is not a real solution. Fixing the data is. 


If you understand how the data needs to be structured to store it properly, and you honor that knowledge in the code as it moves around, then everything else is a thousand times easier. It’s code you need to write to move it here, or there. It’s code you need to write to translate it into another format. It’s code you need to write to combine some of it together to craft derived data. That’s pretty much it.


Then you can spend your creative energies on building up enough sophistication to really help people solve their real problems.

Wednesday, June 2, 2021

What Should Have Happened ...

 The first thing to do is admit that it went wrong.

The next thing is to figure out what should have happened instead.


With the two outcomes, there will be a bunch of points in time where things deviated from the desired path. Find all of them, list all of them out.


In some cases, an earlier deviation invalidates some of the later ones. That is, if it had not initially deviated, then some later deviation would never have occurred and is rendered moot.


That leaves you with a list of ‘critical’ deviations. They are independent, or at least mostly independent enough that they all matter. 


Keep in mind that a bunch of concurrent small problems can interact together to create much larger ones. If one or more of the deviations was a ‘perfect storm’, then you need to list out all of the other contributing littler deviations.


With a final list of significant deviations, you can roughly assign a weight and fixing costs to them. 


An earlier deviation might actually have very little weight, it might just be more of an annoyance. But it still should be documented as ‘contributing’. 


There might be some deviations whose real cost is far too high to realistically fix them. The only solution for those is to put in place controls or mitigations to reduce their effect next time. But they should be noted as expected. 


So now you have:

  • What went wrong.

  • What should have happened instead.

  • A full list of everything significant, including breakdowns of lesser contributions.

  • A list of alternatives and a list of mitigations. Including the approx weights and costs of each.


The only thing left to do is march through the list and fix things.


Wednesday, May 19, 2021

Irrational Mappings

Right there, in the center of all software development projects, is a set of rather odd mappings from our irrational physical world onto a very strict formal computational one. 

These difficult mappings form the core of any system. 


Get them right, and it is easy to package up all of the other aspects of the system around them. Get them wrong and their faults permeate everything else, making it all rather hopeless.


The act of designing and building these mappings is not intrinsically difficult by itself, but there are enough places for variations and trade-offs that it is way too easy to get it completely wrong. 


It's not something that you learn overnight, nor is it easily explainable to others. There are no blog posts to read, nor textbooks to lay it out in any detail. 


The best way to learn it is to work with someone else who understands it, then gradually build up enough experience that you can eventually strike out on your own. Getting it wrong a dozen times doesn't help as there are millions of possible mistakes you can make. It’s only when you see it being done well, that you are able to get some type of sense of how it all comes together. 


You can tell when development projects are led by someone who gets it. The big and small parts all fit together nicely. It’s also pretty obvious when the project is adrift without a strong lead. 


Since this is the blurry intersection between technology and the problem domain, you really have to have spent time on both sides of the fence in order to see how everything lines up. A strong business background tells you about the problem. A strong technical background tells you about the possibilities of the solution. But neither side on their own helps with working through this mapping. You have to connect these two sides together, carefully, but since it's odd, crazy, and irrational, then throughout the process you have to try to balance out any hard edges in order to find some type of reasonable working compromise. 


The strength of any system is the quality of these mappings, which comes directly from the skill level of the people putting them together. It might just be one person, which tends to be more coherent, but if the timescales are too short, a group of people may have to come together to sort it out. 


We have no known defined way of ‘documenting’ this mapping. It underpins all of the other design work, peeking through in artifacts here and there. While you can’t describe it, it is clearly visible when it is missing. 


Since it binds all of the other work together, it is always a huge disruption when the original author leaves and is replaced. The next person might know how to follow suit, maybe slowly changing it, but this only happens if they can ‘see’ the original effort. 


Two or more competing mappings will always clash destructively. They will rub each other in the wrong direction, forming one larger broken mapping. Bad choices will gradually undo any good ones. 


A development project is over when the mapping is complete for the whole problem domain or when the mapping is so hopelessly broken that any effort to fix it exceeds the effort to start over again. In the first case, the system just needs continual ongoing work to move the code forward with the constant changes in dependencies. In the second case, any work going into the system is wasted. The skill level necessary to correct a bad mapping is off the charts.