Irrational Mappings

Right there, in the center of all software development projects, is a set of rather odd mappings from our irrational physical world onto a very strict formal computational one. 

These difficult mappings form the core of any system. 

Get them right, and it is easy to package up all of the other aspects of the system around them. Get them wrong and their faults permeate everything else, making it all rather hopeless.

The act of designing and building these mappings is not intrinsically difficult by itself, but there are enough places for variations and trade-offs that it is way too easy to get it completely wrong. 

It's not something that you learn overnight, nor is it easily explainable to others. There are no blog posts to read, nor textbooks to lay it out in any detail. 

The best way to learn it is to work with someone else who understands it, then gradually build up enough experience that you can eventually strike out on your own. Getting it wrong a dozen times doesn't help as there are millions of possible mistakes you can make. It’s only when you see it being done well, that you are able to get some type of sense of how it all comes together. 

You can tell when development projects are led by someone who gets it. The big and small parts all fit together nicely. It’s also pretty obvious when the project is adrift without a strong lead. 

Since this is the blurry intersection between technology and the problem domain, you really have to have spent time on both sides of the fence in order to see how everything lines up. A strong business background tells you about the problem. A strong technical background tells you about the possibilities of the solution. But neither side on their own helps with working through this mapping. You have to connect these two sides together, carefully, but since it's odd, crazy, and irrational, then throughout the process you have to try to balance out any hard edges in order to find some type of reasonable working compromise. 

The strength of any system is the quality of these mappings, which comes directly from the skill level of the people putting them together. It might just be one person, which tends to be more coherent, but if the timescales are too short, a group of people may have to come together to sort it out. 

We have no known defined way of ‘documenting’ this mapping. It underpins all of the other design work, peeking through in artifacts here and there. While you can’t describe it, it is clearly visible when it is missing. 

Since it binds all of the other work together, it is always a huge disruption when the original author leaves and is replaced. The next person might know how to follow suit, maybe slowly changing it, but this only happens if they can ‘see’ the original effort. 

Two or more competing mappings will always clash destructively. They will rub each other in the wrong direction, forming one larger broken mapping. Bad choices will gradually undo any good ones. 

A development project is over when the mapping is complete for the whole problem domain or when the mapping is so hopelessly broken that any effort to fix it exceeds the effort to start over again. In the first case, the system just needs continual ongoing work to move the code forward with the constant changes in dependencies. In the second case, any work going into the system is wasted. The skill level necessary to correct a bad mapping is off the charts. 

System Inventory

 A list of things that need to exist for any large software development project:

1. A general write-up of the industry, domain, etc. including terminology and conventions.

2. A roadmap for the growth and direction of the system.

3. A set of requirements (by feature) from the user’s perspective.

4. The user’s workflows as they relate to the domain problems they are encountering.

5. A model for each of the different data entities that are encountered during the workflows.

6. A list of real examples and all special cases.

7. A set of requirements from the operational perspective.

8. A design grid for all screens.

9. A color scheme for the GUI.

10. A high-level navigation structure for the GUI.

11. An arrangement of data/functionality/layout for any ‘difficult’ screens.

12. A high-level design/architecture for the system, broken down by major components including frameworks, libraries, and persistence.

13. A high-level runtime arrangement for processes, machines, etc.

14. A lot of mid-level designs for the organization of the code.

15. A code repository with full history, including highlighting the code that was released to the different environments. Also contains all dependencies necessary for building.

16. A development setup guide, including any tools, configurations, etc. 

17. A style and conventions guide for making changes or new additions to the code base.

18. Technical notes on solving complex problems, preferred idioms, references, and other issues.

19. A searchable inventory of the reusable parts of the code.

20. Low-level specifications and/or the code.

21. A set of test suites that match back to the workflows and examples.

22. A full list of all known bugs (testing, user, or operational), solved or currently pending.

23. A set of scripts or instructions on how to build and package the code.

24. A list of expected configuration parameters and flags for testing and production setups.

25. A set of scripts to safely upgrade the system, including persistent schema changes and rollbacks.

26. An operational manual on how to handle recurring problems.

27. A tutorial on how to use the GUI for common tasks.

28. A reference document for any APIs, plus any technical notes for calling it correctly.

Any missing items will cause grief.

Redundancies

 If a computer program is useful, at some point it will get complicated.

That complication comes from its intrinsic functionality, but more often it is amplified by redundancies. 

If you want to delay getting lost in complexities for as long as possible, then you want to learn how to eliminate as many redundancies as you can. They occur in the code, but they can also occur in data as well.

The big trick in reducing redundancies is to accept that ‘abstractions' are a good thing. That’s a hard pill for most programmers to swallow, which is ironic given that all code, and the tech stacks built upon code, are rife with abstractions. They are everywhere. 

The first thing to do is to understand what is actually redundant. 

Rather obviously if there are two functions in the code, with different names, but the code is identical they are redundant. You can drop one and point all of its usages to the other one. 

If there are two functions whose inputs are “similar” but the outputs are always the same, that is redundant as well. If most of the inputs are the same, and the outputs are always the same, then at least one of the inputs is not used by the code, which can be removed. If one of the functions is a merge of two other functions, you can separate it, then drop the redundant part.

The more common problem is that the sum of the information contained in the inputs is the same, yet the inputs themselves are different. So, there are at least 2 different ways in the code to decompose some information into a set of variables. It’s that decomposition that is redundant and needs to be corrected first. The base rule here is to always decompose any information to the smallest possible variables, as early as possible. That is offset by utilizing whatever underlying language capabilities exist to bind the variables together as they move throughout the code, for example wrapping a bunch of related variables in an object or a structure. If you fix the redundant variable decompositions, then it’s obvious that the affected functions are redundant and some can be removed.

As you are looking at the control flow in the code you often see the same patterns repeating, like the code goes through a couple of ‘for’ loops, hits an ‘if’, then another ‘for’ loop. If all of the variables used by this control flow pattern are the exact same data-type (with the same constraints) then the control flow itself is redundant. Even if the different flows pass through very different combinations of functions. 

In a lot of programs we often see heavy use of this type of flow redundancies in both the persistent and GUI code. Lots of screens look different to the users, but really just display the same type or structure of data. Lots of synchronization between the running code and the database is nearly identical. There are always ‘some’ differences, but if these can be captured and moved up and out of the mechanics, then lots of code disappears and lots of future development time is saved.

We can see this in the data that is moving around as well. We can shift the name of a column in the database from being a compile-time variable over to being a runtime string. We can shift the type to being generic, then do late binding on the conversion if and only if it is needed somewhere else. With this, all of the tables in the database are just lists of maps in the running code. Yes, it is more expensive in terms of resources, but often that difference is negligible. This shift away from statically coding each and every domain-based variable to handling them all dynamically usually drops a massive amount of code. If it’s done consistently, then it also makes the remaining code super-flexible to changes, again paying huge dividends.

But it’s not just data as it flows through the code. It’s also the representation of the ‘information’ collected as it is modeled and saved persistently. When this is normalized, the redundancies are removed, so there is a lot less data stored on disk. Less data is also moved through the network and code as well. Any derived values can be calculated, later, as needed. Minimizing the footprint of the persisted data is a huge time saver. It also prevents a lot of redundant decomposition bugs as well. In some cases, it is also a significant performance improvement.

This also applies to configuration data. It’s really just the same as any domain data, but sometimes it’s stored in a file instead of in a database. It needs to be normalized too, and it should be decomposed as early and deeply as possible. 

Redundancies also show up in the weirdest of places. 

The code might use two different libraries or frameworks that are similar but not actually the same. That’s code duplication, even if the authors are not part of the project. Getting rid of one is a good choice. Any code or data ‘dependencies’ save time, but are also always just problems waiting to happen. They only make sense if they save enough effort to justify their own expense, throughout the entire life of the project.

Redundancies can occur in documentation as well. You might have the same similar stuff all over the place, in a lot of similar documents, which generally ages really badly, setting the stage for hardships and misunderstandings. 

Processes can be highly redundant as well. You might have to do 12 steps to set up and execute a release. But you do this redundantly, over and over again. That could be scripted into 1 step, thus ensuring that the same process happens reliably, every time. With one script it’s hard to get it wrong, and it’s somewhat documenting the steps that need to be taken.

The only redundancies that are actually helpful are when they are applied to time. So, for example, you might have saved all of the different binaries, for all of the different releases. It’s a type of insurance, just in case something bad happens. You have a ‘record’ of what’s been released with the dates. That can be pruned to save space, but a longer history is always better. This applies to the history in the repo and the documentation history as well. When something bad happens, you can figure out where it started to go wrong, and then do some work to prevent that from reoccurring next time, which usually saves a lot of support effort and helps convince the users that the code really is trustworthy.

In programming, there is never enough time. It’s the one consistent scarce resource that hasn’t changed over the decades. You can take a lot of foolish short-cuts to compensate, but it’s a whole lot better if you just stop doing redundant work all of the time. Once you’ve figured out how to do something, you should also figure out how to leverage that work so you never have to figure it out again.