Synchronization

If you are building a software system, then collecting data directly from the users isn’t too hard.

You need an interface, usually a GUI, that lets them add or edit stuff. 

If you modelled the data reasonably, the screen presentations are mappable back to the database format, so you just need a bidirectional way to go between them.

If you need to decorate that interface with some relatively static data, it doesn’t get much harder. You manually get the data from somewhere, keeping track of how often it gets updated. If changes are occasional or a little more frequent, you can put that data into a config file, then maybe try to add in some update automation or screens to make updating it easier.

The game gets a lot harder, though, when you split the program in two with a client/server type arrangement. If the communication is super reliable, it's just a little extra error handling to support rainy days. If the communication is not reliable, then there are a series of hard trade-offs you need to make when handling failures.

In some cases, if the front can’t reach the back, it can effectively freeze. But in some cases, it needs to continue on as normal, then carefully resync later when the backend is available again. What it shouldn’t do is show the user old, stale data; if changes in the backend went unnoticed, that just causes chaos. It also shouldn’t clobber new data that was changed in the backend without being updated.

This gets super complicated if there are lots of users and they are changing stuff all of the time. And bumps up to a huge problem if there are lots of users and lots of backends. Keeping that type of arrangement in sync is right at the edge of our knowledge.

Internally, you want to try and wrap it somehow. That is, you have user data, static data, and remote data. You put that in a layer underneath any of the computations and presentations.

The different types of data are polymorphic, but you wire in the worst-case error handling at the top for everything. You do that because data accessibility problems are essentially interface problems, not data ones. That is, the interface asks for some data, but it is told to wait. Or maybe it is told to drop what it asked for earlier and update it with something newer. The problem isn’t reading data; rather, it is keeping a small window of data in the client synchronized with some other source of data, which means that the wiring has to include any and all of the means to detect that it is out of sync and also to fix the circumstances, as well as support two-directional communication.

So we can play with the idea that the client sends an event to ‘ask’ for data; the server sends an event to ‘show’ a specific version of that data.

If you’re going to go right over the edge, then you might wire up a client as sets of widgets that send ‘ask’ events for specific data entries. Later they get ‘show’ events and respond. At some point, they can also send ‘update’ events for changes to that data. They do this on all ‘sets’ of interdependent data rather than individual fields.

Then, we get a scenario where two clients send a bunch of ‘ask’ events and get back ‘show’ events. The users change stuff; the first one to hit Save gets updated, but then the other client receives an ‘update’ event. For each widget on their screen with modified data, it can show both the new value and the changed value on the screen and ask the second user to pick, or back off.

We get another scenario where the user grabbed the data via a bunch of ‘asks’ followed by ‘shows’. Then the server went down. So on Save, they get some visible indication that the ‘update’ was sent, but the code never gets the ‘show’ back again. The user is notified that their changes are pending and not saved. Again, it can ask the user to address the situation. It can even provide a means to temporarily hold their changes until later.

In this way, we can reframe the problem not in terms of logic to write, but rather in terms of just keeping a small pool of data synchronized with a much larger pool. This type of abstraction is data-driven and is far easier to understand than lots of twisted nested error handling dumped all over the code. You can explore the essence of a problem and find simple machinery in order to mitigate it.

We can even extend this idea to really difficult scenarios. The program displays integrated data from a huge number of different sources. The widgets only display derived data; underneath this, the derivation does a series of ‘asks’, then does recomputes based on the incoming ‘shows’. It pumps back a ‘show’ to the above widget. For some sources, using the very last ‘show’ that is cached is allowable; for some, it is not. Daily calcs, for example, can’t use day-old stuff, but weekly calcs can use stuff up to six days old. We wire in that data retention config at the source, and then everybody on top can only do the right thing. It can’t be ignored; it can’t be overridden. If the data far below, buried in some deep derived data, won’t update, then nobody on top will either.

That type of strict handling is necessary because different people will each handle the same problem in a different way. That freedom begets chaos, which ultimately confuses the users, diminishing the value of the code, not enhancing it. It would be better to be consistently wrong than to have a random mix of correctness. At least then it could be fixed quickly in just one place.

The idea, though, is that we can step back to look at the way the data flows around, or doesn’t. If we change our perspective away from coding and towards just keeping different things in sync with each other, it becomes a lot easier to solve this as a general problem, rather than keep specifically solving it over and over again for each frustrating little instance. Good abstractions aren’t special; they are just stronger ways to twist your perspective a little when a bunch of related problems are causing similar trouble, in order to do a better job solving them all at once. We just need to dig down to the fundamental problems, not the symptoms on top.

Preservation

Wealthy people have a very bad tendency to rewrite history for their own benefit.

That is a detriment to our species. History should not be written by the conquerors; it harms us all.

Since we’ve suffered from near-total media capture, all our channels have become untrustworthy. That needs to change.

First, is that we need to promote unaffiliated voices whose primary concern is the truth, rather than profits, influence, or power. There are plenty of people out there who say things that are very close to reality. They are uncorrupted by other desires.

Many are not professional writers; they are just doing their best to work on their own simple blogs. They are clear, with little intent, rational thinking, and the fragments of reality that they glimpse are closer to the truth.

There should be a place where ordinary, everyday people can see these works and know that they are genuine. They have been vetted and preserved.

But we need more than that. Far more.

These writings need commentary. They will not be perfect and are even subject to change over time, as more of reality presents itself.

So the original postings should be preserved, and the authors should be allowed to submit edits, which appear alongside them. History gets recorded in black text on a white background.

Any sort of statements that may be in doubt are rendered with yellow backgrounds. It’s not necessarily wrong, but it’s not necessarily right either. As well, any text that is now known to be wrong is rendered with a red background. It should serve as a warning that the author did accidentally drift away from reality, from the truth, but it does not cast judgment on why they did it.

As links, there should be all sorts of commentary about the text. You can pick a yellow sentence and explore why it is not white. That commentary too is colored based on how it is received. There will be commentary on the commentary, turtles all the way down. A nested world whose complexity reflects our own.

In that way, you can read the original works, but can follow through on any sort of issues related to how well that work actually matches reality.

Obviously, to get this requires a large network of reviewers that are willing to dig into what is said and question it objectively. Within this group, there will be a lot of discussions, which are subject to the same treatment.

As time progresses, our colouring of certain works may change. Those changes should be examinable, and the reader should be able to question them as well. Things that we used to believe were true can sometimes be put in a more evolved light as we learn more. Things that we assumed may be wrong. Reality is immutable, but our perception of it is dynamic.

In that sense, the posts are the tip of the iceberg for much larger discussions and arguments underneath. If we get a community of people dedicated to finding and preserving the truth, this repository will be a trustworthy place to not only get to know reality better, but also to monitor the changes in the way we perceive it as a species.

On top of the words from individuals, we can add significant references from other places, where restrictive laws like copyright allow. If media or institutions donate works, they would be added here as well, and as much as we can get from the public space.

There should be no limits as to the content. It is whatever people feel is important enough to talk about at the time of writing, as it will represent our focus as societies and as a species.

The contributors will be voted in to be included. If you know of people who routinely say well-crafted things that are trustworthy, you can nominate them. If they agree to contribute, the words and images from their blogs will be copied by others into this site. It will archive them for posterity.

Some people will have a small number of works included; some may have their lifetime works there.

It’s worth noting that by opening commenting and fact-checking, some people may find it to be rather brutal. But the goal is not ad hominem attacks; rather, it is to take the words themselves and assess them for being truthful. As such, the posts will not be mostly red. This site is not a place to shame people writing for ulterior motives, just one to preserve the works that we mostly agree are truthful.

The site and repository should be funded with zero strings attached. It can be individuals, governments, or even companies, but it cannot be subject to any sort of restrictions other than that the money is applied to the setup and running of the site. The community, hopefully large and objective, will decide what to pursue. Someone who funded this work can never be allowed to use that as a means of having some works altered or deleted. Once something is preserved, it must stay that way. Changing it would betray the whole purpose of the effort.

Independent government funding, at a higher level like the UN, would be best. Maybe an international institute to gather together the funding that is itself hands-off from the site and the content. As a source of truth, this knowledge is ripe for corruption, so the highest priority would be preservation of what already exists, then ensuring access to the people. Adding to it is important, but only as a third priority.

Getting back to the intent, we have a period in our history where ordinary people can satiate their curiosity about our reality and are able to put some of what they learned into the public domain. This was a rare moment when information was not controlled by wealthy or powerful people. We’ve seen this ability get created, and now we see endless attempts to stop this from continuing. But because it had value, and it led to at least some small enlightenments in our world, it is a very worthy exercise to continue with. Lies only benefit the elite; the truth benefits everyone else. Our world gets better when it is no longer hostage to the whims of madmen.

Bloat

Bloat, the burning of unnecessary computer resources, is a symptom of a lack of knowledge.

Most people are not deliberately trying to make their code resource-inefficient. It’s just that they have a job to do, and they know just enough to get it barely done. Extras like bloat, security, and extendability are off the table. To handle them properly requires knowledge they do not have, nor the time to acquire it.

There are infinite ways to accomplish any task with a computer. But these days they are all wrapped in inconsistent primitives or calls which are often quite messy. If you understand what’s really below that level, you can figure out how to utilize it to do only exactly what you want. But if you don’t really understand how they work, you're left chaining them together and fiddling until the output approximates your goals.

This has been understood since the 60s, often called the software crisis. The higher we build the house of cards, the more disconnected each new generation gets from what's holding it all up.

The rise of don’t look before you leap philosophies of development fuelled the problem. Coders are in such a panic rush that they just have to grasp at what is easiest, but combined with tunnel vision, that is also usually what is pretty wasteful too.

If you’ve got some fiddling to do with a string, then most people pick the most popular primitives. When they realize that the combination does a bit of extra work, they discount it as a micro-optimization. It’s not; it's a lack of knowledge driving the deoptimization or bloat, as we like to call it.

If they knew more about how it worked underneath, they’d pick a different set and augment that with better code of their own. The output would be the same, but the wasted steps would be gone. They weren’t needed.

That’s a trivial example, but we see this with fiddling, calling libraries and APIs, allocations, polling, queuing and caching, pretty much all over the place. There is some reasonable way to encode the steps, but getting there requires understanding what the steps actually do. Not a guess or a vague understanding, but not enough knowledge to be able to implement them yourself.

Q&A sites fuel this, because they give people a magic sequence, thus offering them the option to just blindly use it.

If you knew better, most people would choose to write the code properly; it doesn’t take more time. If they write bloated stuff, it is because they didn’t know better.

When computers were really slow, bloat was obvious and often unworkable. As Moore’s law kicked in, bloat became just another consequence of the race to get more code out there. But as Moore’s law fades, we'll be forced to return to those early days when people coded with more precision. To do that, they’ll need depth. You can’t correctly pick the best underlying steps if you have no idea what’s actually underneath. Figuring that out will keep you ahead of the pack.