Digital Discussions

Every so often I write about software ideas that I would love to see developed, but I'm pretty sure that I won't get the chance to work on them myself. This is another one of those posts, but this time I thought I would mix it up with some of the underlying analysis, since that is a rather poorly understood part of software development.

As the first stage in development, analysis is often where projects go horribly wrong. If you don't know what you are building or it's just a disorganized collection of "stuff" then the design and development stages are unlikely to get it back on the rails again. It's a trainwreck. All of the best code in the world is useless if it is the wrong code for the solution.

The problem I would like to explore is online discussions. Obviously we have a long history of discussing, arguing and debating our different views in the non-digital realm. Once software developers set upon moving that to computers they seemed to focus only on the contributors. That is, they made is easy to join an online discussion and take part, but not so easy to read it back later. We do have blogs with comments, email groups and some dedicated discussion sites, but each one of these is optimized for the writer.

Trying to follow an interesting discussion online is difficult because of the loose structuring and constant noise. There are some quiet, low traffic sites that I quite like, but generally once a discussion becomes popular, the ability to read it and learn from it gets drastically reduced. That seems backwards.

So, we want to keep the site relatively easy for the writer, but also to capture enough structure of the discussion that the readers can filter out the content to whatever depth they find useful. The key word here is 'structure'. We want to collect more meta-information around the discussion that would help people to navigate it afterwards.

The earliest online discussions were essentially just lists of text. This only works well for a small amount of text. Many sites moved to simple trees that tied the children to a parent so the reader could get some sense about how the conversation fragmented. Later people began adding ratings, so that group-designated noise could be ignored. These changes helped somewhat but did not go far enough.

Looking at how things evolved we see that there are at least two different aspects of the conversational structure. One is trying to connect responses back to their origin, and the other is trying to filter out poor quality content like trolls. Let's tackle that second problem first.

A good discussion generally involves people with expertise on a topic. There are always people with more relevant contributions, and for most readers that forms the core of the discussion. The problem is that to distinguish a good comment from a bad one requires intelligence and knowledge of the topic. Right now, that is something that only a human can do, not a computer. Moderators are sometimes used, but an active discussion can produce such a huge amount of effort to fully moderate that it becomes impossible. What we would like to do is to tap into intelligence, but distribute it so that it does not become a burden.

When I am in a discussion with people, I usually a have a definitive opinion about who is currently forming the core. I'll accept that view as being quite common, so that gets us to the point where we might expect that many of the participants have this same opinion. If we mix in a bit of selectivity, then we can leverage this by having each discussion start with a very small number of contributors. One of the originators decides to invite several others to talk about a topic. As the discussion continues, any of that original group can add in new members. To balance out the potential for mistakes, several combined members can kick anyone out of the discussion.

That's fine and all, but it still allows for a specific subgroup to hijack the conversation. Rather than prevent this, we just channel it into something more positive, so what we also need is a way for parallel discussions to exist. At any point, anyone can start a parallel discussion with a completely different group of people that intertwines with the original discussion. That hopefully reduces any incentives to derail the original conversation.

For example, a group of three well-known physicists starts talking about the viability of quantum computers. A parallel discussion gets formed by some PhD students and another one by just techies with an opinion. If the reader wants to see the collective discussion it can be presented as essentially one big tree or even a list. However if they just are curious about the first two groups, they can filter out the other stuff. In a sense they can traverse as deeply into the parallel groups as they want if they're interested in how the topic is going to be perceived by the masses, or they could just focus on the meatier parts.

Obviously parallel groups need to attach to the contributions in their patent discussions, but there doesn't need to be a cap on depth. Groups could form off the original discussion, and other groups could form off of those, and so on. The people taking part in the discussion have formed themselves into a tree of sets of invited contributors, structured by how the contributors feel about each other, and implicitly about how they feel about the quality of their contributions.

It becomes easy to see that most people wouldn't accept invitations from trolls and that at the margins of the discussion irrelevant material can easily be ignored. But at the same time, nobody is explicitly precluded from joining in, they might just not get significant readership because of what they say.

This extra structure is a bit more work for the contributors, but it naturally fits into with how people are drawn in and out of conversations. It also controls the noise.

The other aspect is also about structure. The simple tree structures that currently exist are a crude means to connect the sub-threads of a discussion, but once again we are not collecting enough structural information to be really useful.

Most conversations start by either making some explicit points about a topic or asking a series of questions. The question/answer structure is fairly powerful, which is why we see it used so often for interviews. It binds together loose threads, without having to fill the space with a lot of transitions.

Given both the point/counter-point and question/answer relationships what we would like to do is make them explicit. To do this we replace the unformatted blocks of text with these response types. That is, any entry in the discussion is either a point or a question and the writer has to indicate which.

Still, since the underly type is text, people wouldn't be particularly disciplined in their contributions. We have to expect that. We have to build on that.

Thus a discussion would start with one person choosing their 'point' or 'question' and typing in their text. From there, any responder would highlight some 'subportion' of that text, choose point or question again and then type in their response.

Thus, much like Quora, the discussion could start with a question. But there might be multiple questions, and implicit sub-points implied in the text. Each respondent could highlight a word, sentence, paragraph or whatever and then attach their points and questions to just that part. The discussion could also start with an essay. People could question the assertions or make a point about terminology usage. They could address three different issues independently, or just keep them all together in the same point. The writers would all find their own ways of utilizing this new structure.

Often in discussions people include both positive and negative references. So besides points and questions there would also be links. To further help, each of these three basic response types can be flagged as positive or negative. If someone wrote a couple of paragraphs that you mostly believe, but there was a flaw with some terminology, you could add in a negative point about that specific issue, while also adding an overall point about how you agree, and adding a link that substantiates a couple of their points.

It's not that much more work then hitting reply and writing one big, hard to read, textual entry. Just highlight a portion of the original text, fill out two dropdowns and then go to town.

Being able to strip out all of the questions really sets the breadth of the conversation. Being able to see how the topic breaks into threads and knowing who is on which side sets the context. You might only want to see the positive points, or the negative ones. It offers a lot of depth for the reader to filter down quickly to get knowledge.

Getting back to the quantum computer example if you are curious as to why some experts believe it to not be possible, you can extract these points, even if there are whole sub-topics that you don't understand. You might zero in on a sub-discussion between students on the current definition for a part of the theory that is confusing. All of the extra structure helps the reader navigate what has been written.

The value of software comes mostly from the underlying data that is collected. It's faster to just collect the central independent points and dump them into a simple list, but it is far more useful to collect higher level structural relationships as well. There is however a danger with this approach. It is easy to just create new arbitrary response types, for example. But if these are fuzzy or overloaded, they might actually distort the structure and cause a new set of problems. Because of that, structural modelling needs to justify any sort of categorization or complex relationship. The right ones provide value, the wrong ones add noise.

From a data-structure perspective the above is most likely a graph with polymorphic nodes, although it might also be acyclic. What that means in practice is that care must be taken to insure that the performance remains reasonable as the discussions grow. Obviously implementing an indexed list is both easier and has no such concerns. Too often programmers choose that easy route at the expense of their users. In the earlier generations of software this was not surprising, but given that we've already written this stuff multiple times, the advancement of our industry requires that we start tackling the more complex problems. Sophisticated user functionality is difficult to implement well, but it is also a far more appropriate solution.

Putting it all together, this design looks very much like a mix between Quora, discussion sites, blogs with comments and if you add privacy, even email. It's a little more effort for the writers, but it now captures enough structural data that it is a huge improvement for the readers. Unfiltered, it is the usual chaos, but with minimal work you can utilize this new data to focus in on extracting significant knowledge.

The general context is that in software we tend to focus too heavily on the independent data points, easily missing the more valuable structural relationships connecting these points. Collecting data used to be expensive, which no doubt resulted in us cheating. Now that data costs have come way down, we need to revisit the ways we analyse problems and then model the data. It's not enough to just zero in on the trivially obvious aspects anymore. The next generation of systems should be ones where we pursue complex data models to achieve sophisticated functionality that go deeper into solving the user's problems. Code is nice, data is useful, but really deep knowledge comes from the sophistication of the data model.

Intelligence

I've always been curious about intelligence; both as it relates to our species's ability to reason about its surroundings and to how much we can actually fit into software. I've done little bits of reading, here or there, for years but it's never been intentional. As such I've built up a considerable number of disjoint knowledge fragments that I thought I'd try to organize into something a little more coordinated. Given my utterly amatuer status on this topic any comments, scepticism, corrections or other fragments are highly welcome.

I'll start with a long series of statements, then I'll do my best to explain what I think is both right and wrong about them:

1. Life forms are dynamic, driven internally.
2. Life forms choose how to spend their energy.
3. Smart is the sophistication of that choice.
4. The choice can be very simple, such as fight or flee.
5. The choice can be on-the-fly, thus choosing the best current option.
6. The choice can be based on internal models, this supports longer term benefits.
7. The models are dynamic, both the data and the structure itself can adapt and change.
8. Intelligence is the ability to create models of the world and benefit from them. To be able to adapt to changes faster than evolution.
9. Life forms can be self-aware.
10. Self-awareness makes a life form smarter.
11. Internal questions are spontaneous queries directed at models. They are not direct responses to external stimuli.
12. The purpose of asking internal questions is to refine the models.
13. Consciousness is a combination of models, being self-aware and asking internal questions.
14. Humans are not entirely logical, their models get created for historic and/or emotional reasons.
15. Some understanding comes initially as instinctive models.
16. Some understanding gets built up over time from the environment and cultures.
17. Common sense is only common to inherited cultural models.
18. Contradictions arise from having multiple models.

Definitions

I'll start first with expanding out a few basic definitions  and then gradually delve into the more interesting aspects:

An object can be static, like a rock. With respect to its surroundings, a rock doesn't change. An object can be dynamic, like an ocean. It is constantly changing, but it does so because of the external forces around it. If an object can change, but the reason for the change is encapsulated in the object itself, then I see this as some form of life. In order to accomplish the change, the object must build up a supply of energy. A cell is a good example, in that it stores up energy and then applies that in a variety of different ways, such as splitting, signalling other cells or repairing itself. A machine however is not alive. It may have sophisticated functionality, but it is always driven by some external life form. It is an extension.

One big problem I am having with the above definition is hybrid cars. They store up energy and they choose internally whether to run on electric or gas power. A car is just a machine and it's destination is the choice of the driver, but that energy switchover collides with my first two points. I'll try to address that later.

I generally take 'smart' to be an external perspective on an object's behaviour and 'intelligent' to be the internal perspective. That is, something can behave smartly in its environment even if it doesn't have a lot of intelligence, and intelligent beings aren't necessarily being smart about what they are doing externally. Splitting the two on external vs. internal boundaries helps clarify the situation.

Given the external perspective, a life form that is smart is one that adapts well to its environment in a way that benefits it as much as possible. Depending on the environment, this behaviour could range considerably in sophistication. Very simple rules for determining when it is autumn help Maple trees shed their leaves in the fall. It's likely that behaviour is a simple chemical process that gradually accumulates over a period of time to act as a boolean switch. Cluster enough cold days together over a month or so and a tree decides to hibernate. That may be smart, but we generally do not define that as intelligence. It is a reactionary triggering mechanism to external stimuli.

Some life forms make considerably more sophisticated choices. Not just reactions, but there is a long term component there. In order to project out into the future one needs some data about both the present and the past. But this data can't be unstructured. Since it is internal, at best it can be some combination of symbolic representations and explicit sensory input. The structure is no doubt chaotic (naturally organized), but exists never the less. It is this resource that I call a model. A very simple example might just be a crude map of where to go to get food. A more complex one might be handling social organizational relationships. Practically, it is likely that models get coalesced into larger models and that there are several different internal and external ways in which this is triggered. Still, it is expected that life forms that have this capacity would have multiple models, that are in different stages of development.

The purpose of keeping a model is that it can be used to project future events. It is essentially a running total of all exposed relative information. Once you know that night is coming in a few hours, you can start arranging early to find a safe sleeping place. Certainly, a great many creatures have crude models of the world around them. Many apply crude logic (computation?) to it as well. We have shown a strong preference for only accepting sufficiently advanced models as truly being intelligence, but that is likely a species bias. Sometimes people also insist that the ability to serialize paths through a model and communicate it with others who can then update their own understanding is really what we mean by intelligence. The ability to transfer parts of our models. As far as we know, we're the only creatures with that ability. Other animals do communicate their intentions, but it seems restricted to the present.

Douglas Hofstadter made an interesting point in "I am a Strange Loop" about the strength of self awareness. A feedback loop can have very deep and interesting behaviours. Life forms whose models include an understanding of their own physical existence can no doubt make more sophisticated choices about their actions.

Self-awareness clearly plays a crucial role in behaviour. I'm just not sure how deeply that goes, most mobile creatures have at least the fight or flee responses and it seems that fleeing could require self-awareness. An attempt to save oneself. Perhaps, or it could just be wired into the behaviour at the instinctual level? Or both.

The really big question about life forms has always been consciousness. Some feel that it is just self-awareness, but I think you can make many valid objections to it being that simple. As I mentioned above, lots of creatures will flee at the first sign of danger, but some of those seem simple enough, like insects, that it is hard to imagine calling them intelligent, let alone conscious. Their behaviour most often seems to be statically wired. It doesn't really change quickly, if at all.

Some others feel it is just a thread or stream of conscious thought. That really only works if the stream is aware of itself, and is essentially monitoring itself. Still it doesn't seem to be strong enough to explain consciousness properly, and it doesn't provide any reason why it would have evolved that way.

Recently I've come to think of it as the ability to internally ask questions of one's own models. That is, it is a stream of self-awareness thoughts, that frequently question the model on how (and why) to expend energy. A non-conscious life form would just react to the external changes around it directly via the gradual buildup of rules or simple models. A conscious life form would also question those models and use those questions to further enhance them. Simply put, a conscious life form is curious about itself and it's own view of the world.

That opens the door to someone correctly arguing that there is a big hole in this definition. You could point out that people who never question their own actions are not conscious. For example, someone who joins a cult and then does crazy stuff only because the leader says that they should do it. I'm actually fine with the idea that there are degrees of consciousness, and that some people are barely conscious, and that some people can be turned that way. It matches up with that sense I got when I was younger that suddenly I had 'emerged' as a being. Before that I was just a kid doing whatever the adults specified, then almost suddenly I started getting my own views. Questioning what others were saying or what I was told. I can accept that as becoming gradually more conscious. So it isn't boolean, it is a matter of degrees.

Asking questions does show up in most fiction about artificial intelligence. From Blade Runner to Ex-machina, the theme of the AI deviating from expectations is always driven by their questioning the status quo. That makes sense in that if they just did what they were told, the story would be boring. It's just the story of a machine. Instead, there needs to be tension and this derives from the AI coming to a different conclusion than at least one of the main characters. We could see that as "questions that are used to find and fill gaps in their models". Given that each intelligent life form's model is unique, essentially because it is gradually built up over time, internal expansions would tend towards uniqueness as well. No doubt the models are somewhat sticky, that is it is easier to add stuff then it is to make large modifications. People seem similar in this regard. Perhaps that is why there are so many disagreements in the world, even when there are established facts?

Artificial Intelligence

Now that I've sort of established a rough set of definitions, I can talk about more interesting things.

The most obvious is artificial intelligence. For a machine to make that leap, it would have to have a dynamic, adaptable model embedded within it, otherwise it is just a machine with some embedded static intelligence. To test this, you would have to insure that the model changed in a significant way. Thus, you might start with a blank slate and teach the machine arithmetic on integers. Then you would give it a definition for complex numbers (who are structurally different than integers) and see if it could automatically extend its mathematical definition to those new objects.

These days in software development we always make a trade-off between static finite data types and having less structure in some generic dynamic representation (a much rarer form of programming but still done at times like in applications like symbolic algebra systems), but I've never seen both overlaid on top of each other. It's currently a trade-off.

A fully dynamic structure as a massive graph would give you some form of sophisticated knowledge representation, but that would still not be artifical intelligence as we often think of it. To get that next leap, you'd need this thread of constant questioning, that is gradually refining the model. Testing to see if this exists would be problematic, since by definition it isn't connected to external events. My guess is that you need to use time. You give the machine a partial model with an obvious gap, and latter you return to see that the gap has been filled in. An important point here is that at bare minimum it would require two different tests spaced out by time, and as such shows a huge weakness in the definition of a Turing test. That latter test would have to be really long, in order to allow for the modifications to happen, and possibly by definition it wouldn't because the subject would be preoccupied by the test itself.

A second critical point is that any life form that is questioning itself and its surroundings is inherently non-controllable. That is, you could never guarantee that the answers to any of those questions always came out positive in your favor. As such, any fixed set of rules like Isaac Asimov's three laws of robotics are probably not enough to guarantee a domesticated and benevolent artificial intelligence. It's more like a slot machine, where a jackpot means big trouble. Once the wrong question is asked, all bets are off. A conscious being always has enough free will to extend the model in a way that is not in the best interests of those around it. It is looking after itself with respect to the future, but that forward projection could literally be anything.

Self-driving and self-repairing cars

The current state of self-driving cars leads to some interesting questions. The version at Google clearly models the world around it. I don't know how dynamic that modelling is, but it's aligned in a way that could potentially make it a life form. The differentiator is probably when the car encounters some completely new dynamic object, and not only adds it to it's database but also all it's new, previously unseen, properties as well. That update would have to alter both the data and its structure. But to arrive at a life form it might not even need to be dynamic...

At some point, a self-driving car will be able to drop you off at work, and then find a cheap place to park during the day. What if one day it has an accident after it left you? It could easily be wired with sensors which might notify it that the fender is dented. At this point, it could call up a garage, schedule an appointment and afterwards pay for it with your credit card. This is fairly autonomous, is it a life form now? One could argue that it is still reliant on external objects for the repair, but we have the technology to bypass that. What if it had a built-in 3D printer and some robot arms? At this point it could sculk away somewhere and repair itself, no one needs to know.

So let's say it's a normal day, the car drops you off and you go to work. If later in the afternoon the police show up at your office and start questioning you about a fatal hit and run accident, what happens? You tell them that you've been at work all day, they ask to see the car. You summon it, and it is in pristine condition, nothing wrong. What if, as part of the repair, the car automatically deletes any logs. You ask the car if it got into an accident. It says no. At this point, you would not expect to be held liable for the accident. You weren't there, there isn't any proof and the car itself was involved in the cover up, not you. It does seem that with the right combination of existing technologies that the car could appear to be fully autonomous. Making it's own decisions. Repairing itself. That, at this point has it crossed the threshold between machine and life form? It also seems as if it doesn't even need to be dynamic to make that leap? It has just become smart enough that it can decide how to spend it's energy.

Origins of life

If life is just an internally dynamic process that got set in motion, a crucial question is how and why did it all start? My naive view is that somehow bundles of consumable energy coalesced. From there, all that would be necessary to go forward is a boolean mechanism to build up containment around this energy. Given those two pieces, something like a cell would start. Initially this would work to fully contain the current bundle, and then gradually extend out to more sophisticated behaviours, such as gathering more energy. In that sense, one could start to see how the origins could be derived out of the interplay between mechanical processes.

In a way that's similar to the issues with a self-driving car. What's radically different though is what set it all in motion. Our perspective of evolution implies that some of these internally dynamic processes are a consequence of external dynamic processes. That is, the mechanism to attempt to contain, and later collect energy is a normal part of our physical world. Something akin to saying that it is similar to the way weather and water shape the landscape, but in this case the external force gradually morphs to maintain its own self interests. This is an extraordinarily deep issue, in that it differentiates the line between machines and life forms. If a self-driving car does become a life form, it is because we set it in motion. We're in motion because of all the life forms before us. But before that is seems as if we need to look to physical processes for an explanation?

Dynamic Relationships

There are sometimes very complex relationships between different things. They might be continually dynamic, always changing. Still, they exist in a physical system that bounds their behaviour. Computation can be seen as the dynamic expression of these relationships. In that sense it is a complex  time-based linkage whether explicit or not, that relates things together. We have some great theories for what is computable and what is not. We know the physical world has boundaries as well, a classic example is limits on exponential operations such as paper folding. Over the millennia we have refined all sorts of ways of expressing these relationships, and have dealt with different degrees of formality as well as uncertainty. In that we externally model the relationships and then apply them back to the real world, in order to help us make better choices. If that sounds incredibly like my earlier description of intelligent life forms, it is no coincidence.

We started out communicating parts of the models, but now we are developing ways to work on the models externally. If the addition of internal models helped us along, the ability to create shared external ones is central to our continued survival. Change is the default state of the universe, and the planet; it can be locally faster than many life forms can handle. Thus speed and also accuracy are essential for dynamic processes to continue. The sooner we figure out how to collectively model the world around us, the more likely we will be prepared for the next oncoming major change. There will always be a next major change.

Getting back to computation, we can express any mechanism as a collective series of relationships. If we think of the flow of the universe as going from chaos to natural organization and then back to chaos again, this could be seen as permuting most forms of relationships. That is, eventually, somewhere, a given relationship is going to be instantiated. Most likely more than once. If that relationship exists, then the mechanism exists and if that is the ability to contain energy, then we can see how the dynamic nature of the universe can morph, rather unintentionally, into some internally dynamic process. At that point, the snowball just starts rolling down the hill...

Final thoughts

Given my lack of concentrated depth on this subject, my expectation is that my views will continue to change as I learn more. I am not certain of anything I said in this post, but I do find this line of reasoning to be interesting and it seems to fairly explanatory. It is, explained in the current context, just a minor model that has been gradually accumulating accidently over time. Lots of gaps, and plenty of overlap with the many other models floating about in my thoughts.

We have arrived at a point in our development as an intellectual species were it seems that we really need a deeper understanding of these issues. They need to be clarified, so we can work upwards to correct some of our accidental historic baggage. That is, our loosely organized collective model of the world as it is chaotically distributed between the billions of us is not particularly accurate. With lower population densities and less technological sophistication this wasn't a problem, but that seems to have changed. Every person will predict the future differently because their starting points, their models, will differ. Those differences lead to conflict, and enough conflict leads to us into taking a rather massive leap backwards. History implies this is inevitable, but I tend to think of it as just one more changing aspect of the world that we have to learn to adapt to. Intelligence or our collective lack of it, are crucial to understanding how we should best proceed in the future.

Privacy

There are some things that I only tell my wife. There are things that I share with my family, and there are other things that I share with my closest friends. It's not that any of these things are bad, or dangerous, or in any way harmful to society, but rather that I deliberately choose to manage how I present myself to different people.

One of the pillars of a close personal relationship with someone else is that you trust each other enough with private information. If everything is already in the public, you lack the ability to build up these deeper relationships.

Thus controlling the 'scope' of the information that we share about ourselves is fundamental. I let my goofy side hang out for my friends, but at work, I try to appear professional. I want to be seen as happy and confident to most people, restricting my interactions on bad days to those closest to me. Sometimes I even bite my tongue and don't say what immediately comes to mind, just because it doesn't fit. It would be awful if every little tidbit about me was constantly floating about in a digital tsunami. I don't want to be an 'open book'. Some aspects of my life are for selected people only.

Long before computers disrupted the world we used to be the masters of our own information. We could choose to tell stuff to certain people, and if we picked them correctly that information would not end up in the gossip mill. It would not go public. What was said between friends stayed between friends. If something was leaked, then it was because you trusted the wrong person.

As more and more of our interactions become digital, we started losing control of our private information. Now it is far too easy for any unknown person to alter the scope of what we are communicating. You no longer know who you are implicitly trusting anymore. That currently makes private digital communication impossible.

If we want to restore our necessary privacy in the digital age, we're going to have to set down some very specific rules about how information is properly shared. The most obvious one is that unintentional intermediaries to anyone's data should never, ever, share it. They should not alter that person's original chosen scope. They should not steal control from them.

That is, a system administrator who is not a direct party to a conversation between two other people in email should never give those emails over to a third party. Communications between a group of people is meant only for those people, and it is only one of those people that can widen the scope of the information. If someone was the intended audience, then they have the right to mess with scope, but if they were not, it should be considered morally wrong for them to 'spy' on others.

This is very simple in principle. If you log into a website and fill out the registration forms, you are having a conversation with the company that created the website. If later, you are utilizing their web site to have a private conversation with a couple of other people, that conversation no longer includes the company that runs the site. They should not alter the scope.

If you submit a post to a public forum, it's a public conversation. If you post something to a small select group, then it is not public and should not be visible to the public unless one of the members of that group explicitly makes it so.

Only the people in the discussion can change the scope, no one else. If we set this as the convention, then some of our privacy issues go back to normal. We can start building the next generation of technology that enforces this behavior. We can also easily determine when something leaked was morally wrong and choose not to make it worse. To be a whistleblower, you have to be on the inside of the conversations, not just stumble across them while spying on others.

There is, of course, the rather ugly and remaining law enforcement issues. Previously, in order to spy on people you had to both have some evidence that they were up to no good and you had to get permission from the courts (essentially a sober third-party) before you could proceed. All of that was to prevent any collected information from being abused, and certainly, history shows again and again that abuse will always be rampant.

We need to return to that as well. No organization should ever be collecting mass amounts of any type of information on people in the off chance that it might just be useful for punishing them later. That's so far off the scale of decent behavior that I've really been surprised that more people aren't disgusted by it. A government peeping Tom, located at every house, watching everybody, is beyond creepy.

We need to regain control over the scope of our information. We need to return to a set of rules that doesn't glorify peeping Toms. We need to do this because we respect the rights of our citizens to be able to present themselves as they choose. We need to respect their choices. It's really that simple.

Sweatshops

Decades ago, when I got out of school, I remember being very wary of accidentally working for a software sweatshop. 

It was a frequent conversation between new grads. All sorts of industries have variations on this theme but in programming, sweatshops had already started to become significant career hazards.

What we wanted to avoid in those days were jobs with crazy hours, hacking at mediocre code while under humiliating micromanagement. The sort of work that makes you wish you had pursued an alternate career. Maybe forestry.

It's not that I minded long hours when I was young, but given that I am essentially donating my personal time to a company, in exchange for this I expected to work on meaty problems that would help my career. Most code is fairly routine, and with plenty of practice you can shut off your brain and just pound out stuff that kinda works. The jobs that I have loved over the years have been the ones where we went deeper than that. Where we didn't just 'barely' get things working, rather we carefully crafted sophisticated solutions that solved real problems for people. Partially solving some aspect of a problem is easy, refining that to work flawlessly is exponentially harder, but it's those deep solutions that people really need; that truly make their lives easier.

Of course, the type of code I wanted to write wasn't just chucking some data onto the screen and then saving it a few minutes later. A sloppy GUI to support editing a crappy data model is just automating something already broken. Deep solutions require complex modeling so that they can add value on top of the basic data collection process. Really big, but low-quality data isn't going to be mined for any real insights, there is just too much noise embedded within it. Good user interfaces also require deep insights in order to properly structure the workflow. Littering the screen with tables, dropdowns and buttons is easy for the programmers, but horrible for everyone else.

Besides the long hours and the weak solutions, sweatshops also tended towards hovering over their employees, while trying not to let them know much about the higher-up planning. My guess is that this is done for two reasons.

The first is to keep squeezing as much out of everyone as possible. That's sort of an odd irony since programming is mostly intellectual work; the closer programmers get to burnout, the worse the quality of their efforts. But a really intense sweatshop doesn't get this, so they think continually throwing the development into "crunch mode" will actually net them more results; they'll somehow get a better return on their salaries.

Of course, that directly leads to the second reason. The continual whipping to extract more from the coders will always result in a degenerating code base. If the initial version of the system wasn't too bad, but the next version is expected to be a total bugfest, then it is a strong indication of a sweatshop. When 'mediocre' is the best possible outcome, that turns the morale of many of the programmers. My sense has been that in order to prevent the programmers from realizing this, you have to keep them super busy. If they're just too tired to think at the end of the day, then they are less likely to reach the conclusion that their jobs suck. That they are just churning code.

Thus getting caught up in a sweatshop is a somewhat self-perpetuating cycle. They work you hard enough that you are too tired to escape, so you just hang in there on a day-to-day basis until way too much time has passed or you melt down. If you're really screwed, you don't even have any relevant market experience that would make you attractive to other employers. With no visible options, you'd have to take a serious risk in order to leave.

So, of course, as new grads, we really wanted to avoid this horrific fate. If you want a long career in building modern software, we quickly realized that you had to keep finding good positions that didn't burn you out but also didn't lock you into some unsellable skill.

Decades ago, there were lots of good development shops that were focused on building up reliable systems at a reasonable pace. That, of course, all changed with the dot com era. All of the sudden programmers became scarce, and programming shifted from deep intellectual construction to just an endless panic. Tech has always suffered from stupid fads, but suddenly the fads themselves became the bleeding edge, and the cost to play was that you had to work for a sweatshop.

Not surprisingly, the software being produced dropped heavily in quality. There are always counterexamples, but collectively the mentality shifted away from taking enough time to build something meaningful, to just getting anything out there and then trying to make it look pretty later so consumers don't notice that it's broken.

It certainly hasn't helped that the methodologies drifted into being hyper-reactive towards the "stakeholders". The idea being to cover one's ass when building a mess by rigorously ensuring that the blame can be placed on the clients. If we did "exactly" and "only" what they wanted (even if they don't understand the consequences) then how can that be wrong? To me, that always seemed like a Doctor asking patients to self diagnose their own illness, so that they can't be blamed later if it went wrong. The clients aren't professional Software Developers, so why would they know how to build a working system in a reasonable time? They don't even have a realistic sense of what 'working' and 'reasonable' means. It's a classic recipe for disaster.

Lately, I've been watching the job market again and I am saddened by the number of sweatshops out there. Even worse is that many of them actually brag about it in their job descriptions. Job ads that a few decades back would have resulted in 0 serious resumes are suddenly extremely popular. Programmers seem attracted to these like moths to a bonfire. Companies are so over the top that they add perks like "frequently paying for dinner" as if they were actual benefits, but to me, it seems to be enshrining a rather sick work/life balance. If their priorities are that skewed, the work is probably pretty shoddy as well. I love to work, but not enough that I don't feel bad about missing out on the rest of what life can offer. In those extreme periods in the past where my years just disappeared in an instant, I’ve looked back with disappointment. Coding is fun, but so are a lot of other things.

It seems like we're in an epidemic of the sort of career-destroying culture that as a new grad I was trying so hard to avoid. Programming shouldn't be a continual high-stress hackfest. There are always times in any system's life where it becomes difficult and stressful, but the measure of a good job is that those times are rare, not constant. If the development is thought through and the process is organized, then most days in programming are spent carefully working on building up reliable code. You need to spend the time to make things of decent quality, they don't just get created by accident. When the work is crazy, you might feel proud that you survived, but you can never really feel proud about creating that inevitable ball of duct tape and bandages. It is just ugly. Myself, I am driven to build things, software is just the medium I choose to work in. If I am going to spend a lot time building, then I need to be proud of my creations.

High-stress, of course, leads to rather poor thinking, and code is only ever as good as the thinking that went into it. Computers are "multipliers", thus bad code realistically just makes the problems worse. You can hide that, but only for a while. So it's been clear for a long time now that even if the code started out okay, throwing stressed out teams at extending it is detrimental in the long run. Setting up an environment that glorifies stressed out teams is unlikely to be sustainable (with a few notable exceptions). If you care about what you create, then you have to find an environment that does too.

In many ways, it has been extremely frustrating to watch this continuing madness grow, even as we are becoming dangerously reliant on software. It's like watching a car crash in slow motion. The pools of good jobs are gradually drying up, getting replaced by sweatshops. Each year more people proclaim crazier and crazier approaches to building software that pivot on exploiting some obviously bad shortcut or a hopelessly redundant time wastage. Eventually, enough people catch on that the latest fad isn't helping, but by then something even worse has come along to replace it. We're so trapped in bad approaches right now, so deeply, that many newbies think they are actually the core of our profession.

The truth is that at some point, with enough variation, you start to realize that most of the industry's persistent problems are self-inflicted. And by the time you finally get to this realization, you're heavily wishing that you had pursued an alternate career. Like forestry.

Encapsulation

One of the strongest, but possibly least understood principles of object-oriented (OO) programming is 'encapsulation'.

The OO paradigm explicitly injects structure on top of code which allows programmers to build and maintain considerably larger programs than in the distant past. This extra level of organization is the key to managing complexity. But while this amplifies our abilities to build big programs, there is still a 'threshold of complexity' that once crossed will quickly start to degrade the overall stability of the development project, and eventually the software itself.

An individual programmer has fixed limits on how quickly they can build up instructions and later on how quickly they can correct problems. A highly-effective team can support and extend a much larger code base than the sum of its individuals, but eventually the complexity will grow beyond their abilities. There is always some physical maximum after which the work becomes excessively error prone or consistently slower or both. There is no getting around complexity, it is a fundamental limitation on scale.

However we can significantly minimize it, to prevent crossing the threshold for as long as possible. This obviously comes from the strict avoidance of adding any artificial complexity, such as special cases, twisted logic, arbitrary categorizations and other forms of disorganization. That helps, but there is another approach as well.

Encapsulation can be seen as drawing a 'black box' around a subset of a complex system. That box prevents anyone from outside in seeing the inner workings, but it also ensures that what's on the inside is not influenced by random outside behaviour. The inside and outside are explicitly walled off from each other, so that the only interaction is a precisely defined 'interface'.

To get the most out of encapsulation, the contents of the box must do something significantly more than just trivially implement an interface. That is, boxing off something simple is essentially negative, given that the box itself is a bump in complexity. To actually reduce the overall complexity, enough sub-complexity must be hidden away to make the box itself worth the effort.

For example, one could write a new layer on top of a technology like sockets and call it something like 'connections', but unless this new layer really encapsulates enough underlying complexity, like implementing a communications protocol and a data transfer format, then it has hurt rather than helped. It is 'shallow'. What this means is that for any useful encapsulation, it must hide a significant amount of complexity, thus there should be plenty of code and data buried inside of the box that is no longer necessary to know outside of it. It should not leak out any of this knowledge. So a connection that seamlessly synchronizes data between two parties (how? We don't know) correctly removes a chunk of knowledge out of the upper levels of the system. And it does it in a way that it is clear and easy to triage problems as being 'in' or 'out' of the box.

Once a subset of the complexity has been fully encapsulated, and is easily diagnosable, it can be vetted and ignored for the moment. That is, if the connections library is known to work for all current circumstances used by the project, then programmers don't have to revisit the internals, unless they want to expand them. That's the real power of encapsulation. For a little bit of extra thinking and work, some part of the system is carved off and put to rest. Later, because of reuse or enhancements, it may make sense to revisit the code and widen its functionality, but for the moment it is one less (hopefully large) problem to deal with. The system is more complex, but the system minus the encapsulated parts is relatively less complex, and thus it is easier to work on what remains unsolved.

In little programs, encapsulation isn't really necessary, it might help but there just isn't enough overall complexity to worry about. Once the system grows however, it approaches the threshold really fast. Fast enough that many software developers ignore it until it is way too late, and then the costs of correcting the code becomes unmanageable.

It is for that reason that many seasoned developers have learned the habit of encapsulating early and often. They essentially have a second 'editing pass' on any new code that is aimed at breaking off any potentially encapsulated parts into their own independent chunks. This is essentially 'partial' encapsulation. The code is partitioned, but not rigorously enforced. Doing this regularly and reliably means that at some point later, when it is required, the code can easily be enhanced to full encapsulation.

If you look at any well-written code base, the related instructions tend to be localized, and there is at least an implicit organization that cleanly separates the underlying pieces. That is, the individual lines of code appear in the sub-parts of the system, exactly where you would expect them to be located. Perhaps, some of the code is formally encapsulated, but often that degree of rigor is not necessary at the current stage of the code base, so its just partial. Contrast that with code containing an over abundance of hollow encapsulation or just what seems to be code randomly located anywhere and you can see why this principle is so important in keeping the code useful.

To build big systems, you need to build up a huge and extremely complex code base. To keep it manageable you need to keep it heavily organized, but you also need to carve out chunks of it that have been done and dusted for the moment, so that you can focus your current efforts on moving the work forward. There are no short-cuts available in software development that won't harm a project, just a lot of very careful, dedicated, disciplined work that when done correctly, helps towards continuing the active lifespan of the code.