Privacy

There are some things that I only tell my wife. There are things that I share with my family, and there are other things that I share with my closest friends. It's not that any of these things are bad, or dangerous, or in any way harmful to society, but rather that I deliberately choose to manage how I present myself to different people.

One of the pillars of a close personal relationship with someone else is that you trust each other enough with private information. If everything is already in the public, you lack the ability to build up these deeper relationships.

Thus controlling the 'scope' of the information that we share about ourselves is fundamental. I let my goofy side hang out for my friends, but at work, I try to appear professional. I want to be seen as happy and confident to most people, restricting my interactions on bad days to those closest to me. Sometimes I even bite my tongue and don't say what immediately comes to mind, just because it doesn't fit. It would be awful if every little tidbit about me was constantly floating about in a digital tsunami. I don't want to be an 'open book'. Some aspects of my life are for selected people only.

Long before computers disrupted the world we used to be the masters of our own information. We could choose to tell stuff to certain people, and if we picked them correctly that information would not end up in the gossip mill. It would not go public. What was said between friends stayed between friends. If something was leaked, then it was because you trusted the wrong person.

As more and more of our interactions become digital, we started losing control of our private information. Now it is far too easy for any unknown person to alter the scope of what we are communicating. You no longer know who you are implicitly trusting anymore. That currently makes private digital communication impossible.

If we want to restore our necessary privacy in the digital age, we're going to have to set down some very specific rules about how information is properly shared. The most obvious one is that unintentional intermediaries to anyone's data should never, ever, share it. They should not alter that person's original chosen scope. They should not steal control from them.

That is, a system administrator who is not a direct party to a conversation between two other people in email should never give those emails over to a third party. Communications between a group of people is meant only for those people, and it is only one of those people that can widen the scope of the information. If someone was the intended audience, then they have the right to mess with scope, but if they were not, it should be considered morally wrong for them to 'spy' on others.

This is very simple in principle. If you log into a website and fill out the registration forms, you are having a conversation with the company that created the website. If later, you are utilizing their web site to have a private conversation with a couple of other people, that conversation no longer includes the company that runs the site. They should not alter the scope.

If you submit a post to a public forum, it's a public conversation. If you post something to a small select group, then it is not public and should not be visible to the public unless one of the members of that group explicitly makes it so.

Only the people in the discussion can change the scope, no one else. If we set this as the convention, then some of our privacy issues go back to normal. We can start building the next generation of technology that enforces this behavior. We can also easily determine when something leaked was morally wrong and choose not to make it worse. To be a whistleblower, you have to be on the inside of the conversations, not just stumble across them while spying on others.

There is, of course, the rather ugly and remaining law enforcement issues. Previously, in order to spy on people you had to both have some evidence that they were up to no good and you had to get permission from the courts (essentially a sober third-party) before you could proceed. All of that was to prevent any collected information from being abused, and certainly, history shows again and again that abuse will always be rampant.

We need to return to that as well. No organization should ever be collecting mass amounts of any type of information on people in the off chance that it might just be useful for punishing them later. That's so far off the scale of decent behavior that I've really been surprised that more people aren't disgusted by it. A government peeping Tom, located at every house, watching everybody, is beyond creepy.

We need to regain control over the scope of our information. We need to return to a set of rules that doesn't glorify peeping Toms. We need to do this because we respect the rights of our citizens to be able to present themselves as they choose. We need to respect their choices. It's really that simple.

Sweatshops

Decades ago, when I got out of school, I remember being very wary of accidentally working for a software sweatshop. 

It was a frequent conversation between new grads. All sorts of industries have variations on this theme but in programming, sweatshops had already started to become significant career hazards.

What we wanted to avoid in those days were jobs with crazy hours, hacking at mediocre code while under humiliating micromanagement. The sort of work that makes you wish you had pursued an alternate career. Maybe forestry.

It's not that I minded long hours when I was young, but given that I am essentially donating my personal time to a company, in exchange for this I expected to work on meaty problems that would help my career. Most code is fairly routine, and with plenty of practice you can shut off your brain and just pound out stuff that kinda works. The jobs that I have loved over the years have been the ones where we went deeper than that. Where we didn't just 'barely' get things working, rather we carefully crafted sophisticated solutions that solved real problems for people. Partially solving some aspect of a problem is easy, refining that to work flawlessly is exponentially harder, but it's those deep solutions that people really need; that truly make their lives easier.

Of course, the type of code I wanted to write wasn't just chucking some data onto the screen and then saving it a few minutes later. A sloppy GUI to support editing a crappy data model is just automating something already broken. Deep solutions require complex modeling so that they can add value on top of the basic data collection process. Really big, but low-quality data isn't going to be mined for any real insights, there is just too much noise embedded within it. Good user interfaces also require deep insights in order to properly structure the workflow. Littering the screen with tables, dropdowns and buttons is easy for the programmers, but horrible for everyone else.

Besides the long hours and the weak solutions, sweatshops also tended towards hovering over their employees, while trying not to let them know much about the higher-up planning. My guess is that this is done for two reasons.

The first is to keep squeezing as much out of everyone as possible. That's sort of an odd irony since programming is mostly intellectual work; the closer programmers get to burnout, the worse the quality of their efforts. But a really intense sweatshop doesn't get this, so they think continually throwing the development into "crunch mode" will actually net them more results; they'll somehow get a better return on their salaries.

Of course, that directly leads to the second reason. The continual whipping to extract more from the coders will always result in a degenerating code base. If the initial version of the system wasn't too bad, but the next version is expected to be a total bugfest, then it is a strong indication of a sweatshop. When 'mediocre' is the best possible outcome, that turns the morale of many of the programmers. My sense has been that in order to prevent the programmers from realizing this, you have to keep them super busy. If they're just too tired to think at the end of the day, then they are less likely to reach the conclusion that their jobs suck. That they are just churning code.

Thus getting caught up in a sweatshop is a somewhat self-perpetuating cycle. They work you hard enough that you are too tired to escape, so you just hang in there on a day-to-day basis until way too much time has passed or you melt down. If you're really screwed, you don't even have any relevant market experience that would make you attractive to other employers. With no visible options, you'd have to take a serious risk in order to leave.

So, of course, as new grads, we really wanted to avoid this horrific fate. If you want a long career in building modern software, we quickly realized that you had to keep finding good positions that didn't burn you out but also didn't lock you into some unsellable skill.

Decades ago, there were lots of good development shops that were focused on building up reliable systems at a reasonable pace. That, of course, all changed with the dot com era. All of the sudden programmers became scarce, and programming shifted from deep intellectual construction to just an endless panic. Tech has always suffered from stupid fads, but suddenly the fads themselves became the bleeding edge, and the cost to play was that you had to work for a sweatshop.

Not surprisingly, the software being produced dropped heavily in quality. There are always counterexamples, but collectively the mentality shifted away from taking enough time to build something meaningful, to just getting anything out there and then trying to make it look pretty later so consumers don't notice that it's broken.

It certainly hasn't helped that the methodologies drifted into being hyper-reactive towards the "stakeholders". The idea being to cover one's ass when building a mess by rigorously ensuring that the blame can be placed on the clients. If we did "exactly" and "only" what they wanted (even if they don't understand the consequences) then how can that be wrong? To me, that always seemed like a Doctor asking patients to self diagnose their own illness, so that they can't be blamed later if it went wrong. The clients aren't professional Software Developers, so why would they know how to build a working system in a reasonable time? They don't even have a realistic sense of what 'working' and 'reasonable' means. It's a classic recipe for disaster.

Lately, I've been watching the job market again and I am saddened by the number of sweatshops out there. Even worse is that many of them actually brag about it in their job descriptions. Job ads that a few decades back would have resulted in 0 serious resumes are suddenly extremely popular. Programmers seem attracted to these like moths to a bonfire. Companies are so over the top that they add perks like "frequently paying for dinner" as if they were actual benefits, but to me, it seems to be enshrining a rather sick work/life balance. If their priorities are that skewed, the work is probably pretty shoddy as well. I love to work, but not enough that I don't feel bad about missing out on the rest of what life can offer. In those extreme periods in the past where my years just disappeared in an instant, I’ve looked back with disappointment. Coding is fun, but so are a lot of other things.

It seems like we're in an epidemic of the sort of career-destroying culture that as a new grad I was trying so hard to avoid. Programming shouldn't be a continual high-stress hackfest. There are always times in any system's life where it becomes difficult and stressful, but the measure of a good job is that those times are rare, not constant. If the development is thought through and the process is organized, then most days in programming are spent carefully working on building up reliable code. You need to spend the time to make things of decent quality, they don't just get created by accident. When the work is crazy, you might feel proud that you survived, but you can never really feel proud about creating that inevitable ball of duct tape and bandages. It is just ugly. Myself, I am driven to build things, software is just the medium I choose to work in. If I am going to spend a lot time building, then I need to be proud of my creations.

High-stress, of course, leads to rather poor thinking, and code is only ever as good as the thinking that went into it. Computers are "multipliers", thus bad code realistically just makes the problems worse. You can hide that, but only for a while. So it's been clear for a long time now that even if the code started out okay, throwing stressed out teams at extending it is detrimental in the long run. Setting up an environment that glorifies stressed out teams is unlikely to be sustainable (with a few notable exceptions). If you care about what you create, then you have to find an environment that does too.

In many ways, it has been extremely frustrating to watch this continuing madness grow, even as we are becoming dangerously reliant on software. It's like watching a car crash in slow motion. The pools of good jobs are gradually drying up, getting replaced by sweatshops. Each year more people proclaim crazier and crazier approaches to building software that pivot on exploiting some obviously bad shortcut or a hopelessly redundant time wastage. Eventually, enough people catch on that the latest fad isn't helping, but by then something even worse has come along to replace it. We're so trapped in bad approaches right now, so deeply, that many newbies think they are actually the core of our profession.

The truth is that at some point, with enough variation, you start to realize that most of the industry's persistent problems are self-inflicted. And by the time you finally get to this realization, you're heavily wishing that you had pursued an alternate career. Like forestry.

Encapsulation

One of the strongest, but possibly least understood principles of object-oriented (OO) programming is 'encapsulation'.

The OO paradigm explicitly injects structure on top of code which allows programmers to build and maintain considerably larger programs than in the distant past. This extra level of organization is the key to managing complexity. But while this amplifies our abilities to build big programs, there is still a 'threshold of complexity' that once crossed will quickly start to degrade the overall stability of the development project, and eventually the software itself.

An individual programmer has fixed limits on how quickly they can build up instructions and later on how quickly they can correct problems. A highly-effective team can support and extend a much larger code base than the sum of its individuals, but eventually the complexity will grow beyond their abilities. There is always some physical maximum after which the work becomes excessively error prone or consistently slower or both. There is no getting around complexity, it is a fundamental limitation on scale.

However we can significantly minimize it, to prevent crossing the threshold for as long as possible. This obviously comes from the strict avoidance of adding any artificial complexity, such as special cases, twisted logic, arbitrary categorizations and other forms of disorganization. That helps, but there is another approach as well.

Encapsulation can be seen as drawing a 'black box' around a subset of a complex system. That box prevents anyone from outside in seeing the inner workings, but it also ensures that what's on the inside is not influenced by random outside behaviour. The inside and outside are explicitly walled off from each other, so that the only interaction is a precisely defined 'interface'.

To get the most out of encapsulation, the contents of the box must do something significantly more than just trivially implement an interface. That is, boxing off something simple is essentially negative, given that the box itself is a bump in complexity. To actually reduce the overall complexity, enough sub-complexity must be hidden away to make the box itself worth the effort.

For example, one could write a new layer on top of a technology like sockets and call it something like 'connections', but unless this new layer really encapsulates enough underlying complexity, like implementing a communications protocol and a data transfer format, then it has hurt rather than helped. It is 'shallow'. What this means is that for any useful encapsulation, it must hide a significant amount of complexity, thus there should be plenty of code and data buried inside of the box that is no longer necessary to know outside of it. It should not leak out any of this knowledge. So a connection that seamlessly synchronizes data between two parties (how? We don't know) correctly removes a chunk of knowledge out of the upper levels of the system. And it does it in a way that it is clear and easy to triage problems as being 'in' or 'out' of the box.

Once a subset of the complexity has been fully encapsulated, and is easily diagnosable, it can be vetted and ignored for the moment. That is, if the connections library is known to work for all current circumstances used by the project, then programmers don't have to revisit the internals, unless they want to expand them. That's the real power of encapsulation. For a little bit of extra thinking and work, some part of the system is carved off and put to rest. Later, because of reuse or enhancements, it may make sense to revisit the code and widen its functionality, but for the moment it is one less (hopefully large) problem to deal with. The system is more complex, but the system minus the encapsulated parts is relatively less complex, and thus it is easier to work on what remains unsolved.

In little programs, encapsulation isn't really necessary, it might help but there just isn't enough overall complexity to worry about. Once the system grows however, it approaches the threshold really fast. Fast enough that many software developers ignore it until it is way too late, and then the costs of correcting the code becomes unmanageable.

It is for that reason that many seasoned developers have learned the habit of encapsulating early and often. They essentially have a second 'editing pass' on any new code that is aimed at breaking off any potentially encapsulated parts into their own independent chunks. This is essentially 'partial' encapsulation. The code is partitioned, but not rigorously enforced. Doing this regularly and reliably means that at some point later, when it is required, the code can easily be enhanced to full encapsulation.

If you look at any well-written code base, the related instructions tend to be localized, and there is at least an implicit organization that cleanly separates the underlying pieces. That is, the individual lines of code appear in the sub-parts of the system, exactly where you would expect them to be located. Perhaps, some of the code is formally encapsulated, but often that degree of rigor is not necessary at the current stage of the code base, so its just partial. Contrast that with code containing an over abundance of hollow encapsulation or just what seems to be code randomly located anywhere and you can see why this principle is so important in keeping the code useful.

To build big systems, you need to build up a huge and extremely complex code base. To keep it manageable you need to keep it heavily organized, but you also need to carve out chunks of it that have been done and dusted for the moment, so that you can focus your current efforts on moving the work forward. There are no short-cuts available in software development that won't harm a project, just a lot of very careful, dedicated, disciplined work that when done correctly, helps towards continuing the active lifespan of the code.