Sweatshops

Decades ago, when I got out of school, I remember being very wary of accidentally working for a software sweatshop. 

It was a frequent conversation between new grads. All sorts of industries have variations on this theme but in programming, sweatshops had already started to become significant career hazards.

What we wanted to avoid in those days were jobs with crazy hours, hacking at mediocre code while under humiliating micromanagement. The sort of work that makes you wish you had pursued an alternate career. Maybe forestry.

It's not that I minded long hours when I was young, but given that I am essentially donating my personal time to a company, in exchange for this I expected to work on meaty problems that would help my career. Most code is fairly routine, and with plenty of practice you can shut off your brain and just pound out stuff that kinda works. The jobs that I have loved over the years have been the ones where we went deeper than that. Where we didn't just 'barely' get things working, rather we carefully crafted sophisticated solutions that solved real problems for people. Partially solving some aspect of a problem is easy, refining that to work flawlessly is exponentially harder, but it's those deep solutions that people really need; that truly make their lives easier.

Of course, the type of code I wanted to write wasn't just chucking some data onto the screen and then saving it a few minutes later. A sloppy GUI to support editing a crappy data model is just automating something already broken. Deep solutions require complex modeling so that they can add value on top of the basic data collection process. Really big, but low-quality data isn't going to be mined for any real insights, there is just too much noise embedded within it. Good user interfaces also require deep insights in order to properly structure the workflow. Littering the screen with tables, dropdowns and buttons is easy for the programmers, but horrible for everyone else.

Besides the long hours and the weak solutions, sweatshops also tended towards hovering over their employees, while trying not to let them know much about the higher-up planning. My guess is that this is done for two reasons.

The first is to keep squeezing as much out of everyone as possible. That's sort of an odd irony since programming is mostly intellectual work; the closer programmers get to burnout, the worse the quality of their efforts. But a really intense sweatshop doesn't get this, so they think continually throwing the development into "crunch mode" will actually net them more results; they'll somehow get a better return on their salaries.

Of course, that directly leads to the second reason. The continual whipping to extract more from the coders will always result in a degenerating code base. If the initial version of the system wasn't too bad, but the next version is expected to be a total bugfest, then it is a strong indication of a sweatshop. When 'mediocre' is the best possible outcome, that turns the morale of many of the programmers. My sense has been that in order to prevent the programmers from realizing this, you have to keep them super busy. If they're just too tired to think at the end of the day, then they are less likely to reach the conclusion that their jobs suck. That they are just churning code.

Thus getting caught up in a sweatshop is a somewhat self-perpetuating cycle. They work you hard enough that you are too tired to escape, so you just hang in there on a day-to-day basis until way too much time has passed or you melt down. If you're really screwed, you don't even have any relevant market experience that would make you attractive to other employers. With no visible options, you'd have to take a serious risk in order to leave.

So, of course, as new grads, we really wanted to avoid this horrific fate. If you want a long career in building modern software, we quickly realized that you had to keep finding good positions that didn't burn you out but also didn't lock you into some unsellable skill.

Decades ago, there were lots of good development shops that were focused on building up reliable systems at a reasonable pace. That, of course, all changed with the dot com era. All of the sudden programmers became scarce, and programming shifted from deep intellectual construction to just an endless panic. Tech has always suffered from stupid fads, but suddenly the fads themselves became the bleeding edge, and the cost to play was that you had to work for a sweatshop.

Not surprisingly, the software being produced dropped heavily in quality. There are always counterexamples, but collectively the mentality shifted away from taking enough time to build something meaningful, to just getting anything out there and then trying to make it look pretty later so consumers don't notice that it's broken.

It certainly hasn't helped that the methodologies drifted into being hyper-reactive towards the "stakeholders". The idea being to cover one's ass when building a mess by rigorously ensuring that the blame can be placed on the clients. If we did "exactly" and "only" what they wanted (even if they don't understand the consequences) then how can that be wrong? To me, that always seemed like a Doctor asking patients to self diagnose their own illness, so that they can't be blamed later if it went wrong. The clients aren't professional Software Developers, so why would they know how to build a working system in a reasonable time? They don't even have a realistic sense of what 'working' and 'reasonable' means. It's a classic recipe for disaster.

Lately, I've been watching the job market again and I am saddened by the number of sweatshops out there. Even worse is that many of them actually brag about it in their job descriptions. Job ads that a few decades back would have resulted in 0 serious resumes are suddenly extremely popular. Programmers seem attracted to these like moths to a bonfire. Companies are so over the top that they add perks like "frequently paying for dinner" as if they were actual benefits, but to me, it seems to be enshrining a rather sick work/life balance. If their priorities are that skewed, the work is probably pretty shoddy as well. I love to work, but not enough that I don't feel bad about missing out on the rest of what life can offer. In those extreme periods in the past where my years just disappeared in an instant, I’ve looked back with disappointment. Coding is fun, but so are a lot of other things.

It seems like we're in an epidemic of the sort of career-destroying culture that as a new grad I was trying so hard to avoid. Programming shouldn't be a continual high-stress hackfest. There are always times in any system's life where it becomes difficult and stressful, but the measure of a good job is that those times are rare, not constant. If the development is thought through and the process is organized, then most days in programming are spent carefully working on building up reliable code. You need to spend the time to make things of decent quality, they don't just get created by accident. When the work is crazy, you might feel proud that you survived, but you can never really feel proud about creating that inevitable ball of duct tape and bandages. It is just ugly. Myself, I am driven to build things, software is just the medium I choose to work in. If I am going to spend a lot time building, then I need to be proud of my creations.

High-stress, of course, leads to rather poor thinking, and code is only ever as good as the thinking that went into it. Computers are "multipliers", thus bad code realistically just makes the problems worse. You can hide that, but only for a while. So it's been clear for a long time now that even if the code started out okay, throwing stressed out teams at extending it is detrimental in the long run. Setting up an environment that glorifies stressed out teams is unlikely to be sustainable (with a few notable exceptions). If you care about what you create, then you have to find an environment that does too.

In many ways, it has been extremely frustrating to watch this continuing madness grow, even as we are becoming dangerously reliant on software. It's like watching a car crash in slow motion. The pools of good jobs are gradually drying up, getting replaced by sweatshops. Each year more people proclaim crazier and crazier approaches to building software that pivot on exploiting some obviously bad shortcut or a hopelessly redundant time wastage. Eventually, enough people catch on that the latest fad isn't helping, but by then something even worse has come along to replace it. We're so trapped in bad approaches right now, so deeply, that many newbies think they are actually the core of our profession.

The truth is that at some point, with enough variation, you start to realize that most of the industry's persistent problems are self-inflicted. And by the time you finally get to this realization, you're heavily wishing that you had pursued an alternate career. Like forestry.

Encapsulation

One of the strongest, but possibly least understood principles of object-oriented (OO) programming is 'encapsulation'.

The OO paradigm explicitly injects structure on top of code which allows programmers to build and maintain considerably larger programs than in the distant past. This extra level of organization is the key to managing complexity. But while this amplifies our abilities to build big programs, there is still a 'threshold of complexity' that once crossed will quickly start to degrade the overall stability of the development project, and eventually the software itself.

An individual programmer has fixed limits on how quickly they can build up instructions and later on how quickly they can correct problems. A highly-effective team can support and extend a much larger code base than the sum of its individuals, but eventually the complexity will grow beyond their abilities. There is always some physical maximum after which the work becomes excessively error prone or consistently slower or both. There is no getting around complexity, it is a fundamental limitation on scale.

However we can significantly minimize it, to prevent crossing the threshold for as long as possible. This obviously comes from the strict avoidance of adding any artificial complexity, such as special cases, twisted logic, arbitrary categorizations and other forms of disorganization. That helps, but there is another approach as well.

Encapsulation can be seen as drawing a 'black box' around a subset of a complex system. That box prevents anyone from outside in seeing the inner workings, but it also ensures that what's on the inside is not influenced by random outside behaviour. The inside and outside are explicitly walled off from each other, so that the only interaction is a precisely defined 'interface'.

To get the most out of encapsulation, the contents of the box must do something significantly more than just trivially implement an interface. That is, boxing off something simple is essentially negative, given that the box itself is a bump in complexity. To actually reduce the overall complexity, enough sub-complexity must be hidden away to make the box itself worth the effort.

For example, one could write a new layer on top of a technology like sockets and call it something like 'connections', but unless this new layer really encapsulates enough underlying complexity, like implementing a communications protocol and a data transfer format, then it has hurt rather than helped. It is 'shallow'. What this means is that for any useful encapsulation, it must hide a significant amount of complexity, thus there should be plenty of code and data buried inside of the box that is no longer necessary to know outside of it. It should not leak out any of this knowledge. So a connection that seamlessly synchronizes data between two parties (how? We don't know) correctly removes a chunk of knowledge out of the upper levels of the system. And it does it in a way that it is clear and easy to triage problems as being 'in' or 'out' of the box.

Once a subset of the complexity has been fully encapsulated, and is easily diagnosable, it can be vetted and ignored for the moment. That is, if the connections library is known to work for all current circumstances used by the project, then programmers don't have to revisit the internals, unless they want to expand them. That's the real power of encapsulation. For a little bit of extra thinking and work, some part of the system is carved off and put to rest. Later, because of reuse or enhancements, it may make sense to revisit the code and widen its functionality, but for the moment it is one less (hopefully large) problem to deal with. The system is more complex, but the system minus the encapsulated parts is relatively less complex, and thus it is easier to work on what remains unsolved.

In little programs, encapsulation isn't really necessary, it might help but there just isn't enough overall complexity to worry about. Once the system grows however, it approaches the threshold really fast. Fast enough that many software developers ignore it until it is way too late, and then the costs of correcting the code becomes unmanageable.

It is for that reason that many seasoned developers have learned the habit of encapsulating early and often. They essentially have a second 'editing pass' on any new code that is aimed at breaking off any potentially encapsulated parts into their own independent chunks. This is essentially 'partial' encapsulation. The code is partitioned, but not rigorously enforced. Doing this regularly and reliably means that at some point later, when it is required, the code can easily be enhanced to full encapsulation.

If you look at any well-written code base, the related instructions tend to be localized, and there is at least an implicit organization that cleanly separates the underlying pieces. That is, the individual lines of code appear in the sub-parts of the system, exactly where you would expect them to be located. Perhaps, some of the code is formally encapsulated, but often that degree of rigor is not necessary at the current stage of the code base, so its just partial. Contrast that with code containing an over abundance of hollow encapsulation or just what seems to be code randomly located anywhere and you can see why this principle is so important in keeping the code useful.

To build big systems, you need to build up a huge and extremely complex code base. To keep it manageable you need to keep it heavily organized, but you also need to carve out chunks of it that have been done and dusted for the moment, so that you can focus your current efforts on moving the work forward. There are no short-cuts available in software development that won't harm a project, just a lot of very careful, dedicated, disciplined work that when done correctly, helps towards continuing the active lifespan of the code.

Get Serious

It is time we get serious about building software systems.

Small programs used for entertainment can be hastily written, but they do not scale up for industrial usage. Large organizations need massive amounts of computation that they can rely upon, both to keep up with the ever-changing times but also to master the dragons of complexity that we have unleashed.

So far, in software development, all we've been doing is idly tossing code at a sea of independent silos, while the more questionable people in our industry keep making fabulous claims about how their stuff, this time, will really fix everybody else's problems. The thing is, that each time we dump in more disjoint technology we're just making our problems worse, not better. Another silo is not a positive accomplishment.

It would be really nice if there was some new "magical" technology on the horizon which would rectify the mistakes of the past, but so far I haven't seen anything that is substantially different from our earlier offerings. If we don't change the focus we'll just perpetually re-invent the same whole house of cards that is already collapsing under the weight of our dysfunction.

To hopefully get us out of this self-imposed 'vicious cycle' we need a reasonable set of goals. This would allow us to make better choices along the way and give us some tangible indication of progress. With that in mind, I am suggesting seven basic requirements, or constraints, all driven from outside the technologies, that if met, would better position any new solutions to avoid repeating the past insanities.

Seven constraints for better software systems:

1. There is one and only one system per organization.

2. All of the data is in one logical data persistence location, and all of it is accessible from one general interface. It has global ACID properties as a single system (even if it is distributed across a massive number of machines).

3. There is one all covering general user interface, for everything. There can also be specialized interfaces for particular roles, but all of that functionality is available from the general interface.

4. Data and code partitioning is malleable, on-the-fly (access, security, visibility of both data and code).

5. Functionality is loosely bound to all of the user interfaces, all versions of all functionality are always available. Users can choose which version they want (prefer).

6. Almost everything works at near real time (fast enough to be usable and interactive). For any really slow computations, there is one simple mechanism to schedule work to be completed and users notified. The user can decide to have immediate or delayed access to any and all functionality.

7. Data and code organization is completely handled by the system itself, not by the users. They can name things in any way they want, but the things themselves are organized by the computer and thus are always up-to-date.

Each one of these constraints fits together to encapsulate behaviours that address well-known problems.

Having only one system per organization ensures that the data and functionality aren't siloed. It is potentially usable by all. Organizations are fluid arrangements, so this also means that you can easily split and merge systems. 'Easily' means that it doesn't take years or months. There is only one operations dept, that can get the full range of experience required from dealing with just one system. Expertise can be built up, not thinly spread across dozens of incompatible systems.

There must be one overall, consistent, fast and easy way to access absolutely every piece of data in the system through one programmatic interface (including logging and errors). Underneath, the data is likely distributed, shared and partitioned, but those physical attributes are encapsulated away. Thus at the data level there are no silos. This type of persistence solution, as will become apparent in later constraints, may share the same concepts of some of the modern technologies, but it won't really come into existence until we address it specifically. It will have ACID properties. It will be dependable.

Absolutely every single piece of new or old functionality should be accessible, fairly easily, from one complete overall general user interface. A general interface however would never be optimal for highly specialized users with deeper or more complex needs, so there would also be several more specific interfaces tuned to specific users, groups or roles. The overall interface insures that at the very least, all of the code and all of the data is always accessible to the application support roles, without having to do specific training on specialized interfaces. It might be slower to navigate around, but the consistency and accompanying organization would mean that someone, with the assistance of some domain experience, could reasonably view data, edit data or run some computation. This insures that shifts in the organization, and quirks of the specialized interfaces don't represent risks, but rather that administration might be slower at times.

Organizations change, code tends to be static. That causes an unstoppable amount of rust. Security, sub-groups and any number of access issues crop up. A system should be easily reconfigured in far less than a day, so that it can keep pace with the organization's structure. Re-orgs should not be hampered by technology; they may be unpleasant, but form an important part of preventing stagnation. The system needs to deal with what happens in real life, not just take a convenient way around it.

The programmers should be consistently building functionality, not playing hurry up and wait with the users. Some users will be early adopters, many won't. As such the users themselves should have some control on how they move forward utilizing any new functionality, which will continue to grow along with the organization (at a near exponential rate sometimes). Once tested the functionality should be available, but the older functionality must also continue to work correctly. This is a thorny problem, but it can be handled by decoupling the structure of data away from the data itself, in the same manner that we already decoupled the indexing of data away. In that sense, to make this works demands a primary structure and many secondary ones, all of which are usable by the code. The data should be restructurable on the fly, so that an early version of some functionality that depends on a given structure works whether it is primary or secondary. Also for any set of related data, some it can have one primary structure, while some of it can have an older one. This makes it easy to schedule batch processes to slowly convert the database over time, rather than having to commit to large downtimes. Given that the underpinnings can naturally support many versions, the users should have the ability to explicitly control which versions are bound to their interfaces. That is, one user might set their functionA to version 1.2 while another sets it to 3.4, and both should be able to access and add in similar data at the same time. At the same time the programmers could have moved on to implementing version 5.1. With that achieved the system really does decouple the differences in what the users are doing, from the software construction process.

Processing speed is important, but it is also important for the users to maintain control over their expected results. If some computation is slow, scheduling it to be completed in the background should be trivial and convenient. Any sort of uncertainties in time need to be removed to allow for the users to make better plans with regard to the computations. They need to be informed, kept abreast of the progress. We know that no matter how fast the hardware gets, there will always be slow computations, but so far we've been ignoring this.

The last constraint is one that I've been pondering for a while now. Allowing many users to organize assets gradually over time always seems to result in significant disorganization. Computers as a tool are most valuable when they automatically organize stuff for us. It is an incredibly complex issue, but I think it can be tackled. The idea would be similar to my earlier library example from my post in January called "Organization". As the sizes of the assets, code and data, get larger, the computer would automatically add in newer navigational indexing schemes, allowing the users to meaningfully name them, but not to create new arbitrary half-completed clumps of their own. Mostly this means sets, groups, hierarchies and indexing, on any number of dimensions with some limited combination of subsets. As the underlying size grows, the system would introduce new forms of organization, insuring that all of them are always 'complete' at all times.

The underlying essence of any trustable, effective system is a single well-defined architecture spanning the full organization that allows the data, its structure, and any necessary code to keep pace with the growth and changes that are natural. This constrained flexibility is necessary for any organization that wants to fully utilize the data that it has been collecting. We have most elements for this type of system buried in our technologies, but to my knowledge nothing this comprehensive actually exists, although many a software developer has dreamed of it late at night. There is nothing in this description that is known to be impossible, although some constraints directly confront difficult problems that we have been avoiding so far. It is possible to build something that matches these constraints, but it is also possible that we won't achieve this sort of system for many decades to come; both the software industry and our programming culture have other agendas. Still, modern organizations are crying out to stop the insanity and there has been a long history of people promoting non-existent solutions to these problems. The demand is there, the seeds of the technology are also there, we just haven't oriented ourselves in the right direction yet.