Duality

There are two very similar ways of looking at software systems. 

The most common one is to see it as a lot of code that moves data around. Code is the primary concept. Its dual, however, is that you see it as data, with the code just assisting in getting it from place to place. Code is a secondary issue.

They may seem similar, and it's easy to miss the difference between the two, but from a higher level the second perspective is a lot simpler and a lot more powerful.

When we first learn to program, we are taught to start assembling larger and larger code fragments. First, it is small examples of branches and loops, putting some code into functions, calling other chunks of code to get them to do stuff. That, rather directly, establishes that ‘the code’ is super important. We get caught up in syntax issues, different languages, and various IDEs. Early coding tasks are to ‘write some code that ...’, interviews are uber-focused on the code too. Everything is about ‘code’.

Data is usually introduced as a secondary issue, but most often it is somewhat trivial and primitive. If we take a data structures course, the actual data is even abstracted away, we’re just left with abstract relationships like lists and trees.

That carries on through most programmers' careers. Their work assignments are often crafted as producing some code to support a feature. In many shops, the code is started first, then later they realize that there were some details about the data that were missed. If there is a bug to be fixed, it is because some code is missing or calculating the wrong values.

So its code, code, code, all of the way down. Data Is an afterthought.

The point of taking a data structures course is lost on most people. Sure, the underlying data is abstracted away but it's not because it doesn’t matter. It's the exact opposite. Data structures are a pretty complete means of decomposition. That is, you can take most large and complex programs and rephrase them as a set of data structures. Most programs are just a bunch of primitive operations happening on a fairly small set of structures like lists, trees, stacks, queues, etc. If those structural operations are pulled out and reused, the resulting code is way smaller, and intrinsically has less bugs. That’s why Donald Knuth collected them all together in the Art of Programming, that is why they keep getting taught in classes. They are the ‘power tools’ of programming, but to get them to work you have to flip your perspective on the system.

Data structures aren’t the only opposite approach. The general ideas around them got formalized and explicitly wired into languages as Object-Oriented programming. In non-OO languages, the programmers had to set up the structures themselves and keep any primitives nearby. With Objects that become a part of the language syntax. Objects bind code directly to the data not because it is fun, but so that objects that are structurally similar can be encapsulated and reused and they can point, polymorphically to other types. It is exactly the same as basic data structures, just formalized into the language.

It’s also why people writing a lot of code in OO that is focused on doing super long lists of instructions end up getting into such a mess with Object-Oriented languages. Retrofitting a brute force procedural style into objects is basically going against the grain of the language. Objects as mega-instructions clash with other such objects, which prevent reuse, are hard to understand collectively and are prone to integration bugs. It makes the code awkward, which keeps getting worse as the system gets larger.

While data structures are very powerful, they are just the tip of the iceberg and came of age in an era when data was really rare and hard to get. That all changed when computers became ubiquitous and networked. Now data is plentiful, everywhere, and often of extremely poor quality. Data was always important, but it's getting more important as we collect a lot more of it and want to draw wisdom out of what we have.

So the alternative perspective is to see a system, just by its data, and how that data flows around. 

For data to be valuable, it has to stay around for a long time, which happens when we persist it into a database. In all ways, the schema defines the possible scope and capabilities of a system. If you haven’t persisted the data, it isn’t available for usage. If you saved some of it, in an awkward format, that will percolate upwards through all of the other functionality you use it for. If the data is available, guaranteed to be correct, then most of the features that require it are fairly simple. If its hard to write some code, its often because the incoming data is a mess.

All data starts somewhere in the real world. That may seem like a controversial statement since data like log files originate in response only to changes in the digital realm, but if you accept that those come from the behavior of the underlying hardware then it makes more sense. Besides operational data, the rest of it is entered from end-users, administrators, programmers, or third-party organizations. It starts in the real world, and really only has value in the real world. On top of this raw information, we can derive other useful relationships but it all has to start somewhere.

What becomes important then is how the data flows from one location to another. For example, it may have started as an observation by a person. They used some interface to get it into persistence. Later it might be pulled from storage and used to augment some other data. That new data is shown in another interface or packaged together and sent to a remote location. Maybe in exchange, more data flows into the system from that remote site. 

If you just look at the data and ignore the code, threads, processes, etc. most systems are not particularly complex. They act as a hub to collect and distribute data to a whole bunch of different sources, people, or other machines. 

What’s needed then, to build a system that manages that data is many fragments of code that can move, decorate, or translate the data as it circulates around. If you minimize the fiddling done with that code as it travels around, you’ve optimized a large portion of the system without even realizing it. The closer those data formats are in the widgets, middleware, and persistence, the less work that is needed when it is in motion. Moving data is always expensive.

That perspective puts data first. It is more important than the code, and the idea is to minimize what the code is doing to it whenever possible. There still might be memoization or other macro optimizations that are possible, but those can be seen as refinements.

What becomes important then is keeping dependent pieces of data together and managing the structural relationships between these ‘entities’. This brings us right back to data-structures. They deal with the relationships between these larger chunks and can be built to have indirect references to the underlying entities. Why? Because the structural relationships are smaller and more common. If you get them perfected for one type of entity, you can reuse them for all types of entities. That then just shifts the problem down to picking a good set of entities and relationships that best fit the data as it originated in the real world, or basically ‘modeling’.

Now the data perspective doesn’t magically eliminate performance problems. Systems aren’t intrinsically scalable, handling huge loads has to be explicitly engineered into the system in order for it to work correctly. But seeing the system as flowing data does make it a whole lot easier to lay out an industrial scale architecture. 

Take caching for example. From a code perspective, programmers often just allocate a chunk of memory, set it up as a hash table, then do some type of lookup first to get the value. That seems to make caching easy, but eventually, the real problems show their ugly heads. If you see caching as a form of memoization where we keep a smaller pool of data closer to the action, then what is obvious is how to decide what’s in that pool and what's not in it. Caching stale data is bad, and also letting the cache grow to be the same size as the persistence is rather pointless. But on top of that, if the data may be in two places at the same time, what happens when you need to update it? The code perspective of caching forces programmers to think about memory, while the data perspective forces them to think about the quality of the data. It makes it easier to see the whole picture, which makes it easier to get the implementation correct. Once you realize that removing stuff from the cache and keeping it in sync during writes are the real problems, figuring out what code and architecture are needed to make this happen is a whole lot easier.

The same is true for putting queues between systems to resolve speed differences and with syncing external information for read-only internal copies. Pretty much all of the big enterprise issues get a whole lot less challenging. The difficult problems shift away from some tangled mess of code, to really keeping the whole thing organized and operating correctly. 

It also applies to the really big systems as well. It’s hard to figure out how to massively parallelize systems until you realize that it is just an issue about dependencies. Where data is independent, it can be safely split across threads, processes, or machines. If it's not independent, then there will be problems. So, instead of being a complex coding problem, it is really a complex data modeling one. What underlying model do we need that both approximates the real-world issues, but also guarantees some usable independences that can be parallelized? If you structure the data correctly, the rest is just spending the time to write out the code.

As an industry, we usually start by teaching new programmers to code. That’s okay, but we often fail to teach them how to flip to this dual perspective. Instead, we leave them dangling in the wind, trying to crank out code from what is clearly a much harder perspective. Sure, some people get data structure courses or even computer theory, but then they go out into the industry and none of that sticks. To make it worse, people tag it to ‘abstraction’ and ‘overcomplexity’ and keep trying to insist that writing more overly simple code, faster, is somehow going to make it better. It even percolated back into our bad interview styles. 

It permeates everything, making most serious programming tasks a whole lot harder than they need to be. If you’ve ever suspected that there was an ‘easier’ way to build systems, then you were right. If you forget about the code, and focus on the data, getting that set up correctly, then most of the coding is easy and straightforward.

Knowledge Bases

To me ‘intelligence’ is basically raw human thinking power. It’s our ability to work through problems. In order to employ intelligence effectively in our world, we need data. And that data needs to be structured and interconnected to make it usable to us. Usable, organized data is what I take to be ‘knowledge’. Its all ready to be utilized.

Our various endeavors are categorized into a huge number of fields (or domains) such as law, medicine, finance, physics, math, biology, etc. each of which is mostly an independent ‘base’ of knowledge about the field. What’s really interesting about most knowledge bases is that they are easy to over-simplify and also frequently counter-intuitive in their depths. An outside perspective can easily lead to the wrong conclusions. It’s not until you are steeped in the details, that you’re able to apply your understanding correctly.

Without enough underlying knowledge, intelligence is just hand waving. You can think about something as deep and as long as you want, but your conclusions are unlikely to be reasonable.

Software development is extremely complex. Not only do you need a strong understanding of the underlying technologies and how to use them, but it also cuts across many other knowledge bases.

A software system is a solution to a problem in a user domain. If it doesn’t solve their problems, then it is only adding to them. It consists of a huge number of details, all of which need to be organized and fit into the final work. Managing these pieces, usually under tight time-lines requires a special set of skills. Software is also very expensive to build and maintain, so finding the money to pay for enough resources is always tricky.

Putting this together we see that software spans the following knowledge bases:

• Technology
• User domains (all?)
• Management
• Business

Most programmers tend towards believing that technology is the most significant issue in software development, but usually the problems start in other areas and feed back into the development work. For this post, I’ll go through all four areas in the order that I find have the most impact for big projects.

Management

A large software project has millions of details that are all being juggled at the same time. Keeping most of those balls in the air, without losing track of them, is necessary to deliver a working system.

A good manager is constantly running around, making sure that all of their resources are moving forward, and that any roadblocks are eliminated as swiftly as possible. Some people see software management as a higher creative input role where they just have to inject ideas, but that’s never actually helpful. Most development projects have more than enough ideas, what they need is organization, process and to keep everyone on the same page. In that sense a good manager accepts their role as herding all of the cats in the same direction, while making sure the path forward is clear.

Development meetings are a good place to assess management. If they are retrospective and basically just a forum for people to catch up with what has already happened, then the project is not well-organized. It shouldn’t be necessary for the manager to catch up, if they have been paying attention as they should.

If, however, meetings are really planning sessions for the next upcoming work, then they contribute by identifying the issues long before they become significant enough to derail the process. Plans don’t always work as expected, but planning is a necessity when you are dealing with long running work. Even small tasks in development can take weeks or months, and the worst possible outcome is to keep shifting directions before any of the work is completed properly. What’s started should be finished, or it shouldn’t have been started in the first place. Only a long-term plan will avoid time-crushing ‘make-work’.

Since development flows through different phases, the management of each has to change as well. A basic development iteration has five parts:

• Analysis
• Design
• Implementation
• Testing
• Distribution

Analysis for example is a never-ending effort about depth. Wherever you skimp on analyzing the requirements, it will come back to haunt you as ‘scope creep’. However it would be rare to actually have enough time and patience to fully analyze everything before the work starts. Analysis is also very sensitive to the approach; ask the right questions and you get the right answers. Ask the wrong ones and you get misleading information. Managing analysis means ensuring that both the right questions are being asked, and that the answers are being organized. Experienced analysts need little intervention, but the quality of the work isn’t known until later in the implementation phase, when it is often too late.

Design and Implementation are really about making sure all of the programmers are on the same page; that they are working together well and that they have all of the input necessary to keep them moving forward. For large projects this is basically about team building. A system built by a dysfunctional group is naturally a big ball of mud, and the messiness of the work creates a huge of amount of extra make-work to keep it all from falling apart. A good manager needs to keep the team working, arbitrate disputes and enforce the process and standards even when the time pressure is intense. They also need to protect the programmers from any outside interference to insure that they have enough calmness to be able to think clearly and deeply about their work.

Testing is all about quality vs. time trade-offs. It is impossible to test to 100%, so there is some lower percentage of testing that has to be accepted. Often that might even be just a fixed block of time. The biggest problem during testing is to keep everyone working. Testing is often seen as boring -- particularly after an intense development slog -- and the developers start to lose their attention to the details. Bugs get noticed, but then forgotten. Managing here means dragging people around, holding hands, calming nerves and just trying to get the process as constructive as possible. It can also means having to make brutal choices about quality vs. timeliness. Sometimes the software has to be shipped, even if it is not as good as it should have been.

Distribution of software usually involves an operations dept., or selling the system. Because it is usually forgotten until the end, the distribution is rarely well-planned. Management here needs to insure that the software is supported well enough, but not at the cost of disabling future development work. The handling of feedback and bug reports can all be set up in advance, and most projects require both a standard release and an emergency fast-track process. Most issues take some time, but some need an ‘all hands on deck’ approach in order to deal with them before the situation escalates.

Business

A large software project takes many man-years to build and for most projects the work is never done. It just goes on, year after year. Hiring programmers isn’t cheap and you need a lot of other support staff in the project as well including: project manager, system admin, specialists, etc. Commercial quality systems also need roles like: graphic designer, UX expert, editor, translator, etc.

All together, to build something large that isn’t a weird eclectic mishmash of disorganized functionality means having to shell out a huge amount of money. For software, money is time. That is, if you have enough money, you can hire the necessary resources and experience to get the work done, given some reasonable time frame. A lack of money, means that you have to do more with less, and often times it’s that time pressure that leads people to take ill-advised shortcuts, which will eventually scramble the project so badly it can’t be saved.

As such, setting up a development project always means having to figure out where the money is going to come from. And as is always the case with money, what strings are attached to it. Nothing comes for free.

The business world is very subtle, complex and constantly changing. Most techies who encounter it greatly oversimplify its nature, shades of gray and depth, which generally leads to unrealistic expectations of how things ‘should’ work. Some people have a better intuitive feel than others, but for most people it is best to just write the whole domain off as ‘irrational’ so that they won’t make any false assumptions.

Because it is volatile, the business world inflicts a lot of short-term influence on software development. This conflicts with the long-term nature of the work, so it requires making a large number of very difficult trade-offs. If the work always bends to the short-term concerns it will quickly become a mess. If it always sticks to the long-term, it will likely lose confidence and funding. Thus it needs to perform a very difficult balancing act between the two, so a good leader that finds the right trade-offs will end up taking flak from both sides. If both sides are a little unhappy, then it’s likely balanced properly.

For most people, the ability to balance the business and technical requirements is learned from long, hard and brutal experience. Intuition is usually bias to one side or the other. Even the greatest intelligence can only ‘guess’ its way through the complex interactions, getting more wrong than right. So all that is left is learning from experience; from the successes and mistakes of the past. And it takes some painful introspection to really be objective about the many causes of failure. People like to blame others, but within the spidery web of development, all things are related and no one is immune from influence.

User Domain

Software starts with a problem that needs to be solved. That problem is always domain specific, such as a financial system, inventory or even social networking, etc. Most domains have their own unique set of terminology, usually steeped in history. They often have ‘rules of thumb’ or dirty little secrets lurking in their darkened corners. What they never are, is laid out in a nice rational manner all ready to be modeled in software. Often the data is poorly understood, the process disorganized and each organization within the domain is slightly different. There are always rules, but they are not always followed rigorously, nor particularly logical.

Thus the core problem in software is taking some ill-defined ‘informal system’ and finding a reasonable mapping to a very rigid formal system modeled inside of a computer. In practice this makes the analysis of an existing domain one of the most crucial parts in arriving at a functional software system. However, because it is always gray and messy, it is also the part of the process that people ignore the most often. They just start coding, and hope that somehow, after a while, the answers will come to them. Sometimes that works, but more often getting off on the wrong foot is extremely fatal. If you build too far from the actual answer, then you’ll have no choice but to do it all over again. But since people don’t like to restart their efforts they usually flail at the code, hoping it will somehow work itself out. However if they started really badly, they could bang on it forever and still never get something that works.

A very common problem in development is that the users rarely know what they want, or what would work correctly for their problems. Some have strong opinions, but they often lack a full understanding of the consequences of their choices. Most flip-flop faster than the development can be completed, so tying the process too closely to their wishes frequently results in a mess of half finished, or poorly thought out code. However the converse is also true, code written in an “ivory tower” far away from the users most often oversimplifies the core problems making it somewhat less than helpful. To bridge this gap requires domain experts, and often a very deep understanding of the real problems. Everything the users say is important, but not all of it needs to be taken literally. They usually understand their own informal systems, but the mapping to software, and the code itself are best left to people with plenty of experience. It is easy to write code, but extremely difficult to know what code is right for the solution.

Technology

Last, but not least are the technologies being used. Most projects involve anywhere between 3 and 30 major technologies, and a slew of minor ones. Each technology is eclectic in its own unique way and needs some time and experience to discover its strengths and weaknesses. People often focus on the programming language when assessing skills, but in most projects crafting conditionals, loops and slicing & dicing code are the easy parts. There is skill involved in solving the endless array of coding puzzles, but unless the project is pushing the bleeding edge of technology, it becomes significantly easier as one gains experience. Unfortunately, getting heavily seasoned (>15 yrs) programmers is difficult, since they are expensive and often leave the industry.

Software development is hugely affected by scale and by the desired quality of the final work. For scale I usually break it down as:

• small - 1 developer, <30 lines="lines" span="span">
• medium - 1-4 developers, <99 lines="lines" span="span">
• large - 5-20 developers, <1 lines="lines" span="span">
• huge - teams of developers, millions of lines

Projects don’t jump scale easily, they often require a nearly complete rewrite to go from one size to another. Disorganization and bad practices often work OK for small and medium projects, but they become fatal beyond that.

As well as scale, the desired quality is important. I generally see it as:

• prototype - rough proof of concept
• demo - a small working set of features, mostly works.
• in-house - eclectic, lots of rough edges, inconsistencies and operational issues
• commercial -- solid, dependable and beautifully designed by graphic designer / UX experts. Looks good, works correctly.

Each increment in quality takes considerably more work and requires specialists to focus on their particular strengths. Products often sell although they are just in-house quality, but then they are vulnerable to stronger competitors. Users may not complain directly about inconsistencies, but they do generate a negative impression of the system. Badly architected ‘balls of mud’ can often degenerate in quality as programmers randomly slap weak functionality into the corners. Poor development practices tend to be reflected in the interface, so often the outside of the system is a good indication of the state of the code base.

A rather dangerous development trade-off often comes in the choice to build or buy (paid or free). Depending on the technical specs, building is often extremely time-consuming, but in my career I’ve seen more people fail because of their choice to buy. All technologies are a collection of their author’s eccentricities, and often these play a dominate role in their usage. Buying specialty libraries and packages usually works well because you don’t have to acquire the knowledge to build them, but for the core parts of the system,if you depend on someone else’s solution you limit your options going forward. That can drive the architecture and constrain the path forward in dangerous ways.

Getting a commercial grade large or medium system to users is always a team exercise. It takes a large group of people, all with different specialties, to make it happen. As such, the team dynamics become crucial in determining the outcomes. Badly functioning teams usually produce badly functioning systems. Rogue programmers may get a lot of opportunity to express their creativity in their work, but they often do it at the expense of the overall project. Multiple different coding styles and no consistency generates high bug counts and stability problems. Once a project gets going it only continues to work if all of the people involved are on the same page, which generally means strong leadership, a reasonable process and a well-defined set of objectives. Efficiency means long-term goals, even if the short-term is volatile. Initial speed can be gained from brute force hacks, little or no reuse and heavily siloed programmers, but each of these accrues significant technical debt, which eventually bogs the project down into a mess. If the work is getting harder over time that often means disorganization, no architecture, redundancies and/or no long term plan, which if left unchecked will only get worse. A well run, well thought-out development effort will build up a considerable number of reusable common components, which if documented will guide extensions and new features. Architecture sets this commonality and management enforces its usage.

Finally

What sets software apart from most other professions is that it requires a larger cross-section of other knowledge bases to keep it successful. The danger in software is that from an outside perspective, it all looks so easy. You just have to throw together a bunch of simple instructions for the computer and chuck it all onto a server. But that type of over-simplification has always lead to disasters, caused by people who don’t have enough experience to respect the underlying complexity and trade-offs required for successful development. Most other professions are usually managed by people who have moved through the ranks. Software is often special, in that the most experienced developers are rarely put into full leadership positions. More often it is business people or domain experts that attempt to drive the projects forward, although few have the necessary prerequisites. Lack of knowledge usually equates to bad choices, which always mean more work and a much higher likelihood of failure. The complexities, knowledge and work involved in software development are easy to underestimate, so the failure rate is obscenely high.

Cooperation vs. Competition

I read this excellent article in the July issue of Scientific American. It was called “The Evolution of Cooperation”. The basis of the article was centered around game theory, but the essence of it helped me to better frame my understanding of several deep issues. Often there are interesting abstract high-level constructs that once understood, nicely overlay a sea of information, leading to a better understanding.

Our modern era stresses the benefits of competition. From the earliest ideas of a ‘free hand’ that would fix people on a level playing field, we have been inundated with the notion that things would be better if we were freer to compete. Those notions have always bothered me, since inevitably in order to gain the upper hand, people bend towards pushing the boundaries of the rules. Eventually going too far. Thus it is no surprise that the Olympics needs very serious drug testing, since the rewards of winning often outweigh the risks of getting caught. Competition may start out fair, but there are always people willing to abuse it, and gradually over time as more of them do better, the rest follow. Eventually it always degrades.

So these philosophies of unfettered competition, either sound naive or they seemed to be pushed by people who are already at the margins of fairness. They’re already bending the rules and they’d like to get away with bending them harder. I do hear what they are saying, but I suspect it to either be self-serving spin or a failure to accept the full range of expected behaviors from our species.

What the article clarified is that competition really only exists on the back of cooperation. That is, if there are no rules, there are no rules to bend. There is no competition, just chaos. So in order to compete, initially everyone has to agree on a set of rules. Thus cooperation is by far the deeper principle. It must be in place first. In that sense, it seems more than obvious as a concept, after all we’ve come together to form societies, countries, companies, etc. and these entities are all held together by rules that define how one should behave within them. By sheer scale, it seems that we cooperate far more often than we compete, but if one only reads modern literature, it would seem if the opposite were true. No doubt that is because the adherents of competition are louder and more vocal than those for cooperation.

What fascinated me about the article was that their game theory models predicted that cooperation tended to reinforce cooperation. That is, if it gets root somewhere, it tends to get larger, and while the two ebb and flow with respect to each other, the formal system model tries to balance them out in its stead state. Again, not really a surprise, but it does heavily contradict those that preach progress through increased competition. It points towards that philosophy as being unbalanced (and thus unsustainable).

So far all of what I’ve said has been applied generally across the behavior of our organizations, what does that have to do with software you’re wondering? The underlying cultures of software development have always tended towards programmers getting more ‘freedom’. That’s a recurring theme and one that has always bothered me. Freedom in its simplest form is just a lack of rules. And as society shouts so frequently, a lack of rules means that it is easier to compete. So the essence of our cultural values in programming is towards individuals competing against each other in the guise of being able to push their own creative boundaries for their solutions. It’s every programmer for themselves.

That is fine when the output is limited to the amount of work a single individual can do. So if we wrote programs that were no longer than one man-year’s work of effort, the programmers would do best if they were free to code these in whatever eclectic manner they choose. However, the era of small software ended a long time ago. What we are most often building now is built on a huge number of man-years of effort. Big systems. The little stuff exists already, it is the big stuff we are struggling with.

In all the different projects I’ve worked on, one thing has always remained true. If the project is large, it will absolutely fail if the underlying programmers don’t all get one the same page together. That is, it is a team effort, and a poorly functioning team will never be successful. It doesn’t actually matter what the ‘page’ is, it doesn’t actually matter what the standards, style, architecture or technology are. All that matters is that a group of people come together and agree on how the system will be constructed, otherwise the process of constructing it quickly breaks down and the whole thing fails.

One easy way to determine who's really on a team is by what they say. The members of a team never lie to each other, even if what is being said is not pleasant. Lying is a deliberate misdirection and generally something that you might want to do to your competitors, so that you maintain an advantage over them. Telling the truth, as you know it (even if it turns out to be wrong) is what you do when you are cooperating. You are trading your understanding for theirs, so that it is shared and everyone gets on the same page. Withholding information, particularly in this case, is a form of lying. If you let an opportunity pass without speaking up about something important, then you are doing so for competitive reasons. You are deliberately choosing not to alter the direction and you are essentially lying about the fact that you don’t have any ‘other’ information to share.

Given a huge rise in expectations for software, and that the things we seek to build far exceed the complexity manageable by a single individual, getting better results for software development comes directly from getting programmers to cooperate more with each other. However, our culture (and certainly many of the discussions on the web) show that we programmers are highly competitive. We’d rather have our freedoms, and maximize them at the expense of the whole project failing. Or industry. And when that sort of problem happens, we’re quick to blame each other for the problem, rather than accepting that the root cause was a failure (on our part and theirs) to get onto the same page and cooperate with each other. We see this over and over in the industry. Excessive squabbling about how ‘theirs’ is wrong, and ‘ours’ is better. About ‘right’, and about ‘perfect’. Now one doesn’t expect to get cooperation at every level in the industry, our species would never accept that, but what we’ve seen so far is that that we are far more competitive than we should be. Everyone is out for their own personal glory (although ‘glory’ is not always defined in the same way) and as a result everyone heads out into their own unique direction, waving the banner of ‘freedom’. The consequences of this behavior have been obvious. The failure rate for software is stunningly high and there is more energy applied to telling others they are wrong, then there is for civil discussions on what would be better. The number of programmers in the field has been increasing, but the number of innovations in software (not hardware) has been dwindling. We occasionally see some interesting new ideas, but generally there are constrained to a small group of individuals. Little pockets, here and there. There hasn’t been a real major shift in technologies for well over a decade now. Just a few fragments.

So that article was really informative in laying a basis for learning how to improve things. Competition pushes us, but it also stagnates us. It motivates us, but it also limits us. If we want to move forward, then the only way to do so is via cooperation, and in order to cooperate we have to all be willing to all play by the same rules. The specifics of the rules don’t matter, just that we all play by them. It is sometimes a hard thing for people to accept, and certainly in our modern age it has become harder, but it really represents the only way to move forward. We can choose to all do things differently, but that choice also means that we’ve put a cap on what we can now do, and what we have now isn’t particularly impressive. Or we can come together to build really big spectacular things. Most of us want to build better software but in order to do that, we have to give up many of our cherished freedoms. Excellence comes with a price attached.

What is Complexity?

This is going to be a long and winding post, as there are always fundamental questions that do not have easy or short answers. Complexity is one of those concepts that may seem simple on its surface, but it encompasses a profoundly deep perspective on the nature of our existence. It is paired with simplicity in many aspects, which I wrote about in:

http://theprogrammersparadox.blogspot.ca/2007/12/nature-of-simple.html

It would be helpful in understanding my perspective on complexity to go back and read that older post before reading further.

The first thing I need to establish is that this is my view of complexity. It is inspired by many others, and it may or may not be a common viewpoint, but I’m not going to worry in this posting about getting my facts or references into the text. Instead, I’m just going to give my own intuitive view of complexity and leave it to others to pick out from it what they feel is useful (and to disregard the rest).

Deep down, our universe is composed of particles. Douglas Hofstadter in “I am a Strange Loop” used the term ‘epiphenomenon’ to describe how larger meta-behavior forms on top of this underlying particle system. Particles form molecules, which form chemicals, which form into materials, which we manipulate in our world. There are many ‘layers’ going down between us and particles. Going upwards, we collect together as groups and neighborhoods, based in cities in various regional collections to interact with each other as societies. Each of these layers is another set of discrete ‘elements’ bound together by rules that control their interaction. Sometimes these rules are unbreakable, I’ll call these formal systems. Sometimes they are very malleable: thus informal systems. A deeper explanation can be found here:

http://theprogrammersparadox.blogspot.ca/2011/12/informal-ramble.html

If we were to look at the universe in absolute terms, the sum total of everything we know is one massive complex system. It is so large that I sincerely doubt we know how large it actually is. We can look at any one ‘element’ in this overall system and talk about its ‘context’; basically all of the other elements floating about and any rules that apply to the element. That’s a nice abstract concept, but not very useful given that we can’t cope with the massive scale of the overall system. I’m not even sure that we can grok the number of layers that it has.

Because of this, we narrow down the context to something more intellectually manageable. We pick some ‘layer’ and some subset of elements and rules in which to frame the discussion. So we talk about an element like a ‘country’, and we are focused on what is happening internally in it or we talk about how it interacts with the other countries around it. We can leverage the mathematical terminology ‘with respect to’ -- abbreviated to ‘wrt’ -- for this usage. Thus we can talk about a country wrt global politics or wrt citizen unrest. This constraints the context down to something tangible.

A side-effect of this type of constraint is that we are also drawing a rather concrete border around what is essentially a finite set of particles. If we refer to a country, although there is some ambiguity, we still mean a very explicit set of particles at a particular point in time (inferred).

So what does this view of the world have to do with complexity? The first point is that if we were going to craft a metric for complexity then whatever it is it must be relative. So it is ‘complexity wrt a,b,c,.., z’. That is, some finite encapsulation of both the underlying elements (possibly all the way down to particles or lower) and some finite encapsulation of all of the rules that control their behavior, at every layer specified. Complexity then relates to a specific subsystem, rather than some type of absolute whole. Absolute complexity is rarely what we mean.

In that definition, we then get a pretty strong glimpse of the underpinnings of complexity. We could just take it as some projection based on all of the layers, elements and rules. That is of course a simplification of its essence and that in itself is subject to another set of constraints imposed by the reduction. Combined with the initial subsystem, it is easy to see why any metric for complexity is subject to a considerable number of external factors.

Another harder. but perhaps more accurate way of looking at complexity is as the size of some sort of multidimensional space. In that context we could conceive of what amounts to the equivalent of a ‘volume’, a spatial/temporal approach to looking at the space occupied by the system. This allows use to take two constrained subsystems and roughly size them up against each other. To be able to say that one is more ‘complex’ than the other

Complexity in this way of thinking has some interesting attributes. One of them is that while there is some minimum level of complexity within the subsystem, organization does appear to reduce the overall complexity. That is, in a very simple system, if the rules that bind it are increased, but the increase reduces the interactions of the epiphenomenon, the overall system could be less complex than the original one. There is a still a minimum, you can’t organize it down to nothing, but chaos increases the size of complexity (which is different from the way information theory sees the world). So there is some ‘organizational principle’ which can be used to push down complexity to its minimum, however this principle is still bound by the similar constraints that hold for any restructuring operation like simplicity. That is, things are ‘organized’ wrt some attributes.

Another interesting aspect of this perspective of complexity is how it relates to information. If complexity is elements and rules in layers, information is a path of serialization through these elements, rules and layers. That is, it is a linearized syntactic cross-section of the underlying complexity. It is composed of details and relationships that are interconnected, but flattened. In that sense we can use some aspect of Information Theory to identify attributes of an underlying subsystem. There is an inherent danger in doing this because the path through the complexity isn’t necessarily complete and may contain cycles and overlaps, but it does open the door to another method of navigating the subsystem besides ‘space’. We could also use some compression techniques to show that a particular information path is near a minimal information path. So that the traversal and the underlying subsystem are in essence as tightly woven as they could possibly be.

A key point is that complexity is subject to decomposition. That is, things can appear more or less complex by simply ignoring or adding different parts of the overall complexity. Since we are usually referring to some form of ‘wrt’, then what we are referring to is subject to where we drew these lines in the space. If we move the lines substantially, a different subsystem emerges. Since there are no physical restrictions on partitioning the lines, they are essentially arbitrary.

Subsystem complexities are not mutually independent of the overall complexity. We like to think they are, but in that all things are interrelated. However, there are some influences that are so small that they can be considered negligible. So for instance fluctuations on the temperature of Pluto (the planetiod) are unlikely to affect local city politics. The two seem unrelated, however they both exist in the same system of particles floating about in space, and they are both types of epiphenomenon, although one is composed of natural elements while the other is a rather small group of humans interacting together in a regional confrontation. It is possible (but highly unlikely) that some chunk of Pluto could come crashing down and put an end to both an entire city and any of its internal squabbling. We don’t expect this, but there is no rule forbidding it.

The way we as a species deal with complexity is by partitioning it. We simply ignore what we believe is on the outside of the subsystem and focus on what we can fit within our brains. So we tend to think that things are significantly less complex than they really are, primarily because we have focused on some layer and filtered down the elements and rules. Where we often get into trouble with this is with temporal issues. For a time, two subsystems appear independent, but at some point that changes. This often misleads people into incorrectly assessing the behaviors.

Because we have to constrain complexity, we choose to not deal with large systems, but they still affect the complexity. For the largest absolute overall system, it seems likely that there is a fixed amount of complexity possible. One has to be careful with that assumption though because we already know from Godel’s Incompleteness Theorem that there is essentially an infinite amount of stuff theoretically out there as it related to abstract formal systems. One could get caught up in a discussion about issues like the tangibility of ‘infinite’, but I think I’ll leave that for another post and just state an assumption that there likely appears to be a finite number of particles, a maximum size, an end to time and thus a finite number of interactions possible in the global system. For now we can just assume it is finite.

Because of the sheer size of the overall system, there is effectively no upper limit on how complex things in our world can become. We could apply the opposite of the earlier ‘organizational principle’ to build in, what amounts to artificial complexity and make things more complicated. We could shift the boundaries of the subsystem to make it more complex. We could also add in new abstract layers would again would increase the complexity. It is fairly easy to accomplish, and from our perspective there is effectively an infinite amount of space (wrt a lifetime) to extend into.

One way of dealing with complexity is by encapsulating it. That is cleaving off a subsystem and embedding it in a ‘black box’. This works, so long as the elements and rules within the subsystem are not influenced by things outside of the subsystem in any significant way. This restriction means that working encapsulation is restricted to what are essentially mutually independent parts. While this is similar to how we as people deal internally with complexity, it requires a broader degree of certainty of independence to function correctly. You can not encapsulate human behavior away from the rules governing economies for instance, and these days you cannot encapsulate one economy from any other on the planet, the changes in one are highly likely to affect the other. Encapsulation does work in many physical systems and often in many formal system, but only again wrt elements in the greater subsystem. That is, a set of gears in a machine may be independent of a motor, but both are subject to outside influences, such as being crushed.

Overall, complexity is difficult to define because it is always relative to some constraints and it is inherently woven through layers. We don’t tend to be able to deal with the whole, so we ignore parts and then try to convince ourselves that these parts are not effecting things in any significant way. It is evident from modern societies that we do not collectively deal with complexity very well, and that we certainly can’t deal with all of the epiphenomenon currently interacting on our planet right now. Rather we just define very small artificial subsystems, tweak them and then hope for or claim positive results. Given the vast scale of the overall system, we have no realistic way of confirming that some element or some rule is really and truly outside of what we are dealing with, or that the behavior isn’t localized or subject to scaling issues.

Mastering complexity comes from an ever-increasing stretching of our horizons. We have to accept external influences and move to partition them or accept their interactions. In software, the complexity inherent in the code comes from the environment of development and the environment of operations. Both of these influence the flow and significance of the details within the system. Fluctuations from the outside needs and understanding, drive the types of instructions we are assembling to control the computer. Our internal ‘symbols’ for the physical world align or disconnect with reality based on how well we understand their influences. As such, we are effectively modelling limited aspects of informal systems in the real world with the formal ones in a digital world. Not only is the mapping important, but also the outside subsystems that we use to design and built it. As the boundaries increase, only encapsulation and organization can help control the complexity. They provide footholds into taming the problems. The worst thing we can do with managing complexity is to draw incorrect, artificial lines and then just blind ourselves to things crossing them. Ignoring complexity does not make it go away, it is an elementary property of our existence.

Micromanaging Software Development

Software development projects fail; often and badly. I’ve blogged a lot about the different causes so I won’t repeat most of that again, only to say if you include all of the various people involved in one way or another, at every level (including the end-users), the whole thing is known to be a very difficult exercise in ‘herding cats’. All of the problems stem from a lack of focus.

There are as many “theories” out there about how to prevent failure as there are ways to fail. They range all over the map, but a few of them rely on micromanagement at their core.

What is micromanagement? My first real introduction to it was from working in middle end restaurants as a kid. Most of them would be highly chaotic places but for their managers. The managers sit on the employees and make sure that they are all following the rules. They need to clock in when they start and clock out when they leave. They have to wash their hands, and the kitchen staff have to put on clean ‘whites’ and hairnets. Employees aren’t allowed to eat the food and they are always supposed to keep busy even if the restaurant is not. There are rules for preparing the food, rules for serving it and rules for handling the waste. As an employee your job is not to think, but to get your work done as quickly as possible, while obeying all of the rules.

When the manager is doing his or her job well, they’re running all over the place causing the restaurant to function like clockwork. A finely tuned machine for grinding out food and collecting revenues.

As a patron I’ve always appreciated a well-running restaurant with good food. As an employee I didn’t actually mind the discipline. At least you always knew where you stood and you didn’t have to think hard about stuff. After a while you just fell into the grove and did whatever the manager wanted you to do. Once work was over, it was over. You were done for the day.

What I  realized is that doing a job like ‘line cook’ in a restaurant is a semi-skilled labour position. It takes a bit to learn the ropes, but once you’ve acquired the skills they are pretty much constant. Even into my middle age, I can still grill a ‘mean’ steak (I’ve mastered medium rare and rare. mmmm). Micromanagement is a very good way to manage non-skilled and semi-skilled labour, since it really comes down to a manager just extending their capabilities by directing the physical work of others. And if the micromanager is found to be annoying (as sometimes they are), changes in moral really don’t significantly affect people’s ability to get the job done. It’s probably one of the best way of handling employees in this type of position.

For decades now I’ve seen many a person look enviously at how these types of organizations function and then try to apply this back to intellectual jobs like computer programming. As I’ve often said, by far the bulk of most programming is ‘jogging for the mind’ so it doesn’t always require the uber-deep levels of thought that something like research or mathematics needs. From the outside this seems to confuse people who mistake it for being closely aligned with semi-skill labour. Both require some thinking so bought ought to be the same they surmise. There is however a huge difference. When cooking for instance, my main output is the way I am handling the food as I prepare it. When programming, the byproduct of my work is typing on a keyboard or swinging a mouse around, but the main output of my work is actually in ‘understanding’ the solutions I am creating. An outsider can see that I am using the correct techniques to cook and if they have acquired the same skill set, they can help me correct flaws in how I am working. An outsider however cannot peer into my brain and see that I am correctly understanding the puzzles I am solving and worse, depending on what I am building, they may need decades to acquire the same skill set, just to be able to critique my working habits.

Even if programming is jogging, and it gets considerably easier the more you do it, there is still a huge amount of learning involved. We solve a vast array of problems, on a wide range of equipment, with a plethora of different technologies. Too many things for any single programmer to master them all, and they change fast and frequently. Stop coding for long enough and suddenly the landscape looks completely foreign. It isn’t really and a good knowledge of the many theories of computer science can really do help to keep abreast, but still the amount of knowledge required is staggering.

In principle this means that a micromanager in software is unlikely to have more expertise than his employees, and that he has no way of judging how well his employees are working through the problems (until it is too late).

I’ve seen the first aspect of this solved by creating half-managers/half-programmers. In some sense it has been very common, since well-run teams are usually lead by the seniors in the group. Some organizations have just formalized this relationship and built on it.

The second aspect of this however is more difficult to deal with. Nothing is worse than letting a bunch of programmers run wild for months or years, only to find out that they really didn’t understand what they were supposed to build or how they should build it. A great many projects have failed this way. The micromanagement solution is to reduce the work to very tiny time increment, say a few days. The manager issues the work, the programmer delivers it and then it becomes very obvious if the manager/programmer communication and the programmer’s skills are both at a functional level.

The downside to this is ownership. Once you take away a long-term commitment to building something from the coders, they’re just drones to belt out the pieces. A skilled micromanager is required to direct the process, but the inspiration and quality of work from the programmers is likely to be lackluster. The job has become akin to working in a restaurant and of course once the day is done, the programmers rush off to do better things with their time. This of course can be a functional way to develop software, but it is at the far end of the spectrum caused by trading off development risk for increased surveillance. Lacking satisfaction, the turnover for programmers is high and few of them will have the ability to become micromanagers effectively.  

There are no doubt circumstances where micromanagement works in software development, I’ve seen a couple of good examples. The work gets done, but from my personal experiences I’ve always seen that the overall quality suffers greatly for this. Attempts to formalize this relationship fail unless the significance of the manager’s role and exceptional level skill they require, is factored in correctly. Even then, with the manager basically over-extended and the staff turn-over often high, it’s no surprise that the organization is subject to significant volatility in both their output and quality. The risks have shifted from each individual programmer over to the relationship between the programmers and their handlers, but they are still there.

In general, although I find this type of organization structure interesting, I’d have to say that micromanagement is not a particularly effective way of getting over the herding cats problem. I’ve always felt that a well-educated and well-balanced team, with strong leadership is probably a more sustainable way to direct the work. A happy, engaged programmer is much more likely to catch significant issues before they’ve become too ingrained to fix. If they can transmit this upwards, while getting enough of a downwards context from management, they can be well positioned to deliver their individual work both quickly and correctly. Well that’s my “theory” anyways …

Another Stinkin’ Analogy

Yea, yea, I know. Programmers hate analogies. But I think that this attitude leads one to miss their importance. Sure the world is based on details -- facts -- but these facts are just chunks of information rooted in our physical existence. And more importantly this information isn’t mutually independent, it is tied by context to all of the other information floating about. These ties form a meta-layer of information that binds together the underlying information. It’s these higher level relationships that things like analogies, metaphors, similes try to address at a higher level of abstraction. Sure there are simplifications involved, and there isn’t always a one-to-one correspondence with all aspects of an analogy, but what there is a relationship that can be learned, understood, and applied back to help organize and utilize the facts. Facts don’t help unless you can turn them into actual knowledge …

This analogy comes in two parts. The first is simple: programming is like solving a Rubik’s cube. That is, it is a puzzle that involves figuring out a series of steps to solve it. Like Rubik’s cube there are many solutions, but there is often a minimal number of steps possible. Also, like solving the cubes, there are some states between ‘randomized’ and ‘solved’ where some of the sides appear solved, but the whole puzzle is not. Of course, programming has a significantly wider array of puzzles within it then just this one that need to be solved. They come in all sorts of shapes and sizes. Still, when coding it’s not unlike just working your way through puzzle after puzzle, while trying to solve them as fast as you can. It can often be a race to get the work completed.

The second part of the analogy comes in after the puzzle has been solved. Some people just toss the completed work into a big pile, but big software development needs structure not piles. In that sense, once each puzzle is complete it is neatly stacked to form an architecture. Usually the architecture itself has its own needs and constraints. The puzzles form the substance of the walls, but the walls themselves need to be constructed carefully in order for the entire work to hold together. Sometimes you meet programmers that don’t get that second half of the work. They think it’s all about the puzzles. But it really should be understood that the user’s don’t use the puzzles directly, they use the system and it’s the way the system holds together that makes the difference in their lives. The code could contain the cleverest algorithm ever invented, but if that’s wrapped in a disorganized mess, then its effect is lost on them.

Ok, I lied there is a third part to this analogy as well. I didn’t pick Rubik’s cubes lightly as the underlying type of puzzle. In the second part, obviously the cubes can also act as bricks in a larger structure, but there is still more. For anyone that has spend time solving Rubik’s cube from scratch -- no hints -- they know that it isn’t any easy puzzle. You have to work really hard to work out the mechanics of the combinations that will leave a solution in place. There are, of course, lots of books you can buy to teach how to solve the cubes quickly. Once you get the rules, it’s not very hard at all in fact these days kids compete for speed in times that are quite frankly somewhat unbelievable. Some programmers will argue that using Rubik’s cube as the iconic puzzle is a gross simplification, but I choose it on purpose since it has a rather dual nature. If you don’t know how to solve it, it is a hard puzzle. But if you do your homework first, you can get it down to an incredible speed. That ‘attribute’ holds true for programming as well. Computer languages are simple in their essence, and all libraries are essentially deterministic. If you dive in without any context, it is a difficult world where it almost seems magical how things get built. But if you gain the knowledge first to solve the types of problems you are tackling, then you can belt through them quite quickly. In a prior post I referred to it as ‘jogging for the mind’. It’s not effortless and it takes time, but it’s not nearly as hard as it is if you have no context. Write enough code, and coding becomes considerably faster and more natural.

In turning what we know into knowledge we need to collect the details together then find the rules and patterns that bind them. It’s this higher level that we apply back to our lives, to hopefully make them better in some way. This applies back to software development as well. Programming can be very time intensive. We rarely get enough time to really bring the work up to an exceptional quality. And it’s this deficit in resources that drives most of our problems in development. To counter this, we need to be as effective as possible with our limited resources, and it is exactly this point that comes right back to the analogy. I’ve spent my life watching programmers struggle in a panic to solve Rubik’s cube-like problems faster than is possible, but mostly because they didn’t just stop and find some reference on how to actually solve the puzzles first. They’ve become so addicted to doing it from scratch that they see no other way. Meanwhile, they could have saved themselves considerable stress by just looking up the answer. And to make it worse, because of the time constraints, they often settle on getting a couple of sides nearly done, not even the whole puzzle.

There are, of course, a huge variety of Rubik’s-like puzzles that we deal with on a regular basis, but easily the majority of them are well-understood and have readily available answers scattered about the World Wide Web. The time taken to find an answer is just a fraction of the time required to solve it yourself, and the more answers you find the more puzzles you can breeze through. The puzzles are fun and all, but what we’re building for the user is the structure of the system. Getting that done is considerably more fun.