Cooperation vs. Competition

I read this excellent article in the July issue of Scientific American. It was called “The Evolution of Cooperation”. The basis of the article was centered around game theory, but the essence of it helped me to better frame my understanding of several deep issues. Often there are interesting abstract high-level constructs that once understood, nicely overlay a sea of information, leading to a better understanding.

Our modern era stresses the benefits of competition. From the earliest ideas of a ‘free hand’ that would fix people on a level playing field, we have been inundated with the notion that things would be better if we were freer to compete. Those notions have always bothered me, since inevitably in order to gain the upper hand, people bend towards pushing the boundaries of the rules. Eventually going too far. Thus it is no surprise that the Olympics needs very serious drug testing, since the rewards of winning often outweigh the risks of getting caught. Competition may start out fair, but there are always people willing to abuse it, and gradually over time as more of them do better, the rest follow. Eventually it always degrades.

So these philosophies of unfettered competition, either sound naive or they seemed to be pushed by people who are already at the margins of fairness. They’re already bending the rules and they’d like to get away with bending them harder. I do hear what they are saying, but I suspect it to either be self-serving spin or a failure to accept the full range of expected behaviors from our species.

What the article clarified is that competition really only exists on the back of cooperation. That is, if there are no rules, there are no rules to bend. There is no competition, just chaos. So in order to compete, initially everyone has to agree on a set of rules. Thus cooperation is by far the deeper principle. It must be in place first. In that sense, it seems more than obvious as a concept, after all we’ve come together to form societies, countries, companies, etc. and these entities are all held together by rules that define how one should behave within them. By sheer scale, it seems that we cooperate far more often than we compete, but if one only reads modern literature, it would seem if the opposite were true. No doubt that is because the adherents of competition are louder and more vocal than those for cooperation.

What fascinated me about the article was that their game theory models predicted that cooperation tended to reinforce cooperation. That is, if it gets root somewhere, it tends to get larger, and while the two ebb and flow with respect to each other, the formal system model tries to balance them out in its stead state. Again, not really a surprise, but it does heavily contradict those that preach progress through increased competition. It points towards that philosophy as being unbalanced (and thus unsustainable).

So far all of what I’ve said has been applied generally across the behavior of our organizations, what does that have to do with software you’re wondering? The underlying cultures of software development have always tended towards programmers getting more ‘freedom’. That’s a recurring theme and one that has always bothered me. Freedom in its simplest form is just a lack of rules. And as society shouts so frequently, a lack of rules means that it is easier to compete. So the essence of our cultural values in programming is towards individuals competing against each other in the guise of being able to push their own creative boundaries for their solutions. It’s every programmer for themselves.

That is fine when the output is limited to the amount of work a single individual can do. So if we wrote programs that were no longer than one man-year’s work of effort, the programmers would do best if they were free to code these in whatever eclectic manner they choose. However, the era of small software ended a long time ago. What we are most often building now is built on a huge number of man-years of effort. Big systems. The little stuff exists already, it is the big stuff we are struggling with.

In all the different projects I’ve worked on, one thing has always remained true. If the project is large, it will absolutely fail if the underlying programmers don’t all get one the same page together. That is, it is a team effort, and a poorly functioning team will never be successful. It doesn’t actually matter what the ‘page’ is, it doesn’t actually matter what the standards, style, architecture or technology are. All that matters is that a group of people come together and agree on how the system will be constructed, otherwise the process of constructing it quickly breaks down and the whole thing fails.

One easy way to determine who's really on a team is by what they say. The members of a team never lie to each other, even if what is being said is not pleasant. Lying is a deliberate misdirection and generally something that you might want to do to your competitors, so that you maintain an advantage over them. Telling the truth, as you know it (even if it turns out to be wrong) is what you do when you are cooperating. You are trading your understanding for theirs, so that it is shared and everyone gets on the same page. Withholding information, particularly in this case, is a form of lying. If you let an opportunity pass without speaking up about something important, then you are doing so for competitive reasons. You are deliberately choosing not to alter the direction and you are essentially lying about the fact that you don’t have any ‘other’ information to share.

Given a huge rise in expectations for software, and that the things we seek to build far exceed the complexity manageable by a single individual, getting better results for software development comes directly from getting programmers to cooperate more with each other. However, our culture (and certainly many of the discussions on the web) show that we programmers are highly competitive. We’d rather have our freedoms, and maximize them at the expense of the whole project failing. Or industry. And when that sort of problem happens, we’re quick to blame each other for the problem, rather than accepting that the root cause was a failure (on our part and theirs) to get onto the same page and cooperate with each other. We see this over and over in the industry. Excessive squabbling about how ‘theirs’ is wrong, and ‘ours’ is better. About ‘right’, and about ‘perfect’. Now one doesn’t expect to get cooperation at every level in the industry, our species would never accept that, but what we’ve seen so far is that that we are far more competitive than we should be. Everyone is out for their own personal glory (although ‘glory’ is not always defined in the same way) and as a result everyone heads out into their own unique direction, waving the banner of ‘freedom’. The consequences of this behavior have been obvious. The failure rate for software is stunningly high and there is more energy applied to telling others they are wrong, then there is for civil discussions on what would be better. The number of programmers in the field has been increasing, but the number of innovations in software (not hardware) has been dwindling. We occasionally see some interesting new ideas, but generally there are constrained to a small group of individuals. Little pockets, here and there. There hasn’t been a real major shift in technologies for well over a decade now. Just a few fragments.

So that article was really informative in laying a basis for learning how to improve things. Competition pushes us, but it also stagnates us. It motivates us, but it also limits us. If we want to move forward, then the only way to do so is via cooperation, and in order to cooperate we have to all be willing to all play by the same rules. The specifics of the rules don’t matter, just that we all play by them. It is sometimes a hard thing for people to accept, and certainly in our modern age it has become harder, but it really represents the only way to move forward. We can choose to all do things differently, but that choice also means that we’ve put a cap on what we can now do, and what we have now isn’t particularly impressive. Or we can come together to build really big spectacular things. Most of us want to build better software but in order to do that, we have to give up many of our cherished freedoms. Excellence comes with a price attached.

Smart

We’re dropped into the torrent of churning information, then driven along by its current towards destinations unknown. In this frantic melee we’re often able to grasp at nearby facts, both large and small. Some hold these for curiosity, but others are able to leverage them against the flow, easing their passage. Smart isn’t just what we remember, but rather it’s what we understand -- well-enough -- to be able to wield it effectively as we tumble through the darkness. Smarter is going beyond that to position ourselves for a chance at the better side of the next fork as it comes surging by.