Big Data from Small Source Code

Editor’s Note: This is a guest post by Denton Gentry. He’s an experienced software developer and fellow blogger from California. This years New Year resolution for Denton was to cross-post to other blogs. He’s off to a good start. Read more from Denton at his blog Coding Relic.

---

"Nobody would be interested in this code, why open source it?"

"I'm not really looking for people to help out on this code, its just a simple little thing."

"I wrote this to learn the language, its not good enough for anyone to see."

I suspect that most developers are proud of the code they've written, but most of that code is owned by an employer and considered proprietary. We couldn't open source it even if we wanted to. Code written on one's own time and not related to the employer's business could be open sourced, but rationalizations such as these keep us from doing so.

I know I've used them.

The traditional reasons for open sourcing code revolved around collecting a community of like-minded developers to collaborate with. It is a lot of work to build such a project, and even then most such efforts fail to get traction and simply languish. Driving such a project would take a lot of time and effort, time not worth investing for something one doesn't feel truly passionate about.

The thing is, the world has changed.

When it comes to hiring, I'll take a Github commit log over a resume any day.5 Feb via webJohn Resig
jeresig

Very subtly over the last few years, we've transitioned to a new model for information on the Internet. Its a world of Big Data, where we can draw inferences from collections of data even if the individual pieces go mostly unexamined.

• Code search engines mean someone can reference your code to figure out arguments to a particular API, even if they don't use the code itself or care about the project it is a part of.
• Framework and platform developers can use statistics from crawlers to know what parts of their API are widely used versus what is not getting traction. This can inform decisions about deprecation or API evolution.
• Aggregate volumes of code can compare the rise and fall in popularity of different programming languages, useful information for developers looking to keep their skills current.
• Resumes are dry reading, and get stale. Code commits, blog posts, etc are a living resume of one's work and skills.
• Also, quite frankly you may be surprised at how many people benefit from a posting of code. Someone stuck on a problem will go through many pages of search engine results looking for an answer.

The question now should be if there are any reasons not to publish the source of personal coding projects. Is there a reason to keep it to private? Tools like GitHub and Stack Overflow have made programming into a social activity, with new opportunities for personal and professional advancement. The only cost in taking advantage of them is time, and even at that only a little time.

The question now of whether to publish the source revolves around internal factors, not external. "Am I proud of this code?" not "will anyone use this code?"

Top-down and Bottom-up

When I was younger, there was a lot of debate about how to design and build software. The prevailing theories were that you either started at the top and worked your way into the details, or that you started at the bottom and built up the pieces.

Software systems are tools used by people, so it is important to understand what their problems are and what is necessary to solve them. The only way to do this, is to see it from their perspective. If you understand what the user is trying to accomplish and how they are doing that, you can find the best design that simplifies their lives.

Functionality should not be arbitrary buried in dis-organized menus, it needs to be at their finger-tips, right when they need it, and out of the way the rest of the time.

Design then, only comes together if you’re looking down at the problem. You have to start at the 10,000 foot view and then wind your way through all of the steps necessary for someone to complete their work. Empathy for the users, and a deep understanding of both their environment and their goals is key to creating the tools that actually make their lives better.

But, from the other perspective, software is extremely slow and expensive to write. As the user expectations have increased, even the small systems of today are significantly larger then just a few decades ago. And the more we depend on underlying libraries, the faster the complexity increases. Even if we don’t have to write all of the code, each underlying dependency brings with it a unique set of problems that requires time to understand it and time to manage it properly. Declining standards, poor design, forgotten knowledge and sloppy release procedures in these external pieces don’t help.

Our only defense against spiraling complexity is to try isolate the work as much as possible. That is, highly redundant code that haphazardly calls the underlying components, anywhere, is the type of spaghetti that quickly burns through all of the development resources by wasting time chasing sloppy mistakes. A well-architect-ed system that correctly encapsulates all of its underlying parts allows for the scope of change to be controlled. Once a problem has been settled, it should no longer crop back up in arbitrary places. Each piece must fully encapsulates a specific section of the system. This is the only way to provide a solid foundation on which bigger and better functionality can be added without fear of setting off a chain reaction of cascading bugs.

This type of design can only be achieved by building upwards. Starting with the depths of the system, not unlike a real building, each layer is carefully designed, built and stacked. The lines between the layers need to be well-understood and carefully mapped out. This type of bottom-up construction insures that the whole does not become unstable as it gets pushed and extended. It can also insures that the momentum of the project doesn’t grind to halt because of exponential explosions in complexity. It requires a bit more effort and foresight, but it is the only way to build large systems that are dependable and can continue to grow as the needs of the users increase.

So, is it top-down or bottom-up? Really it is both. The design and requirements need to be seen from the top, but the construction needs to focus on building up solid and reliable pieces from the bottom. Foresight and experience are necessary to understand the scale of the system, but projects still need to be adaptable to changes in the environment, technologies or the user’s priorities. A solid foundation is necessary to insure that the system will survive for its maximum lifespan while continuing to grow.

Although our industry pretends to turn over every five years, by now it is clear that the life expectancy for big systems is ten to twenty years. And for some systems, they may be around far longer than that. With that understanding of our history, it seems even more reckless to hack together something without a design, apply duct-tape excessively as leaks appear, or just hope to catch the flaws in testing. Good software development demands a longer-term focus as well as seeing the whole process from both the ground and the aerial view.

Facts, Opinion and Context

Over the years I seen many instances of people stating that discussions are broken down into either facts or opinion. Usually this dichotomy is used as the basis for their not wanting to read or discuss other people’s opinions, precisely because they are not facts.

Facts are the concrete building blocks that we use for knowledge. They are mostly universal (although not always) and tend to be related to specific points in time or objects. Opinions are generally personal views on the facts. They can be anchored in experience, but they may also be emotional or based on intuition. They are usually presented with some sort of bias.

However, there is -- I believe -- more than just these two categories. After all, facts by themselves are meaningless. They’re just points of data, but without the ability to tie them together in some meaningful way their only usefulness is for winning games of Trivial Pursuit. You can know all of the relevant facts, but still have absolutely no real understanding of what they mean, both on a personal level or with regard to the world around you.

What we mean by knowledge is not just the ability to regurgitate facts, but rather “a knowledgeable person” is someone who could utilize the things that they know for discussion and for accomplishing great works. Memorizing all of the syntax of a programming language for example, does not make someone a good programmer. It’s not the language syntax that matters, it is how it is used to build systems. A programmer’s ability to debug something that they didn’t write themselves is a far better indication of their understanding of a technology, then is the ability to just string together a long list of syntactically correct instructions. And it is this understanding that is the key to building some usable.

It is this other category that contains the ability to see the underlying patterns in both facts and experience, and from them build up internal mental models. A set of contexts to hold the empirical data.

Again, we do store facts in our memory,  but our understanding comes from the way we model the world around us in our minds. We can, for instance, memorize that the English word for the property of a glowing stove element is ‘hot’, but that fact needs to be coupled with an internal model that warns us that touching hot things can cause damage to our bodies. Knowing what ‘hot’ is, and what it means, is only useful if someone is able alter their behavior around objects that are ‘hot’. Understanding comes not from memorizing the facts, but instead from updating and being able to utilize these internal models.

Communications -- talking and writing -- are our attempts to share these internal models with others around us. In a sense, our efforts are only shallow projections of our internal models into either speech or the characters that make up a written language. Our internal models are too dense and too interconnected for these limited pathways to fully contain all of what we know. Often, just to get across a simple point, we have to re-iterate the same things over and over again, in many different ways before even the basic aspects of the model have been transferred to others. This limited bandwidth makes sharing what we know difficult at times.

Getting back to my point, when someone divides information up into only facts and opinions, and then ignores the opinions, they pass up -- on so many levels -- the ability to do more than just echo back trivial facts. While discussions in any medium can easily turn into heavily layered personal bias, there is often a strong component of direct knowledge or perceived patterns being shared as well. Some people are better than others at providing ‘objective observations’, but to some degree everything we say or do reveals some of our internal models. To dismiss that, is to choose to not understand more of the world around them, or the essence of a specific problem.

You can’t learn to program from a reference manual on language syntax alone. Sure it will tell you the syntax, variables, grammar, etc. about the language, but it provides no insight on how to utilize these things properly. Reading just a tutorial, if it is written well, provides way more context on which to base your internal models. And it is exactly this context that is necessary in order to utilize the language properly. Some people can get going from just a reference guide, but if you look carefully you can see that their works are highly eclectic, and while they may work sometimes, they certainly shouldn’t be considered to be good, or even reasonable examples.

Understanding the overall context in which things are happening, or are required to happen is essential to being able to build up a complete enough internal model. The ‘ten-thousand foot view’ is just as critical as the facts themselves.

In this sense, when I see people dismiss opinions I know that in their choice to deliberately wear blinders they are denying themselves the access to the necessary resources that will aide their own understanding. My opinion is that, since everything to some degree is related to everything else, there is no knowledge that is useless for people to learn, if and only if they can put the underlying facts correctly into their internal models so that, what they learn, they can use. Programmers for instance can easily make use of an understanding of writing, art or politics, since all three of these “unrelated” knowledge bases easily effect their output. But inspiration, understanding or patterns can come from any other collection of information, no matter how unrelated.

Sometimes it does require some effort to ‘filter’ out the underlying knowledge from an opinion that is not strongly based on either facts or experience, but even in wading into those depths one’s understanding can be greatly enhanced by considering these alternative perspectives. If you read enough different viewpoints, not unlike stereoscopic vision, you become more capable of re-constructing the real factual landscape from all of the different angles. Even the lamest opinion contains usable information, even if it only shows that someone out there can be thinking that way.

Over the years I’ve met lots of people who have filled their brain with a tremendous amount of facts, but very little knowledge. Like the character of Sheldon in the sitcom Big Bang Theory, they tend to believe in their own superiority, even though they are moving awkwardly through the world around them. But its not the facts that make us act smartly, it is how we’ve groked the context and utilized that understanding to create vast internal models that we can then use to smooth the way through this life. Success is not being able to list from memory all of the elements of the period table, in order, but rather being able to dream about accomplishing something, and then finding one’s way through the maze fast enough to make it a reality. Without the ability to utilize them, facts are nothing.