Organization

There is one simple, yet fundamental rule for organization: if someone hasn’t explicitly organized it, then it is disorganized.

I believe that one can infer this from the properties of our physical reality. We know for instance, that entropy always wins. What starts as chaos, ends in chaos. Order is a temporary state of affairs. So, without explicit action, order is highly unlikely.

This rule is especially important when building software. A reasonably large system may consist of millions of individual details, all of which need to be embedded into the system. The scope of any development project includes all aspects of analysis, design, implementation, testing and operations. It includes the process of completing the work as well as the details that go into it. There may be regulatory or licensing issues floating about as well. Each area may not appear daunting, but when taken together there is an awful lot between the genesis of an idea and actual utilization of it for practical purposes.

All of these different aspects need coherent organization. They need to be collected, sorted, categorized and normalized so that they are available to the builders, operators and often, users of the system.

In a small project, you can easily get away with a casual approach to managing these details. If there are three or four parts to something, they don’t need any explicit arrangement. But as the size of the project increases, the number of parts grows extremely fast and thus juggling, say a few hundred unorganized parts easily results in a significant number of unexpected problems. And these problems may cascade into other problems.

This effect of scale often means that sloppy habits learned from small projects generally lead to overconfidence in big ones. Just because some aspect didn’t require organization when the project was a mere 20,000 lines, doesn’t mean that it won’t when it balloons to a few hundred thousand lines. And millions of lines requires something else altogether.

As the project size increases, the effects of disorganization become more pronounced. They require more effort to control, and can balloon into more formidable problems if left unchecked for too long. Thus in serious software development, scale is everything. It’s the first thing you investigate, and it’s always the most significant aspect of any large project. The only way to tame this rampant complexity growth is via organization.

Realistically, it doesn’t matter how it is organized, so long as that organization is consistent and sustainable. But it does matter that the organizational system spans the entire scope of the effort. You can’t just organize a sub-part of the project or process and hope that it will somehow magically propagate to the other areas. You have to cover over all of the details, and all of the work getting done, and you have to insure consistent application.

And it also matters how deep the organization goes. You may have some higher level methodology, and be very organized right at the bottom in terms of the code, but if what’s left in the middle is left untouched, eventually it too will cause significant problems. Every part of the process, architecture and sub-problems needs some coherent organizing principle.

Once the dust settles, what usually brings down ambitious software projects, or grinds them to a standstill, is an explosion of complexity. Fundamentally there is nothing that you can do about that, other then partition it carefully and encapsulate it into manageable sub-parts. But it is possible to prevent any secondary complexity caused by disorganization. And it is this avoidable artificial complexity that generally spirals out of control. When stopped the project becomes tractable, but when ignored it combinatorially explodes as the project expands. A little bit of organization goes a long way ... and enough of it may save the project from a premature death ...

Lost in Thought

Computers are stupid. They take massive lists of instructions and mindlessly process them one by one. That’s purely mechanical, the result of billions of nearly invisible gates flip-flopping in a sea of electrons. A consequence set in motion by these sets of instructions.

Software -- when it is written well -- can be quite intelligent. It can remember things for you, help you organize them, control devices and even assist you in interacting with other people. Where does this intelligence come from?

The act of ‘programming’ a computer is the rather long and often painful process of assembling larger and larger lists of instructions. To get intelligence embedded into these lists it must come from the programmer. Some of these ‘smarts’ are just derived from long understood ways to combine the instructions. Some of them come from the programmer’s intuitive sense of how the world works. Some of them come from the knowledge passed onto the programmer other people who are familiar with the problems.

Thus, the source of any intelligence in the software is the thinking done by a programmer on the vast amount of knowledge acquired in relation to the solution. Programming -- often shorted to ‘coding’ is thinking, and the by-product of this thought is code.

Some code requires deep intensive thought. The programmer needs to visualize the interactions of the data in their mind, in order to linearize it down into reliable instructions. They need to explore all of the dark corners that get implied by the computer’s behavior.

With experience, most code is pretty light thinking, the mental equivalent of ‘jogging’ for the mind. It’s just smaller collections of conditionals and loops that solve very specific sub-problems. When the number of these types of pieces explodes, the heavy thinking shifts towards ways of reining in their organization and consistency. Solutions to the micro-problems in coding are most often black and white, they have definitive answers. While solutions to the larger organizational and context issues tend to revolve around difficult trade-offs, where no answer is completely satisfactory.

One of the main consequences of programming just being thoughts is that muddled or distracted thinking directly determines the quality of the code produced. If someone is rushing through a sub-problem, their answer may be fragile or they may have actually solved the wrong problem. When elegance is achieved in software development -- an increasingly rare occurrence -- it is a consequence of a well-thought out approach and a consistent implementation. The programmer understood a problem so well, that they were able to express it in its most simplest terms, and their own self-discipline was so strong that they dotted all of the ‘i’s and crossed all of the ‘t’s. The details were attended to, no matter how small.

Of course loud environments, excess stress, long hours and rampant disorganization are all easy disruptors for the ability to think clearly. So is rushing through the work as quickly as possible.

For any coding veteran, they can immediately see the underlying quality of thought that has gone into the work, whether it was an interface, API or code-base. The poor quality of workmanship almost ‘glows’ if you know what you are looking for. An interface with endless out-of-place functionality tacked all over its design speaks of disorganization and coders that don’t work well together. Inconsistent primitives for an API cry out the pain of wave after wave of quick hacks to slap on poorly thought-out extensions. A tangled mess of repetitive code sings a sad song of confusion or a rush to just get it barely working. These existing examples of other people’s work not only instruct the computer, but also tell the tales of how they were constructed and how much care and effort went into that process. They reveal far more than just functionality.

Programming is thinking. Poor quality thinking results in poor quality code. Rushed thinking results in poor quality code. Distractions result in poor quality code. Bad morale results in poor quality code. Ultimately the utility of a piece of software is dependent on how intelligent it is, and that comes directly from the efforts of the people that built it. Crap quickly tossed together in a hyperactive sweatshop is, well, crap. If you really want to help people solve their problems, you have to spend a lot of time thinking very deeply about a solution that works. There is no other way.

Working Environments

I’ve become increasingly interested in the different types of working environments currently available for programmers and software developers. My past has included a wide variation of project sizes, companies, technologies and system quality. That gives me a sense of how much variance is out there in different development shops, but my exposure has been limited to companies in Toronto, London and Waterloo. I’d like to know what other environments exist, and how people feel about them.

For anybody interested in describing where they work or have worked in the past, you can comment on this post or if you’d rather, you can send me email directly. If I get enough feedback, I’ll put together some type of summary in a post (but no company names).

Questions about different environments:

What type of development work is going on? What’s the domain and the core functionality? Is the system sophisticated? What sort of planning and design occurs? How long does that extend into the future? What’s the methodology used when building?

Is the culture to get it down quickly, or is it to get it done right? How long do people spend analysing vs. building vs. testing? How much research goes into the underlying algorithms? How about interface standards? How is technical debt. handled? What types of documentation is getting done? What about code reviews?

Who is making the product decisions? Who is making the technical ones? Who does the analysis? What role do domain experts play, if any? Who is making the interface choices? If the project is large, is there consistency to the interface? Are there graphic designers? Editors?

How many bugs are being found? Is there a separate QA group? A separate operations dept? When bugs are fixed, do they ever reoccur again? Do related problems occur frequently?

Are projects coming in close to their schedule? How often are releases happening?

What’s the overall environment like: fun or serious? Is it expected that everyone works overtime? How much overtime do people normally work? Do they encourage people to have a healthy work/life balance?

What is the office environment like? Is it quiet? Private? Are the software developers all together? What about management? Are there technical managers? Do they code as well?

How does this environment compare to others you’ve experienced?