Partial Knowledge

We live in an amazing time. Collectively our species knows more about the world around us than any one individual could ever learn in their lifetime. Some of that knowledge is the underlying natural complexity of our physical existence. Some of it is the dynamic complexity of life. On top of this, over a long history, we have built up hugely complex social interactions, that have been evolving to maintain our growing technological innovations. Altogether, the details are staggering, and like swiss cheese, our understanding of it all is filled with countless holes.

Full knowledge is rather obviously impossible for any individual. The best we can do is focus in on a few areas and try to absorb as many details as possible. We all have partial knowledge. To make matters worse, often the details are non-intuitive. An overview is a simplification, so it is easy on the outside to misunderstand, to think that the issues that matter are different from what really matters on the inside. We see this often, with people misdirecting resources because their knowledge is too shallow.

Given our out-of-control complexity, it is not surprising that there are frequent and long-running problems with our societies. Each new generation hyper-focuses on their own partial understanding, pursuing changes that rather than fix the problems, just shifts them elsewhere and slowly amplifies them. This gets continually repeated, as the underlying general knowledge gets watered down. In time most people start knowing less about the specifics because they get swapped with the volume, their understandings become increasingly destructive. Their changes converge on making things worse.

What we need as a species is to be able to merge our partial knowledge. To bring it all back together, generation after generation, so that we are moving forward, rather than just cycling between calamities.

Knowledge was initially trapped in an individual. We eventually learned to verbally communicate it to our peers, then to preserve it for generations in physical form, but other than the gradual simplifications brought on by our repetitive communication we really don’t know how to extract the essence of underlying truth from it. If we could pull out the truth from the things we say, and we could combine it across large numbers of people, we could reassemble it in ways that enlighten our perspective on our world, our lives, our future.

What lies under the heart of all communications is a cloudy mix of aspects of our existence. Never really correct, but often not obviously wrong. As it flows through people over time it maintains that anchor to its origins but reshapes itself from other exposure. It is slowly merging, but what we really want is to be able to quickly identify it, strip it down, collect it and build it back up again. We want to accelerate this, but also to ensure that we improve in quality, rather than ebb and flow. We have some means of moving forward, like the scientific method and the ideas underpinning engineering, but daily life is a constant reminder that these approaches fail often. They are not perfected and they really come to rely on individuals for success. That makes them subject to being overwhelmed by the growing tsunami of misinformation.

What we need are tools to help the individuals work together, to help them merge their partial knowledge and to allow the merging to reliably converge on a quality of understanding that allows us to fix our past mistakes without just shifting them elsewhere. We have such a tool: computers. They can amass large amounts of information and they are patient enough to sift through it, using the underlying structural information to measure fitness. We can build tools to hold massive contexts and to gradually merge these into larger and larger ones. We can build tools to help leverage these huge structures to correct and fix grand problems.

We do some of this now, but we do it on an ad hoc basis with very specific modeling. The next generation of tools needs to do it across everything we know, rather than just for tiny sub-disciplines. It’s a model and merge problem, but one that spans the perspectives and understandings of billions of people, not just a small group of researchers.

If we can’t utilize our partial knowledge to improve things then we know that complexity will continue to grow, now at an increasing rate, up to some threshold where it in itself will become the most significant problem for our species. That is, technology, being neither good nor bad, is just an amplifier. And at the point that it expands things to be large enough, the core gets so watered down that the entire edifice will stall, or even worse it will implode. It is a recurring pattern in history, it also seems to be a physical limitation of our universe, and it is this path that we are currently traveling on.

We see that we can not go on like this forever, but our reliance on partial knowledge also means that without better tools we will. That no single individual can lead us away from our fate, that now not even a small number of people will suffice anymore. We need to harness a gigantic understanding in order to correct our path and we haven’t even begun to accept this as ‘the problem’ let alone started working on it yet. In a sense, technology got us into this mess; we have to admit that and redirect our resources to craft technologies that will really get us out, not just pile on more dangerous bandages or keep shifting the problems around in circles. This isn’t a practical problem that will get accidentally solved by a couple of people in a garage, it has grown too large for that now. If we don’t choose to accept, understand and work towards a solution the universe will eventually find one for us...

Backwards

It is easy to get lost when tackling a difficult problem. It is easy to go off on a dead end, wasting a lot of valuable time. When we are presented with a lot of choices, it is very difficult to pick the right one.

For whatever reason, most people are far more comfortable with approaching problems ‘forwards’ from the top down. However, we could see the path to a solution as winding through a rather dense tree of decisions and effort. At every opportunity there are plenty of choices, options. If you make the right choice, you move to down to the next level of the problem. If you pause or head down an irrelevant path, well…

The most optimal way I’ve found is to visualize everything backwards. That is, instead of starting at the beginning, you go straight to the end. If you understand what parts of a solution ‘must’ exist, then you can work backwards to bring them together. Along the way you can collapse any special cases to simplify them. As you keep going backwards from the end, you are naturally pruning off all of those dangerous decisions, all of that useless work. You are avoiding massive amounts of wasted time.

When the outcome is predetermined, the path between there and here is straightforward. If you can walk just that path, and only that path, then you can create a solution in the fastest time possible. All of the work to be accomplished has a very specific, and targeted use, none of it is wasted.

Software gets a little more difficult because it isn’t just one version, it grows over a long period of time often changing directions a bit. As you build a system, there are usually multiple destinations at once and with usage new ones become apparent. Still, if you go backwards from each, and you understand the overall territory you are moving through, then you easily become aware of multiple overlapping paths. You can take advantage of these to further optimize not just the next solution, but a whole series of them. The better you become at identifying viable opportunities to leverage existing work, the more ambitious you can be to tackle bigger and more interesting problems. Time is always the one resource that a Software Developer has far too little of, so saving huge chunks of it is way better than trading off little chunks now for massive problems in the future. The direction of a big project is never arbitrary, and if it seems like it is, then it is only because people haven’t spend enough time to really think it through. If you don’t get that time, because you are wasting it on unnecessary decisions and dead ends, then you’re squandering your most precious resource.

Most development projects are not nearly as difficult as they seem. Most of the endless series of decisions we make are not really necessary. Much of the work done has no real practical value. We get into time crunches most often because we are not spending our time doing valuable things. While often it's hard to see the right path going forward, it is much easier to see it if you are backing up from the solution. All you need to do is flip your perspective and the rest comes naturally.

Meta-methodology

How we go about building complex software is often called ‘methodology’.

To be complete, a methodology should cover every step of the process from gathering all necessary information about the problem of deploying and operating the solution in a production environment.

What has become obvious over the years is that for the full range of software development there is no one consistent methodology that will work effectively for every possible project. The resources, scale, and environments that drive software development to impose enough external constraints that each methodology needs to be explicitly tailored in order to remain effective.

Sometimes this lack of a specific answer is used as reasoning to not have any formalities with the development, but for any non-trivial project, the resulting disorganization from this choice is detrimental to the success and quality of the effort. A methodology is so necessary that a bad one is often better than nothing.

At a higher level, however, there are certainly many known attributes about software understanding that we have learned over the last half-century. What's needed to ensure success in development projects is to extract and apply these lessons in ways that contribute to the effort, rather than harm it. Thus, if we can’t specify the perfect methodology, we can certainly specify a meta-methodology that ensures that what happens in practice is as good as the circumstances allow.

The first and most import property of a methodology is that it is always changing. That is it needs to keep up with the overall changes in the resources and environment. That doesn’t mean changes are arbitrary; they need to be driven by feedback and sensitivity for any side-effects. Some parts of the methodology should actually be as static as possible, they need to be near constants throughout the chaos or work will not progress. A constantly shifting landscape is not a suitable foundation for building on. Still, as the development ebbs and flows, the methodology needs to stay in sync.

To keep any methodology in focus, it needs to be the responsibility of a single individual. Not someone from the outside, since they would only have a shallow view of the details, but rather the main leading technologist. The lead software developer. They are the key person on the development side whose responsibility is to ensure that the parts of the project get completed. That makes sense given that their role is really to push all of the work through to completion, so how that work gets done is a huge part of their responsibilities. Rather obviously that implies that they have significant prior experience in the full breadth of the software process, not just coding. If they only have limited experience in part of the effort, that is where they will incorrectly focus their attention. If only part of the development process is working, then overall the whole process is not.

This does tie the success of the project to the lead developer, but that has usually been the case, whether or not people have been willing to admit it. Projects without strong technical leadership frequently go off the rails, mostly by just endlessly spinning in circles. Expecting good leadership from the domain side is risky because they most often have expertise in any or everything but software development, so they too tend to focus on what they understand, not the rapidly accumulating problems.

For very large scale development, a single leader will not suffice. In that case, though, there should be a hierarchy of sub-leaders, with clear delineations between their responsibilities. That’s necessary to avoid competition and politics, both of which inject external complexity into the overall process. When leadership is spending too much effort on external issues, it has little time to correct or improve internal ones. At the top, if this hierarchy, the overall picture still falls to a single individual.

Any usable methodology for software addresses all five different, but necessary, stages: analysis, design, programming, testing, and operations. Each of these stages has its own issues and challenges. To solve a non-trivial problem, we need to go out into the world and understand as much of it as possible in an organized manner. Then we need to bring that knowledge back, mix it with underlying technologies and set some overall encapsulating structure so that it can be built. All of that work needs to be coded in a relatively clean and readable manner, but that work also requires significant editing passes to be able to fit nicely into any existing or new efforts. Once it's all built, it is necessary to ensure that it is working as expected, both for the users and for its intended operating environment. If it is ready to go, then it needs to be deployed, and any subsequent problems need to be fed back into the earlier stages. All of this required work remains constant for any given software solution, but each stage has a very different perspective on what is being done.

Most problems in the quality or the stability of the final running software come from process problems that occurred earlier. An all too frequent issue in modern development is for the programmers to be implicitly, but not directly responsible for the other stages. Thus major bugs appear because the software wasn’t tested properly; because the programmers who set the tests were too focused on the simple cases that they understand and not on the full range of possibilities.

In some projects, analysis and design are sub-tasked to the programmers, in essence, to make their jobs more interesting, but the results are significant gaps or overlaps in the final work, as well as lack of overall coherent organization.

The all too common scope creep is either a failure to properly do analysis or a by-product of the project direction wobbling too frequently.

Overall stability issues are frequently failures in design to properly encompass the reality of operations. They skip or mishandle issues like error handling. Ugly interfaces or obtuse functionality come directly from design failures, such that the prerequisite skills to prevent them were not available or believed necessary. Ugliness is often compounded by inconsistencies caused by lack of focus; too many people involved.

Following these examples, we can frame any and all deficiencies in the final product as breakdowns in the process of development. This is useful because it avoids just setting the blame on individuals. Most often if a person on the project is producing substandard work it is because the process has not properly guided them onto a useful path. This property is one of the key reasons why any methodology will need to continuously tweaked. As the staff change, they will need more or less guidance to get their work correct. A battle-hardened team of programmers needs considerably less analysis and specifications than a team of juniors. Their experience tends to focus them on the right issues.

Still, there are always rogue employees that don’t or can’t work well with others, so it is crucial to be able to move them out of the project swiftly. Responsibility for evaluating and quickly fixing these types of personality issues falls directly on the technical lead. They need full authority over who's involved in the work at most stages (operations is usually the exception to this rule) and who is no longer part of the project.

All of this sets a rather heavy burden on the technical lead. That is really unavoidable, but the lead’s still subservient to the direction of the domain experts and funding so while they can modify the methodology to restructure the priorities, they can’t necessarily alter the overall scope of the work. They can’t run off and build something completely different, and if they end up not meeting at least the basic requirements the project should be deemed a failure and they should be removed. Most times this is both what the different types of stakeholders want and what they need.

Sometimes, however, what the users need is not what the main stakeholders want. In those situations tying the responsibility for the system entirely to the lead developer is actually a good thing. Their strengths in doing their job come from being able to navigate these political minefields in order to get the best possible result for the users. Without at least a chance of moving this dial, the project is ultimately bound for disaster. With the responsibilities defined properly, at least the odds are better. And if the project does fail, at least we know who to blame, and what skills they were missing.

There is currently a huge range of known static methodologies. The heavy-weight ones follow the waterfall approach, while the lighter ones loosely are called agile. For any project, the static adoption of any one of these is likely as bad as any other for reasons previously mentioned. So the most reasonable approach is to pick and choose the best pieces or qualities. This may seem like a good way to get a mess, but this should really only be the starting point. As the project progresses, the understanding of its problems should be applied as fixes to the methodology and this should be ongoing throughout the whole life of the project.

In practice, however, most gnarled veterans of software have experienced at least one decent, mostly working methodology in the past, so it's rather obvious that they start with that and then improve upon it. Rather obviously, for qualifications to lead a big development project, a lot of interest should be shown about the methodology they intend to follow, and less about the specifics of their past coding, design, analysis, testing and operation experiences, but clearly these are all tied together.

As for the pieces, software development is too subjective to trends. We forget the past too quickly and seem to keep going around having to relearn the same lessons over and over again. Good leadership has risen above this, so the right qualities for a methodology are not what is popular, but rather what has been shown to really work. For example, it is quite popular to say bad things about waterfall, but the justifications for this are not soundly based. Not all waterfall projects failed, and those that did frequently did so because of a lack of leadership, not the methodology. It does take time for waterfall projects to complete, but they also have a much better long-term perspective on the work and when run well can be considerably more effective and often more consistent. It’s not that we should return entirely back to these sorts of methodologies, but rather that some of them did have some excellent properties and these should be utilized if needed. 

At the other end of the spectrum, many of the lighter approaches seem to embrace chaos by trying to become ultra-reactive. That might fix the time issue and prevent them from steering away from what the stakeholders want, but it comes at the cost of horrendous technical debt, which sets a very short shelf life on the results.

A good methodology would then obviously find some path between these extremes, but be weighted on one side or the other because of the available resources and environment. Thus, it would likely have variable length iterations, but could even pipeline different parts of the work through the stages at different speeds. Some deep core work might be slow and closer to waterfall, while some functionality at the fringes might be as agile as possible. The methodology would encompass both of these efforts.

Because of the past, many people think that methodologies are vast tomes. That to be a methodology, everything has to be written down in the longest and most precise of detail. For a huge development effort that might be true, but for smaller scales what needs to be written is only what will be quickly forgotten about or abused. That is, the documentation of any methodology is only necessary to prevent it from not being followed. If everyone involved has remembered the rules, then the documents are redundant. And if each time there are changes, the rules can change too, then the documentation will also be redundant. As such, a small tiger team of experts might have exactly zero percent of their methodology on paper, and there isn’t anything wrong with that if they are consistently following it.

There are occurrences however whether outsiders need to vet and approve the methodology for regulatory or contractual reasons. That’s fine, but since parts of the methodology change, the dynamic parts need to be minimally documented in order to avoid them becoming static or out-of-date.

Another reason for documentation is to bring new resources up to speed faster. That is more often the case for a new project that is growing rapidly. At some point, however, in the later stages of life, that type of effort exceeds its value.

From this it is clear that methodologies should include the intercommunication between the different people and stages of the project. All of this interim work eventually influences how the final code is produced and it also influences how that code is checked for correctness. Some of the older heavyweight methodologies focused too intensely on these issues, but they are important because software really is a sum of these efforts. Thus, for example, the structure and layout of any analysis does make a direct difference to the final quality of the work, but it also can help show that some areas are incomplete and they need further analysis. The analysts then in a large project should be laying out their results for the convenience of the designers, testers and the lead developer. They may need to confirm their work with domain users, but the work itself is targeted to other stages.

Communication of the types of complex information needed to build non-trivial systems is a tricky issue. If every detail is precisely laid out in absolute terms, the work involved will be staggering and ironically the programmers will just need to write a program to read the specifications and then generate the code. That is completely hopeless in practice. The programmers are the specialists in being pedantic enough to please their computers, so most other people involved are going to be somewhat vague and sometimes irrational. The job of programming is to find a way to map between these two worlds. Still, that sort of mapping involves deep knowledge, and that type of knowledge takes decades to acquire, so most of the time the programmers have a bit of the ability, but not all of it. A good methodology then ensures that for each individual producing code they have everything they need to augment their own knowledge in order to successfully complete the work. Obviously, that is very different for each and every programmer, so the most effective methodology gives everybody what they need, but doesn’t waste resources by giving too much.

Then the higher level specifications are resolved only to the depth required by specific individuals. That might seem impossible, but really it means that multiple people in the development have to extend the depth of any specifications at different times. That is, the architects need to produce a high-level structuring for the system that goes to the senior developers. They either do the work, or they add some more depth and pass it down to the intermediates, who follow suit. By the time it arrives on a junior’s desk, it is deeply specified. If a senior does the work, they’ll just mentally fill in the blanks and get it done. This trickle-down approach prevents everything from being fully specified and resources wasted but does not leave less experienced people flailing at their workload. It also means that from a design perspective, people can focus just on the big issues without getting bogged down in too many details. All of the different individuals get the input they need, only when they really need it and the upper-level expertise is more evenly distributed across the full effort.

There are many more issues to be discussed with respect to methodology, but I think pushing them upwards to be properties or qualities of a meta-methodology is a viable way to proceed. It avoids the obvious problem with one approach not fitting for different circumstances, while still being able to extract out the best of the knowledge acquired in practice. We still have a long way to go before most software development produces results that we can really rely upon, but our societies are moving too quickly into dependence now. At this early stage, ‘software eating the world’ might seem to be an improvement, but we have yet to see the full and exact costs of our choices. Better than waiting, it would be wiser to advance our knowledge of development up to the point where we can actually rely on the results. That might take some of the mystery out of the work, but hopefully, it will also remove lots of stress and disappointment as well.

Analysis

As I said in my last post on Quality, the ‘real’ world is a very messy place.

So messy that we are designed with filters to obscure the details for stuff we are not willing to accept. These filters are relative to how we internally model the world, which is unique for every individual. Software developers seek to define precise mappings between reality and the underlying formal machinery of a computer, but intrinsically we are unable to be objective; to be able to accept the world as it is.

The foundations of great software are rooted in getting as close to reality as humanly possible. That is if we capture a significant portion of high-quality data that is correctly modeled, then augmenting that with code to perform useful functionality for the users is considerably easier. Data forms the base of this construction and the ability to structure it correctly is bound to our own perspectives. So if we want to build more sophisticated, and thus more useful software, we need to understand how to ‘objectively’analyze the world around us.

The first and most important point is to realize that all individuals have this limited perspective and most likely an agenda. They see what they want, and they bend that towards their own goals. With an information stream like that, it makes it extremely difficult to consistently get objective facts, given that the bulk of the input for analysis is these streams. However, each stream likely contains shades of the underlying truth, so examining enough of them helps to us to converge on the underlying knowledge. This equates to a very simple rule of thumb of never taking only one source of information as being correct. It is best to assume that everything from any one source might be wrong, that it needs to be verified from many sources even if it seems to be simple, complete or intuitive. Cross-reference all data, all knowledge. Always.

Assumptions are the enemy of objectivity because they are just extrapolations of data consumed from untrustworthy underlying streams. That is, the analyst has been fed some not entirely correct or incomplete knowledge, that they extend in intuitive ways but that then bends from the external reality. Poor quality at the base propagates into the conclusions, usually as an assumption of some form. The situation is also recursive since all of the prior streams that fed into this are also somewhat bent as well. Somewhere, way down, someone made a weak assumption and that has percolated upwards to cause at least one issue. Given this, it is often so amazing that we have achieved the degree of sophisticated in some areas that we have. Our interactions are so often based around low-quality information.

Once an analyst has accepted that knowledge collection is intrinsically difficult, the next big challenge is to properly organize what is collected. Disorganized knowledge is not usable, in that it is artificially more complex. Chaos injects information.

Multi-dimensional information is also inherently difficult for humans to organize, but any organizational scheme is better than none. All that matters is that a scheme is applied to everything consistently. Inconsistent subparts are just another form of disorganization. They make it easy to overlook similarities and differences, which are the generalized structure necessary to eliminate special cases.

Knowledge that is artificially fragmented is likely to be misinterpreted. It just opens the door to inconsistencies, that then lead to bad assumptions. All of these problems feed on each other.

At some point, a really good analysis will boil down to a concrete understanding of all that is currently knowable, but reality being rather informal still contains unexpected surprises. What this means is that no analysis, however perfect, is immune to time. As the clock progresses, things change. That ensures that any analysis can never be complete. It can never be done. It is a perpetual requirement that is always converging but never arriving, at a conclusion.

For software that means that any underlying data model is, and will always be, incomplete. As time progresses, the deltas might get smaller, but they will never get to zero. Accepting that, ongoing analysis is a fundamental necessity to deal with software rusting, and it is absolutely necessary to continually expanding the underlying model, not just amplifying chaos by appending new detached submodels.

Analysis is the creation of that mapping between reality and the underlying formal systems that are the core of software. Sorting out the information flows may seem to be mechanical, but the greyness of reality prevents it from being trivial. Given that the output is foundational to software, analysis should be one of the keys areas of study for Computer Science but oddly it seems to have been overlooked. One could guess that that occurred because small aspects, such as database normalization, came quickly and easily, so on the surface, they appeared as obvious and dull. The failure in that assumption is that it is only correct from the formal side of the mapping. From the other side, it clearly isn’t true. If you normalize that wrong structure, you still have the wrong structure. Normalization isn’t really the problem, so it is only a small part of the answer. We can’t formally prove that any informality is correct, so we can’t ever prove the mappings, but we still show that some are much closer to being objective than others.

I personally think we need to deeply revisit our understanding of analysis in a big way, particularly in an age where misinformation dominates truth. Not only is this affecting our software, but it also taints many other aspects of our societies. If we can’t objectively converge on truth, we’ll just spend forever lost in unrealistic agendas.

Quality

On the outside is the irrational world of the users, that has evolved chaotically over generations. It is murky and grey, with a multitude of intertwining special cases. It is ambiguous. It is frequently changing, sometimes growing, other times just cycling round and round again. The users have turned towards the computer to help them with this problem.

On the inside, deep down is the formal logic of a discrete system. It is nearly black and white, pure and it can be viewed with a rational, objective lense. There are plenty of theories, and although there are intrinsic trade-offs such as space vs. time, they can be quantified. If a system is built up well-enough from this base, it is a solution waiting to be applied.

Software is a mapping between these two worlds. It is composed of two distinct pieces: the computer code and the equally important underlying structure of the data that is being collected and manipulated. The quality of a mapping depends on the combination of both code and structure. A good piece of software has found a mapping that is effective; useful. As time progresses that mapping should continue to improve.

Both sides of the mapping are tricky, although the outside also has that extra level of irrationality. This leaves considerable leeway for there to be many different, but essentially equivalent maps that would act as reasonable solutions. There is no perfect mapping, but there are certainly lots of ill fitting ones.

To talk about quality in regards to software, we can assess this mapping. Mapping problems manifest as bugs and they diminish the usability of the software and its collected data. A good mapping needs to be complete, and it should not have gaps nor overlaps. It doesn’t have to solve all of the problems outside, but where it attempts to tackle one it should do it reliably. This means that all of the greyness on the outside needs to be identified, categorized and consistently mapped down to the underlying formal mechanics. Solving only a part of a problem is just creating a new one, which is essentially doubling the amount of work.

Because software development is so slow and time intensive, we also need to consider how the mapping is growing. Having a good initial mapping is nearly useless if the pace of development rapidly degrades it. Mappings are easy to break, in that small imperfections tend to percolate throughout. The longer the problem has been around, the more work it requires to fix it. For code, that is relative to adding more code, but for the structure it is relative to adding more data, so that imperfections are always growing worse even when no active development.

As a map, we can talk about the ‘fit’, and it is in this attribute that we can define quality. An ill fitting map is qualitatively poor, but as noted there are many different maps that would fit with higher quality. If we decomposed the map into submaps and assess their fit, we can get a linear metric for the overall system. The most obvious indicator of not fitting is a bug. That is, a system with 70% quality means that 30% of the code and structure has some form of noticeable deficiency in their usage. If you could decompose the code into 5 pieces and the structure into another 5 (ignoring weights between them), then if 6 of these pieces are complete, while 4 of them are really bad and need to be reworked the quality is 60%. If however, all ten pieces contained problems, then at least from the top-down perspective the overall quality of the system is 0%.

That may seem like an excessively brutal metric, but it is mitigated by the fact that we are only considering problems that directly bother the users, the future developers and the operations people. That is, theses are bugs, not trade-offs. Thus if the system is slow because the code was badly written, that is a bug, but if it is slow because it was weighted on a space trade-off made because of hardware limitations then unless the hardware situation changes it is not really a bug.

Obviously this metric is similar to measuring a coastline, in that getting down to finer details will change it. If you decompose part of the system into a single module of 100 lines with a bug, then the quality for the module is 0%, but if you isolate only the one line that is bad, it seems to jump back up to 99%. Quite the difference. But that points to a significant fundamental property of the mapping. Some parts of it are independent, while other parts are dependent. In the previous case of the 99 lines of code, if they are all dependent on the bad one then all 99 are tainted, thus the quality really is 0%. If only 49% are dependent, then the quality is 50%. A line of code is meaningless in isolation, it only has value when it is brought together with other lines within the system.

What we a have to assume with code though, is unless proven otherwise, there is a dependency. That again could be problematic. A programmer could see a huge code base with only 1 bug and assume that the quality is 0%. We do this quite naturally. But that illustrates the need for an architecture in a large code base that clearly delineates these dependencies. If the code can’t be easily decomposed, then the mapping is unlikely to be changeable, thus the quality really is lower, although probably not 0%.

Now that was for code, but the situation is even more dire for the underlying structure of the data. If you build a large system on a bad denormalized database, then those underlying bugs taint all of the code above it. Code written to collect bad data is inherently bad code. This is a reasonable proposition, since the code is 100% dependent on that broken structure. If the database is screwed, the value of the code is too.

Reuse and redundancy have an effect as well. If you have a tight engine for the main functionality of the system with 100K lines of code, and you have an administrative part, badly written, with 500K of spaghetti code, you might assess its quality incorrectly. Lets use the Pareto rules and say that 80% of the usage is that core engine. If there are lots of bugs in the administrative stuff, from an LOC perspective it might appear as if the overall quality was less 20%. However, if the engine was great then from a usage standpoint it is clearly at least 80%. This flip-flop shows that the value of any line of code is weighted by its functionality, and also its reuse or redundancy. A line of code in the engine is worth 20x more than one in the administration. That extremeness of the weighting is not surprising, and should be captured by the architecture at a higher level, and driven by usage or business needs. That difference shows why good architectures are necessary to assess priorities properly because they help pinpoint both dependencies and weights.

Another significant issue is trying to apply a static mapping to an inherently dynamic problem. If the essence of the problem keeps changing, then all of the collected static data is constantly out-of-date by definition. The mapping needs to keep pace with these changes, which is either a rapidly increasing amount of static coding work or the dynamic elements must be embedded in the map itself. The former approach might appear quicker initially, but will not scale appropriately. The latter is more difficult, but the scaling problems are mitigated. This comes across in the quality metric because the completeness attribute is always declining, making the structure out-of-date and then associated code. Note that this happens whether or not the code is in active development. Dynamic problems gradually degrade the quality in operations as do changes to any external dependency. They are both forms of rust.

Now given this metric, and its inherent weakness, it can be applied to any existing development project with the obvious caveat that an outsider is mostly going to undervalue quality. That is, where someone doesn’t know the weights and they don’t know the dependencies, the results will skew the effect of bugs, which every system has. That’s not as bad as it sounds, because what makes a project sustainable is the knowledge of this mapping. If that knowledge is lost, or blurry, or incorrect, then from a larger context the quality just isn’t there. If all of the original developers left and the new crew is just putting a detached bag on the side of the system instead of really enhancing it, although this new work is mostly independent, it is also redundant and full of gaps or overlaps with the existing pieces, which is reducing the quality. This gets caught quite nicely by this metric. If the knowledge is readily available, as it should be if the system is well-designed, then that feeds into both raising the quality but also making sure the existing deficiencies are less expensive to fix. In that sense a ball of mud really does have the quality of 0, even if it has a few well-written sections of code and a couple of decent structures. Good work on a small part of a bad system won’t change the fact that it is a bad system.

This metric is fairly easy to calculate in that to get a reasonable estimate all you need to know is the number of lines of code, the architecture, the denormalization of the database, the current bug list and the current usage. Often, you can even get close by just playing around with the main features of the application and guessing at the underlying code organization from the user interface. That is, if the screens are a mess and there are plenty of places where the functionality should have been reused but wasn’t, and the structure of the data being captured is erratic, then you know that the overall quality is fairly low even if you’ve never seen the code or the database. Bad mappings propagate all of the way to out to the interface. They can’t be hidden by a nice graphic design.

As Software Developers we want to build great systems; it’s why we were drawn to computers. We can’t know if the work we are contributing is any good unless we have some means of assessing the quality. We can’t know if some work is really an improvement if we have no way of understanding how it affects the overall system. Because we care about doing a good job and we want to know that our efforts have made things better, we need a means of quantifying quality that is oriented to what’s important for the usage of the software. Having a notion such as technical debt is fine, but because it is so vague it is easily misused or ignored. Knowing that there are rapidly growing problems in the mapping however allows us to visualize the consequences of our short-cuts, disorganization and design decisions. Realizing that haphazard work or muddled thinking really does cause a significant drop in the quality gives us a much better chance of raising the bar. It also gives us a means of determining if a project is improving over time, or if it is just getting worse. What the world needs now is not more code or more poorly structured data, but reliable mappings that really solve people’s problems. Once we have those, programming is just the act of making them accessible.

Factual

If I had a lot of money I’d create a new website. It would be similar to Wikipedia, but instead of descriptive text it would only contain cold, hard facts. Explicit relationships between specific things relative to time.

Structurally, the underlying encoding would be some formulaic knowledge based representation of the relationships, thus avoiding any vague or misrepresented information. Just facts, pure and simple.

It should be noted however that what we often think of facts are not necessarily ‘universally’ true. That is, even the most concrete facts are relative to a specific context. For example, we know that the sun always rises in the east and sets in the west. That will remain true for our lifetimes, but there will eventually come a time when the sun ages enough that engulfs the planet turning it into a molten cloud (or something equally as unappealing). Truth with respect to time is a tricky thing.

What is also true is that much of what we think are facts, are actually just opinions. Humans can’t grasp the full context of our existence, so we over-simplify relative to our own perspectives. In that way, informality creeps into the underlying representation. Any given fact is believed true by only a subset of our population, and there are many different subsets all with their own variety of beliefs. What is a fact to one person is guaranteed to be a falsehood to another. Only the purest of abstract mathematics is immune to this property.

To account for this, the underlying representation would contain everything that anyone would consider factual. It would, of course, also have some metric to show how popular one version is over the other, but it would allow for the encoding of all ‘alternate’ truths, theories or perspectives.

Each fact would also have an associated list of links back into the Internet, and an associated list of publications back into meatspace. These references would be there to backup the justification for preserving the relationship.

The point of collecting this huge, rather complex and highly interlinked data would be too allow us to get a sense of what we collectively know. It would also allow us to navigate through the relationships, as they are all deeply interconnected, and enhance or confirm our individual perspectives. We could use this knowledge to better understand reality, to make better decisions.

Of course this data would be massive. Eventually it would encompass everything we know, think we know or even guessed at. It would have every major and minor idea from our history. Somewhere it would list the world as being a sphere, but would also list out the flat earth concept (with the obvious metric that almost nobody believes it anymore). It would list out every political theory and every known historical interpretation. It would list everything, without bias and without fear of it being misinterpreted. Just the facts we think we know, drawn from billions of opinions, spanning the globe.

It would take a lot of money to create such a thing, and it would take considerable experimentation and research to find a suitably expressive knowledge representation, but if even a portion of it ever got created it would open up our ability to really see the world around us in all of its beautiful interconnectivity and complexity. It would enable us to become more realistic about what we collectively know, and how we often make decisions based on limited context. That, and it would be quite enjoyable to build...

Thinking Problems

What’s most interesting about reading lots of code is that you get a fairly deep insight into how people think. Building software solutions involves decomposing a bunch of related problems into discrete pieces and then recomposing them back into a solution that allows users to get their work completed. Examining these pieces shows both how the individuals involved break apart the world, and how much they understand of it to be able to put it all back together again.

Supplemented with an understanding of the history of the development and enough experience to be able to correctly visualize a better solution, this can allow us to analyze various types of common thinking problems.

The first and most obvious observation is whether or not the programmers are organized. That just flows right from how much structure and consistency is visible. Since most large bodies of code are written over years, there can sometimes be local organization, but that might change radically in different parts.

Organization is both external and internal. It applies both to the overall process of development and to how the analysts, designers, and programmers arrived at the code.

Internal organization, as it applies to thinking, is in a sense how fast one can head in the right direction, rather than just wander aimlessly in circles. Often times checking the repo sheds light, in that disorganization usually generates a large number of small fixes that come in sporadically. A good testing stage should hide this from the users, but in broken environments that doesn’t happen. So really messy code, and a boatload of little fixes, generally sets the base thinking quality. If the fixes are spread across fairly large time spans -- implying that they are driven by users, not testers -- then a significant amount of the thinking was rather disorganized.

Disorganized thinking is generally fairly shallow, mostly driven by the constant lack of progress. That is, people who make little progress in their thinking tend to want to avoid thinking about things because it doesn’t ultimately help them. That pushes them to the tendency to react to the world in the short-term. But long-term planning and goal achievement, almost by definition, require deep thinking about both the future and the means to get there. Thus disorganized thinkers are more often a-types that prefer the bull-in-the-china-shop approach to handling problems. For the most part there seems to be a real correlation between the brute force approach to coding and disorganized thinking. It’s the “work harder, not smarter” approach to getting things done.

Another really common thinking problem that is visible is with local simplification. It’s not uncommon to see programmers go from A to B via all sorts of other crazy paths. So instead of going straight there, they go A -> M -> Q -> B, or something equally convoluted. Generally, if you get a chance to inquire, many of them will rather confidently state that since A -> M is shorter than A -> B, that they have actually simplified the work and that they are correctly following an approach like KISS. What they seem to have difficulty understanding is that A -> M -> Q -> B is inordinately more complex. That the comparison between the two approaches is flawed because they aren’t comparing two equal things, rather just a part of the solution vs. the whole solution. You see this quite frequently in discussions about architecture, tools, design, etc. and of course these types of complexity choices get burned directly into the code and are quite visible.

In a programmatic sense, any code that isn’t directly helping to get to the destination is a possible suspect for being an M. More specifically, if you ignore the code, but follow the data instead, you can see its path through the runtime as being a long series of data copies and translations. If the requirement of the system is to get from the persistence storage to a bunch of widgets and then back again, then unless intermediate destinations are helping to increase performance, they are really just unnecessary waypoints. Quite often they come from programmers not understanding how to exploit the underlying primitive mechanics effectively, or from them blindly following poor conventions. Either way, the code and the practices combine to consume excess work that would have been better spent elsewhere. In that way, local simplifications are often related to shallow thinking. Either the programmer came to the wrong conclusion about what really is the simpler approach, or they just choose not to think about it at all and decided to blindly follow something they read on the web. Either way, the code points back to a thinking problem.

The third, and most complex problem is with the decomposition of things into primitive parts. It’s easy enough to break a large problem down into its parts, but what isn’t often understood well is that the size and relative positioning of the parts actually matters. There is nearly an infinite number of ways to decompose a problem, but only a select few really make it easy to solve it for a given context. Ideally, the primitives fit together nicely; that is there are no gaps in between them. Also, they do not overlap each other, so that there is a real canonical decomposition. They need to be balanced as well so that if the decomposition is multi-level, each level only contains the appropriate primitives. This whole approach is mathematical in nature since it adheres to a rigid set of relative rules, created specifically for the problem at hand.

A problem broken up in this manner is easily reassembled for a solution, but that’s generally not what we see in practice. Instead, people tear off little bits at a time, rather randomly, then they belt them out as special cases. The erraticness of the decomposition provides great obstacles towards cleanly building a solution and as it progresses over time, it introduces significant spaghetti back into the design. It’s the nature of decomposition that we have to start from the top, but to get it reasonable we also have to keep iterating up and down until the pieces fit properly; it's that second half that people have trouble with because that type of refinement process is often mistaken as not providing enough value. However, it really is one of those a-stitch-in-time-saves-nine problems, where initial sloppiness generates significantly more downstream work.

A more formal background makes it easier to decompose into solid primitives, but given the size and complexity of most problems we are tackling, knowledge or even intrinsic abilities does not alleviate the grunt work of iterating through and refactoring them to fit appropriately. This then shows up as two different thinking problems, the first being able to notice that the pieces don’t fit properly, while the second is being able to fix that specific mismatch. Both problems come from the way people think about the underlying system, and how they visualize what it should ultimately look like.

Now so far, I have been using the term ‘thinking’ rather loosely, to really refer to what is in between observations and conclusions. That is, people see enough of the problem, and after some thinking, they produce a solution. That skips over issues like cognitive bias, where they deliberately ignore what they see that does not already support their conclusions. Given that it is internal, I do consider that part of the thinking process; the necessity to filter out some observations that don’t fit the current context and focus. Rose-colored glasses, however, are a very common problem with software development. Sometimes it related to the time pressures, sometimes it's just unreasonable optimism, but sometimes it is an inability to process the relevant aspects of the world around us. It is a thinking failure that prevents us from being able to build up internal models that are sophisticated enough to match observed evidence. To some degree, everyone is guilty of wearing blinders, of filtering out things that we should not have, but in some cases, it is considerably more extreme than that. It’s really a deep-seated need to live within an overly simplified delusion, rather than reality. Strangely that can be as much of a plus as a minus in many cases, particularly if success is initially difficult. Ignoring the obvious roadblocks allows people to make a full-hearted attempt. But as much as it can launch unexpected successes (occasionally), it becomes increasingly negative as time progresses and the need for more refined thinking grows more significant.

Analysis, design, and programming are all heavily dependent on observation and thought. In the end, a software system is just an instantiation of various people’s understanding of a solution to a problem. In that, it is really only as strong as the sum of its commonly used weakest links (some parts of the system are just irrelevant). So, if the quantity of the observations is compromised by time, and this is combined with poor thinking, the resulting system most often falls far short of actually solving the problem. It is not uncommon to see systems that were meant to cut down on resources need significantly more resources than they save. That is, a grunt job for 4 people is so badly automated that 6 people have to be involved to keep it running. In that sense, the system is an outright failure and underlying that is not a technical problem, but rather one about how various people came to think about the world, and how they combined their efforts to solve things relative to their conclusions.

Thinking well is a prerequisite for handling complex problems, with or without technology. If people can’t clearly understand the issues, then any attempt to solve them is as equally likely to just make them worse. If they break down the problem into a convoluted mess of ill-fitting parts, then any solutions comprised of these are unlikely to work effectively and is more likely to make things worse. Software, as it is applied to sophisticated problems, is inordinately complex and as such cannot be constructed without first arriving at a strong understanding of the underlying context. This comes from observations, but it doesn’t come together until enough thinking has been applied to it to get it into a usable state. Software in this sense is just well-organized intelligence that gets processed by a computer. There is no easy way around this problem.