Defensive Coding: KISS

Nothing gets programmers into more hot water more than the ‘keep it simple, stupid’ (KISS) principle.

There are 2 important ways it causes grief. 

The first is when programmers assume it applies directly to them. They want to keep their work simple, so they want to keep their code simple.

The failure here is that there is always some ‘intrinsic’ complexity that is totally unavoidable as a byproduct of our physical universe and its history. If a programmer ignores some of that complexity and makes their code super simple, the complexity hasn’t gone away, it has just moved elsewhere. More often than not, it has gone directly to the users. 

So, now the code is simple, but using it has become significantly more complicated than necessary. A super simple feature either doesn’t do much or it is horrifically hard for users to get real things accomplished. Either way, its value is limited.

If you want to keep users interested in using a system, then the code in it has to really solve their problems and it has to keep it simple for them. KISS applies to the solution, not to the way it was built. That is not simple code to write since it means ensuring that the users don’t need to lean on any outside resources. The system needs to know anything and everything about the problem, to keep it all up-to-date, and to never forget stuff, which is, of course, large and very complicated. 

KISS can be applied to a specific sub-component of the mechanics, but it only works if you are avoiding some explicit overcomplication. That is, if you are thinking about adding stuff that will definitely never get used, then you can remove that. However, if it might get used someday, then by removing it, you are actually setting up a new problem. Maybe a small one, but it also could be a huge one in a short time from now.

The other way programmers get into trouble with KISS is that they assume that if they have to build a series of components, that making each one of them simple will help simplify the whole thing. The opposite is often true.

Simplifications, almost by definition are things that are not there. Gaps in what could have been. Those things might have been necessary or they could have been completely extra, but they are still absent. Collecting together a bunch of non-related things all together in the same place is another way to describe disorganization, and oddly, collecting together a bunch of missing things, related or not, is almost the same. 

The growing absence of stuff is a form of complexity and if not handled explicitly it will get more complex. So, a small piece missing from one area can be worked around fairly easily, but when there are dozens of things missing all over, the workarounds are harder and more complex. Just remembering what isn’t there becomes a huge burden. So, if there are lots of things that are missing, their combined effect will be multiplicative, not additive, if they are not 100% independent (which they usually aren’t). Simple is not a cumulative property, complexity is. 

“Everything should be made as simple as possible, but no simpler.” -- Albert Einstein

The key part of this quote is that while you can overcomplicate things, you can also do the opposite and oversimplify them, which is bad too. When KISS is misapplied or taken to an extreme not only does it craft a system that users hate, but it will also slowly degrade into a horrific ball of complexity that at some point becomes unfixable. 

It’s always a shock to the programmers, particularly if they have been diligent about trying to simplify everything when it suddenly kicks back up and becomes the problem itself.

Stuff that is missing or not included is always going to be a big problem if you find out later that you need it. It’s that old ‘a stitch in time save nine’ saying. Filling the code with a lot of extra stuff that isn’t used ain’t great, but then it is a little better than waking up one day and realizing that what you didn’t do earlier -- when you had the chance -- is now going to derail everything.

Don’t overcomplicate things, but don’t use that as an excuse to swing way out to the opposite extreme, either. 

Developer Management

It’s very difficult to bring a group of software developers together and get a reasonably well-built software system out of the process.

The problems stem from the underlying programming culture. Coders tend to be introverted, their thinking is often extremely black-and-white and they go rogue quite frequently. 

This leads to a number of unfortunate outcomes. 

First, they are not really good at communicating the problems and issues that they are having getting the system built and running. They don’t like to admit problems. Coupled with the modern impatience to get things done too quickly, this often reduces the timeframes to considerably less than is what is necessary to do a good job. If the stakeholders aren’t aware of many of the ongoing problems, they aren’t going to have realistic expectations on the progress of the work.

Coding requires a rather rigorous logical mode of thinking. Stuff either works or it doesn’t. Things are either right or they are wrong. The computer does exactly what the programmer tells it to do, there is no room for ambiguities. If you spend your day writing code with these properties, it tends to leak out into the rest of your thinking, life, and interactions with people. The world, however, is grey and murky, and often requires a soft touch to work through the issues. A big team of people working together generates a lot of different agendas and politics. None of this is nicely black and white, so there is a lot of friction between how the programmers think the world ‘should’ operate and how it actually does operate. This leads to a lot of misunderstandings, anxiety, and confused goals. 

With a different perspective on the priorities, and a desire to not want to talk about it, programmers are infamous for just making a decision on their own and then going off with full confidence to get those things done. The problem is that those attempts are often not in sync with the rest of the project, so they basically go rogue and end up doing something that doesn’t provide value. A sub-group of coders incorrectly heading towards the wrong objectives will conflict with the important ones, so the result is poor or useless work.

It’s hard for management to distinguish between a rogue programmer and one not doing any work at all. In both cases, what gets accomplished is nothing. Usually given that their expectations are off too, this builds up a lot of tension between the developers and management. 

In the past, people like to blame the “waterfall methodology” for going off in the wrong direction and returning with stuff that was not useful. They insisted that it was the methodology that was at fault, that it was a ‘time’ problem, but there is a little more to it than that. 

If it's a big, well-defined project that takes 1.5 years to accomplish, doing it in one long continuous unit of work is a whole lot more efficient than breaking it up into little iterations and trying to stitch them together somehow. Mostly batching together similar work is more effective, is better for consistency, and for focus. 

The big failures before the turn of the century drove the stakeholders to seek out better ways of tightly controlling their software projects. The culture of programming itself helped. Both sides settled on an invasive form of micromanagement. The coders seeded control of their work. So, the prerequisites for deciding on the right work like analysis and design get ignored, while the whole effort is gamified with a rather childish bent on formality. You don’t have “excessive” daily status meetings, you have ‘standups’ instead. There isn’t a long list of pending work, it’s a ‘burndown’ chart. We don’t break down the work into tiny, little, verifiable chunks, it's called a ‘sprint’. Planning is a game, not a necessity, and the stick is called a ‘retro’ which is somehow supposed to good for you. 

Each time management was compelled to reach for the thumbscrews and lockdown inappropriate behavior, consultants came up with cutesy little names and games for implementing it, and pushing it as propaganda to the rest of the industry. It’s unfortunate. 

For me though, the fundamental problem is not upper management controlling how the system is constructed. Rather, it is the role of leading a group of developers that is confused. It’s not a technical role and it's not a business role. 

Ultimately, there are some high-level goals that need to be accomplished. The people setting those goals do not have the ability to break them down and express them as a very, very long list of complicated programming tasks. That’s the communications impedance mismatch. If you hire a bunch of programmers and can’t tell them what to do, and they won’t tell you what problems they are having, then it is pretty obvious that the project is not going to function.

So, you need an intermediary. Someone who has spent a lot of time programming, but also someone who has been around the higher-level objectives enough to understand them. They have to have a foot in both worlds because they have to accurately translate between them. 

They might not be the best programmer, or be able to solve silly little fake coding issues. They just need to have spent time in projects of similar scale. They need to get their priorities straight. They might not fully understand all of the business objectives, or be a domain expert, but they need to have empathy for the users and to have some depth in their specific domain problems. They sit in the middle, and they ensure that the upper goals are progressing, while the lower work isn’t going rogue. 

Over the decades, I’ve heard many an entrepreneur reach the conclusion that all they need is a group of students that can code a little bit in order to get their product to market. That’s kind of the classic delusion. It sees coding as a commodity that just requires enough ‘energy’ to drive it forward. Oddly, that can work for a demo or a proof-of-concept or some other introductory software that just needs to kinda work in order to grab more interest, but it fails miserably once it becomes real, mostly because the necessary skills to keep it all organized and keep it growing are missing. So, it’s a start that can be used to evaluate ideas, but not a product that will work when needed. 

Moving up to that next level means getting serious about keeping the work under control. It’s not a small gap, but actually a rather huge one. It’s not intuitive and the kids that threw together the prototype code won’t be able to magically pull it from the ethos. This is where coding switches from being a game to getting serious. It can’t afford to be ineffective anymore, it can’t afford to be disorganized anymore. Everything changes.

For medium, large and massive projects, even the smallest issues have huge, wide-ranging consequences. You learn to deal with them via experience, and it is these issues that are far more important at this point, than the actual coding itself. Fixing the code is cheap, unrolling a ball of mud is expensive. An intermediary who knows it is important to restrict ongoing dependencies, for example, is a much better asset than a coder who can craft unique algorithms. The wrong algorithm is useless, while the wrong dependencies are often fatal. 

In an industry known for its agism, and for still having a high rate of failure, you’d think it would be obvious by now that we’d know there is a missing critical component in the effort. But oddly, the stakeholders still think programmers are just cogs, and the coders still think that if they just had “more code” their problems would magically disappear. The technologies have changed, the methodologies have gotten crazier, but the underlying problems are still the same. Breaking up a months’ worth of work up into hundreds of artificial 2-week tasks doesn’t ensure that it will go any better or be more appropriate. Instead, it tends to build up a counter-culture of gaming that process. Since it’s all indecipherable from above, it does nothing to ensure that progress is actually moving as best as possible. It just provides a false sense of momentum. And the games that coders play may distract them for a while, but the necessary satisfaction from doing a good job is missing, so they aren’t getting what they want either. 

Part of programming is really boring, routine, software production. It’s just work that needs to be done. Some small parts of getting a big product out to market are creative, but more often than not the creative portions fall into the business, design, and architectural efforts. If the ideas and issues are worked through in advance, and any difficult technological issues are prototyped up front, then the rest of getting out a new release is just the careful assembling of all of the pieces in an organized manner. It’s not a game, it’s not a contest. Having someone who knows what is really important during this phase of the work is going to prevent a lot of predictable quality issues from materializing. Like any other profession, programming isn’t “fun”, but when it is done well it can be quite satisfying. 

Duality

There are two very similar ways of looking at software systems. 

The most common one is to see it as a lot of code that moves data around. Code is the primary concept. Its dual, however, is that you see it as data, with the code just assisting in getting it from place to place. Code is a secondary issue.

They may seem similar, and it's easy to miss the difference between the two, but from a higher level the second perspective is a lot simpler and a lot more powerful.

When we first learn to program, we are taught to start assembling larger and larger code fragments. First, it is small examples of branches and loops, putting some code into functions, calling other chunks of code to get them to do stuff. That, rather directly, establishes that ‘the code’ is super important. We get caught up in syntax issues, different languages, and various IDEs. Early coding tasks are to ‘write some code that ...’, interviews are uber-focused on the code too. Everything is about ‘code’.

Data is usually introduced as a secondary issue, but most often it is somewhat trivial and primitive. If we take a data structures course, the actual data is even abstracted away, we’re just left with abstract relationships like lists and trees.

That carries on through most programmers' careers. Their work assignments are often crafted as producing some code to support a feature. In many shops, the code is started first, then later they realize that there were some details about the data that were missed. If there is a bug to be fixed, it is because some code is missing or calculating the wrong values.

So its code, code, code, all of the way down. Data Is an afterthought.

The point of taking a data structures course is lost on most people. Sure, the underlying data is abstracted away but it's not because it doesn’t matter. It's the exact opposite. Data structures are a pretty complete means of decomposition. That is, you can take most large and complex programs and rephrase them as a set of data structures. Most programs are just a bunch of primitive operations happening on a fairly small set of structures like lists, trees, stacks, queues, etc. If those structural operations are pulled out and reused, the resulting code is way smaller, and intrinsically has less bugs. That’s why Donald Knuth collected them all together in the Art of Programming, that is why they keep getting taught in classes. They are the ‘power tools’ of programming, but to get them to work you have to flip your perspective on the system.

Data structures aren’t the only opposite approach. The general ideas around them got formalized and explicitly wired into languages as Object-Oriented programming. In non-OO languages, the programmers had to set up the structures themselves and keep any primitives nearby. With Objects that become a part of the language syntax. Objects bind code directly to the data not because it is fun, but so that objects that are structurally similar can be encapsulated and reused and they can point, polymorphically to other types. It is exactly the same as basic data structures, just formalized into the language.

It’s also why people writing a lot of code in OO that is focused on doing super long lists of instructions end up getting into such a mess with Object-Oriented languages. Retrofitting a brute force procedural style into objects is basically going against the grain of the language. Objects as mega-instructions clash with other such objects, which prevent reuse, are hard to understand collectively and are prone to integration bugs. It makes the code awkward, which keeps getting worse as the system gets larger.

While data structures are very powerful, they are just the tip of the iceberg and came of age in an era when data was really rare and hard to get. That all changed when computers became ubiquitous and networked. Now data is plentiful, everywhere, and often of extremely poor quality. Data was always important, but it's getting more important as we collect a lot more of it and want to draw wisdom out of what we have.

So the alternative perspective is to see a system, just by its data, and how that data flows around. 

For data to be valuable, it has to stay around for a long time, which happens when we persist it into a database. In all ways, the schema defines the possible scope and capabilities of a system. If you haven’t persisted the data, it isn’t available for usage. If you saved some of it, in an awkward format, that will percolate upwards through all of the other functionality you use it for. If the data is available, guaranteed to be correct, then most of the features that require it are fairly simple. If its hard to write some code, its often because the incoming data is a mess.

All data starts somewhere in the real world. That may seem like a controversial statement since data like log files originate in response only to changes in the digital realm, but if you accept that those come from the behavior of the underlying hardware then it makes more sense. Besides operational data, the rest of it is entered from end-users, administrators, programmers, or third-party organizations. It starts in the real world, and really only has value in the real world. On top of this raw information, we can derive other useful relationships but it all has to start somewhere.

What becomes important then is how the data flows from one location to another. For example, it may have started as an observation by a person. They used some interface to get it into persistence. Later it might be pulled from storage and used to augment some other data. That new data is shown in another interface or packaged together and sent to a remote location. Maybe in exchange, more data flows into the system from that remote site. 

If you just look at the data and ignore the code, threads, processes, etc. most systems are not particularly complex. They act as a hub to collect and distribute data to a whole bunch of different sources, people, or other machines. 

What’s needed then, to build a system that manages that data is many fragments of code that can move, decorate, or translate the data as it circulates around. If you minimize the fiddling done with that code as it travels around, you’ve optimized a large portion of the system without even realizing it. The closer those data formats are in the widgets, middleware, and persistence, the less work that is needed when it is in motion. Moving data is always expensive.

That perspective puts data first. It is more important than the code, and the idea is to minimize what the code is doing to it whenever possible. There still might be memoization or other macro optimizations that are possible, but those can be seen as refinements.

What becomes important then is keeping dependent pieces of data together and managing the structural relationships between these ‘entities’. This brings us right back to data-structures. They deal with the relationships between these larger chunks and can be built to have indirect references to the underlying entities. Why? Because the structural relationships are smaller and more common. If you get them perfected for one type of entity, you can reuse them for all types of entities. That then just shifts the problem down to picking a good set of entities and relationships that best fit the data as it originated in the real world, or basically ‘modeling’.

Now the data perspective doesn’t magically eliminate performance problems. Systems aren’t intrinsically scalable, handling huge loads has to be explicitly engineered into the system in order for it to work correctly. But seeing the system as flowing data does make it a whole lot easier to lay out an industrial scale architecture. 

Take caching for example. From a code perspective, programmers often just allocate a chunk of memory, set it up as a hash table, then do some type of lookup first to get the value. That seems to make caching easy, but eventually, the real problems show their ugly heads. If you see caching as a form of memoization where we keep a smaller pool of data closer to the action, then what is obvious is how to decide what’s in that pool and what's not in it. Caching stale data is bad, and also letting the cache grow to be the same size as the persistence is rather pointless. But on top of that, if the data may be in two places at the same time, what happens when you need to update it? The code perspective of caching forces programmers to think about memory, while the data perspective forces them to think about the quality of the data. It makes it easier to see the whole picture, which makes it easier to get the implementation correct. Once you realize that removing stuff from the cache and keeping it in sync during writes are the real problems, figuring out what code and architecture are needed to make this happen is a whole lot easier.

The same is true for putting queues between systems to resolve speed differences and with syncing external information for read-only internal copies. Pretty much all of the big enterprise issues get a whole lot less challenging. The difficult problems shift away from some tangled mess of code, to really keeping the whole thing organized and operating correctly. 

It also applies to the really big systems as well. It’s hard to figure out how to massively parallelize systems until you realize that it is just an issue about dependencies. Where data is independent, it can be safely split across threads, processes, or machines. If it's not independent, then there will be problems. So, instead of being a complex coding problem, it is really a complex data modeling one. What underlying model do we need that both approximates the real-world issues, but also guarantees some usable independences that can be parallelized? If you structure the data correctly, the rest is just spending the time to write out the code.

As an industry, we usually start by teaching new programmers to code. That’s okay, but we often fail to teach them how to flip to this dual perspective. Instead, we leave them dangling in the wind, trying to crank out code from what is clearly a much harder perspective. Sure, some people get data structure courses or even computer theory, but then they go out into the industry and none of that sticks. To make it worse, people tag it to ‘abstraction’ and ‘overcomplexity’ and keep trying to insist that writing more overly simple code, faster, is somehow going to make it better. It even percolated back into our bad interview styles. 

It permeates everything, making most serious programming tasks a whole lot harder than they need to be. If you’ve ever suspected that there was an ‘easier’ way to build systems, then you were right. If you forget about the code, and focus on the data, getting that set up correctly, then most of the coding is easy and straightforward.

Normalization Revisited

Over 10 years ago, I wrote 3 rather rambling posts on ideas about normalization:

1. http://theprogrammersparadox.blogspot.com/2008/11/code-normal-form.html
2. http://theprogrammersparadox.blogspot.com/2008/10/structure-of-elegance.html
3. http://theprogrammersparadox.blogspot.com/2008/10/revisiting-structure-of-elegance.html

It’s not a subject that has ever grabbed a lot of attention, but it’s really surprising that it didn’t. 

Most code out there is ‘legacy code’. That is, it was written a while ago, often by someone who is gone now. Most projects aren’t disciplined enough to have consistent standards, and most often any medium-sized or larger system consists of lots of redundant code, a gazillion libraries, endless spaghetti, broken configurations, no real documentation, etc. 

Worse, a lot of code getting written right now depends on this pre-existing older stuff and is already somewhat tainted by it. That is, a crumbling code base always gets worse with time, not better.

These days we use good automated tooling like gofmt (in Golang) or rubocop (in Ruby) for different languages that are capable of enforcing light standards by automatically fixing the syntax, often they are set to reformat during saving in the editor. This lets programmers be a bit sloppy in their coding habits but auto-corrects it before it stays around for any length of time. 

Just putting some of these formatting tools into play in the editors is a big help in enforcing better consistency, getting better readability, and thus better overall quality.

What does that have to do with normalizations? The idea behind normalizing things is that much like in a relational database, there is a small set of rules that control relationship properties. In a database, it is applied to structural issues for the data. In code, it can also be applied to structural issues. 

The execution of code is a serialized list of instructions that each processor in the computer follows, but we often see it as a “tree” of function calls. To do this each function is a node, calling all of the other functions as children. Mostly, it looks like a tree, but since we can have reuse, and there can be infinite loops, it’s really a directed graph. A stack dump then is just one specific ‘path’ through this structure. If we dumped the stack a lot of times and combined it together we’d get a more complete picture.

We can take any of the instructions in this list of execution steps and move them ‘up’ or ‘down’ into other function calls in the graph. We can break up or reassemble different function calls, nodes, so that the overall structure has properties like symmetry, consistent levels, and encapsulated component structure.

It is probably incredibly slow to take a lot of code and rework it into a ‘semantic’ structure, but once that is in place, it can be shifted around based on the required normalizations, then returned to the source language. It might require a lot of disk space and a lot of CPU, but the time and resources it takes are basically irrelevant. Even if it needed a few weeks and 100 Gb, it would still be incredibly useful. 

The idea is that if you take a huge pile of messy code, and set up a reasonable number of ‘non-destructive’ normalized refactors, it would just go off and grind through the work, cleaning up the code. It absolutely needs to be non-destructive so that the final work is trustworthy (or at least as trustworthy as the original code), and it needs to be fully batched, to avoid a lot of user interaction. 

You could literally set it running, go off for a two-week vacation, and come back to a trustworthy, cleaned up codebase, that had exactly 0 operational differences from the mess you had before you left. You could just flip that into production right away, without even having to do regression testing.

A well-set group of rules could clean up variable naming, shift around the layers to unwind spaghetti, put a scope on globals, and even put the basis there for commenting. As well, it could identify redundant code, even if it doesn’t do the merge itself, and it could definitely identify useless data transformations, missing error handling, inconsistent calls, and a huge number of trivial and obvious problems. Basically, it’s a code review on steroids but applied across the entire system, all at once.

It could create menial unit tests, redo the configurations, shift behavior from one type of resource to another (config file -> database, or vice versa). Basically a huge number of super useful cleanup work that programmers hate to do and procrastinate on doing until it causes grief.

To deal with interactivity, suggested changes could be put into comment blocks. So, it generates a better method in the code, comments it out, but with the same name as the replacement method. Much like dealing with SCM merges, you could read both versions, then pick the better one (and add some type of test to vet it properly). 

Once the code is normalized, it takes considerably less cognitive effort to read through it and see the problems. Most of them will be obvious. Starting something, without closing it. Doing weird translations on the data, not checking for errors, and not doing any validations on incoming data. Most bugs fall into these categories. It could also help identify race conditions, locking issues, and general resource usage. For instance, if you normalize the code and find out that it has 4 different objects to cache the same user data, it’s a pretty easy fix to correct that. Bloat is often a direct consequence of obvious redundancies. 

You could look at the resulting code right away, or just leave it to explore on an as-needed basis. Since it’s all consistent and follows the same standards, it will be way easier to extend. 

It would also be useful if it has an idiom-swapping mechanism, for languages where there are too many different ways to accomplish the same thing. If there are 4 ways to doing string fiddling, then swapping the other 3 down to 1 consistent version helps a lot for readability. 

What kills a lot of development projects is that the code eventually becomes so messy that people are afraid to change it. Once that happens, either it goes crazy as an onion architecture (new stuff just wraps around the old stuff with lots of bugs) or it slows down into maintenance mode. Either way, the trajectory of the codebase is headed downwards, often quite rapidly. 

If all it took to get out of that death spiral was to spend a week fiddling with the configuration and then a week of extreme processing, that would make a huge difference. 

Code is encapsulated knowledge about the domain and/or technical problems. That is a lot of knowledge, captured over time, of intermittent quality, and it would be a huge amount of work to go backward, throwing all of it away and starting over. Why? If you can take what was already there, already known, and leverage it, but not as an unreadable opaque box, rather a good codebase that is extendable, then life just got easier. People are somewhat irrational, programmers all have different styles, and as a big ‘stew’ of code, it is overwhelming. If you can just normalize it all into something you want to work on, then most of those historic influences have been mitigated. 

Innovation

For a long time now, I’ve been noticing that the pace of innovation in software has been slowing. I started paying attention to the industry in the mid-80s, so that breadth gives me a pretty good perspective on the (cyclic) changes.

There are a couple of really great links that I thought shed insight into these problems. This first one gives a great general idea about knowledge transfer: 

https://www.youtube.com/watch?v=pW-SOdj4Kkk

And this second one takes the perspective of how it is acting from an economical standpoint:

https://issues.org/behind-technological-hype/

I’ve had a strong sense that ‘software projects’ act as fishbowl allowing us to observe complexity problems in a reduced, constrained scale. Our projects usually have all of the core fuzzy issues including personalities, agendas, irrational behavior, politics as well as the underlying technical ones like constraints, strict logic, discrete components, etc. That is, they exist in that intersection between mathematics and the physical world. This volatile mix plays out all across our societies, in different industries, in different ways, but it is somewhat smaller and easier to observe in software because the projects are often tiny. 

In the mid-90s I saw the lure of commercialization pull away a lot of the pure and applied researchers. There are pockets left out there, but they are all underfunded. Decades of little research eventually resulted in stagnation on moving forward. We basically locked ourselves into construction techniques that haven’t moved for at least 20 years, all while the number of programmers has exploded and the amount of available code that we can leverage grew even larger. Lots of people, lots of code, but we still don’t see innovative products coming out very often, and we’ve seen a huge decrease in the quality of the code produced. 

“And some things that should not have been forgotten were lost. History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge.”

― Galadriel in The Lord of the Rings: The Fellowship of the Ring, The Lord of the Rings

If there are any billionaires out there that would like to help reverse this problem, there are a lot of people like myself that are fountains of innovative ideas but have no serious way to explore them. Once life grants you a few dependencies, and takes away some of your energy, working a full-time, high-stress job to pay the bills is cognitively demanding enough that we need our nights and weekends to recover. Personal research projects tend to take hundreds of times longer as side-projects and are prone to disruptions and so they aren’t pursued often and have a super-high rate of not getting finished. Instead of toiling away on another half-finished idea, I’ve tended to just write them up in the blog and move on. It’s less frustrating.

Of the brighter sources of help is MacArthur fellowship grants, but they are pretty much just focused on academics in the USA. The idea is great though, that someone can grant you a five year, no strings attached, opportunity to pursue your ideas. I think if this type of financing was more easily available, we’d see a lot more people trying out innovative software ideas, and eventually, some of those ideas would result in huge improvements in our industry. 

The reason we don’t see this type of innovation in private companies is that most of the focus, as expected, is on monetization. The biggest and most pressing problems in software development don’t have direct ways of making money. They layout a base platform on which the upper layers may be monetizable, but much like the Internet, if they get bent too early in just making money, that hurts both their construction and usage. We need cooperative platforms first before we can rest competitive games on top of them. Skipping step one tends to craft speculative games, not solid ecosystems. 

If we wanted to lift the software development with innovation, we really have to go backward and explore a lot of vague ideas that are attached to very deep root causes. We suspect that there are better ways of framing our construction processes that would result in faster, more stable systems. We know that as we become more dependent on software, we can’t continue to live with our low-quality construction methods. If we rely on some code, that code has to work within a very tight tolerance, and we currently can’t afford the massive amount of time we know is needed to achieve that type of precision. There are so many fundamental issues that we’ve ignored over decades in our mad rush to just put half-baked products into the market. At some point, this has to change if we want to move forward.

Technology

I'll start by proposing a significantly wider definition for the word 'technology'. 

To me it is absolutely 'any' and 'all' things that we use to manipulate our surrounding environment. Under this rather broad definition it would include such age old technologies as fire, clothes and shelter. 

I like this definition because it helps lay out a long trajectory for how technologies have shaped our world, and since many of our technologies are so firmly established -- like fire or clothing -- it really frames our perspective on their eventual impact.

My view is that technologies are neither good nor bad, they just are. It's what we choose to do with them that matters. 

Fire, for instance, is great when it is contained; we can use it for light, warmth or cooking food. It is dangerous when it is burning down houses, forests or whatever else is in its path. We've long since developed a respect for it,  we have an understanding of its dangers, and so we react reasonably whenever its destructive side emerges. 

You wouldn't try to ban fire, or declare that it is bad for our societies. People don't protest against it, and to my knowledge pretty much every living human being utilizes it in some way. It has been around long enough that we no longer react to it directly, but rather to the circumstances in which it appears.

This holds true for any technology, whether it be fire, clothing, machines, radio or computers. 

Upon emergence, people take strange positions on the 'goodness' or 'badness' of the new technology, but as time progresses most integrate it ubiquitously into our lives. Specific usages of the technology might still be up for debate, but the technology itself becomes mainstream. 

Still, new technologies have a significant impact.

Marshall McLuhan seemed to take a real dislike to TV, particullarly as it displaced the dominance of radios. His famous tag line 'the medium is the message' was once explained to me as capturing how the creation of any new technology inevitably transforms us. 

That certainly rings true for technologies like lightbulbs, radio an TVs. Their initial existence broadened our abilities. 

Lightbulbs made us free from the tyrany of daylight. Radios personalized information dissemination well beyond the limits of newsprint and pamphlets. And TVs dumped it all down for the masses into a form of endless entertainment. 

Each came with great advantages, but also with significant dark sides. By now we've absorbed much of the impact of both sides, such that there are fewer and fewer adverse reactions. Some people choose to live without any of these technologies -- some still don't have access -- but they are few.

Technology acquisition seems to have been fairly slow until the early 19th Century spawned the industrial revolution, with its nearly endless series of clever time-saving machines. 

We amplified theses wonders in the 20th Century to create mass production factories and then added what seems like a huge new range of addition technologies: computers, networks and cell phones. 

As these new inventions have swept through our societies, they too have shown their good and bad sides. 

The Internet as a technology went were previous information communications technology could never go, but also has its own massive dark underbelly. A place were danger lurks. Computers have overturned many of the tedious jobs created in the industrial revolution, but replaced their physical aspects with intellectual ones. Cell phones broke the chains on where we could access computers, but chained the users back to an almost mindless subservience to their constant neediness. 

None of these things are bad, but than neither are they good. They just are part of our slow assimilation of technologies over the ages. 

To many it may seem like we are in a combinatorical explosion of new technologies, but really I don't think that is the case. Well, not directly. 

Somewhere I remember reading that it takes about twenty years for a technology to go from idea to adoption. That jives with what I've seen so far and it also makes sense in that that period is also roughly about 'generation'. 

One generation pushes the existing limits, but it takes a whole new one to really embrace something new. Collectively, we are slow to change.

If this adoption premise is valid, then the pace for inventions is basically independent from our current level of progress. It remains constant.

What I think has changed, particullary since the start of the industrial revolution, is the sheer number of people available to pursue new inventions. 

Changes to our ability to create machines enhanced our ability to produce more food, which in turn swelled our populations. Given that weapons like nukes dampened the nature of conflicts around the globe, we are experiencing the largest ever population for our species in any time in history (that we are aware of). 

Technology spawned this growth and as a result it freed up a larger segment of the population to pursue the quest for new technologies. It's a cycle, but likely not a sustainable one.

It's not that -- as I imagined when I was younger -- we are approaching the far reaches of understandable knowledge. We are far from that. We don't know nearly as much as we think we do and that extends right down to the core of what we know. 

Our current scientific approach helps refine what we learn, but we built it on rather shaky foundations. There is an obvious great deal of stuff to learn for practically every discipline out there and there is just a tonne of stuff that we kinda know that needs to be cleaned up and simplified. 

Healthcare, software, economics, weather, management; these are all things that we do optimistically, but the results are not nearly as predictable as we would like, or people claim. On those fronts our current suite of technologies certainly has a huge distance left to go. 

Each new little rung of better predictability -- better quality -- represents at least an exponential explosion of work and knowledge acquisition. For any technology, it takes a massively long time to stabilize and really integrate it into our civilizations. 

Controlling fire was exotic at one point, but now it is no longer so magical. Gradually we collectively absorbed the ability to get reliable usage from it and lessoned its negative side, or as in the case of firemen, at least we built up a better understand of how to deal with any problems rapidly.

For each new technology, such as software, it is a long road for us to travel before we achieve mastery. It will take generations of learning, experience and practice, before these technologies will simply become lost in the surroundings. They'll no longer be new, but we'll find better ways to leverage them for good, while minimizing the bad. This is the standard trajectory for all technologies dating right back to the first one -- which was probably just a stick used to poke stuff.

With this broader definition of technologies, because it extends so far back, it is somewhat easier to project forwards. 

If we have been gradually acquiring new technologies to allow us to manipulate our environment, it is likely that we have been chasing low hanging fruit. That is, we have been inventing technologies and integrating them roughly in the order that they were needed. 

Shelter might have been first, followed by fire then perhaps clothing. Maybe not, but it would not be unreasonable to assume that people tended to put their energies into their most significant problems at the moment; we do not generally have really good long-term vision, particullarly for things that go beyond our own life times. 

With that in mind, whether or not you believe in global warming, it has become rather obvious that our planet is not the nice, consistent stable environment that we used to dream that it was. 

It's rather volatile and possible easily influenced by the life forms trapsing all over it. 

That of course shows that the next major technological trend is probably going to be related to our controlling the planet's environment in the same way that clothing and shelter helped us deal with the fickle weather. 

To continue our progress, we'll need to make sure that the continual ice ages and heat waves don't throw us drastically off course. Any ability we gain that can help there is a technology by my earlier definition. 

As well, the space available on our planet is finite. 

Navigating outer space seemed easy in the science fiction world of last century, but in practice it does appear to be well beyond our current technological sophistication. 

We don't even have a clue how to create the base technologies like warp drives or anti-gravity, let alone keep a huge whack of complicated stuff like a space shuttle running reliably. 

We're talking a lot about space exploration but our current progress is more akin to our ancestors shoving out logs into the ocean to see if they float. It's a long way from there to their later mastery of crafting sailing ships and another massive leap to our state-of-the-art cruise liners. Between all of those is obviously a huge gulf, and one that we need to fill with many new technologies, great and small.

Given our short life spans, we have a tendency to put on blinders and look at the progress of the world across just a few decades. That incredibly tiny time horizon doesn't really do a fair job in laying our the importance, or lack of importance, in what is happening with us as a species. 

We're on a long-term trajectory to somewhere unknown, but we certainly have been acquiring lots of sporatic knowledge about where we have come from. 

Of course it will take generations to peice it all together and further generations to consolidate it into something rational, but we in our time period at least get to see the essence of where we have been and where we need to go. 

Our vehicles for getting there are the technologies that we have been acquiring over millennia. They are far from complete, far from well understood, but we should have faith that they form the core of our intellectual progress. 

They map out the many paths we have been taking. 

Technology is the manifestation of us applying our intellect, which is the current course set by evolution. It tried big and powerful, but failing that it is now trying 'dynamic'; an ability to adapt to one's surrounding much faster than gradual mutations ever could.