Innovation

For a long time now, I’ve been noticing that the pace of innovation in software has been slowing. I started paying attention to the industry in the mid-80s, so that breadth gives me a pretty good perspective on the (cyclic) changes.

There are a couple of really great links that I thought shed insight into these problems. This first one gives a great general idea about knowledge transfer: 

https://www.youtube.com/watch?v=pW-SOdj4Kkk

And this second one takes the perspective of how it is acting from an economical standpoint:

https://issues.org/behind-technological-hype/

I’ve had a strong sense that ‘software projects’ act as fishbowl allowing us to observe complexity problems in a reduced, constrained scale. Our projects usually have all of the core fuzzy issues including personalities, agendas, irrational behavior, politics as well as the underlying technical ones like constraints, strict logic, discrete components, etc. That is, they exist in that intersection between mathematics and the physical world. This volatile mix plays out all across our societies, in different industries, in different ways, but it is somewhat smaller and easier to observe in software because the projects are often tiny. 

In the mid-90s I saw the lure of commercialization pull away a lot of the pure and applied researchers. There are pockets left out there, but they are all underfunded. Decades of little research eventually resulted in stagnation on moving forward. We basically locked ourselves into construction techniques that haven’t moved for at least 20 years, all while the number of programmers has exploded and the amount of available code that we can leverage grew even larger. Lots of people, lots of code, but we still don’t see innovative products coming out very often, and we’ve seen a huge decrease in the quality of the code produced. 

“And some things that should not have been forgotten were lost. History became legend. Legend became myth. And for two and a half thousand years, the ring passed out of all knowledge.”

― Galadriel in The Lord of the Rings: The Fellowship of the Ring, The Lord of the Rings

If there are any billionaires out there that would like to help reverse this problem, there are a lot of people like myself that are fountains of innovative ideas but have no serious way to explore them. Once life grants you a few dependencies, and takes away some of your energy, working a full-time, high-stress job to pay the bills is cognitively demanding enough that we need our nights and weekends to recover. Personal research projects tend to take hundreds of times longer as side-projects and are prone to disruptions and so they aren’t pursued often and have a super-high rate of not getting finished. Instead of toiling away on another half-finished idea, I’ve tended to just write them up in the blog and move on. It’s less frustrating.

Of the brighter sources of help is MacArthur fellowship grants, but they are pretty much just focused on academics in the USA. The idea is great though, that someone can grant you a five year, no strings attached, opportunity to pursue your ideas. I think if this type of financing was more easily available, we’d see a lot more people trying out innovative software ideas, and eventually, some of those ideas would result in huge improvements in our industry. 

The reason we don’t see this type of innovation in private companies is that most of the focus, as expected, is on monetization. The biggest and most pressing problems in software development don’t have direct ways of making money. They layout a base platform on which the upper layers may be monetizable, but much like the Internet, if they get bent too early in just making money, that hurts both their construction and usage. We need cooperative platforms first before we can rest competitive games on top of them. Skipping step one tends to craft speculative games, not solid ecosystems. 

If we wanted to lift the software development with innovation, we really have to go backward and explore a lot of vague ideas that are attached to very deep root causes. We suspect that there are better ways of framing our construction processes that would result in faster, more stable systems. We know that as we become more dependent on software, we can’t continue to live with our low-quality construction methods. If we rely on some code, that code has to work within a very tight tolerance, and we currently can’t afford the massive amount of time we know is needed to achieve that type of precision. There are so many fundamental issues that we’ve ignored over decades in our mad rush to just put half-baked products into the market. At some point, this has to change if we want to move forward.