Technology

I'll start by proposing a significantly wider definition for the word 'technology'. 

To me it is absolutely 'any' and 'all' things that we use to manipulate our surrounding environment. Under this rather broad definition it would include such age old technologies as fire, clothes and shelter. 

I like this definition because it helps lay out a long trajectory for how technologies have shaped our world, and since many of our technologies are so firmly established -- like fire or clothing -- it really frames our perspective on their eventual impact.

My view is that technologies are neither good nor bad, they just are. It's what we choose to do with them that matters. 

Fire, for instance, is great when it is contained; we can use it for light, warmth or cooking food. It is dangerous when it is burning down houses, forests or whatever else is in its path. We've long since developed a respect for it,  we have an understanding of its dangers, and so we react reasonably whenever its destructive side emerges. 

You wouldn't try to ban fire, or declare that it is bad for our societies. People don't protest against it, and to my knowledge pretty much every living human being utilizes it in some way. It has been around long enough that we no longer react to it directly, but rather to the circumstances in which it appears.

This holds true for any technology, whether it be fire, clothing, machines, radio or computers. 

Upon emergence, people take strange positions on the 'goodness' or 'badness' of the new technology, but as time progresses most integrate it ubiquitously into our lives. Specific usages of the technology might still be up for debate, but the technology itself becomes mainstream. 

Still, new technologies have a significant impact.

Marshall McLuhan seemed to take a real dislike to TV, particullarly as it displaced the dominance of radios. His famous tag line 'the medium is the message' was once explained to me as capturing how the creation of any new technology inevitably transforms us. 

That certainly rings true for technologies like lightbulbs, radio an TVs. Their initial existence broadened our abilities. 

Lightbulbs made us free from the tyrany of daylight. Radios personalized information dissemination well beyond the limits of newsprint and pamphlets. And TVs dumped it all down for the masses into a form of endless entertainment. 

Each came with great advantages, but also with significant dark sides. By now we've absorbed much of the impact of both sides, such that there are fewer and fewer adverse reactions. Some people choose to live without any of these technologies -- some still don't have access -- but they are few.

Technology acquisition seems to have been fairly slow until the early 19th Century spawned the industrial revolution, with its nearly endless series of clever time-saving machines. 

We amplified theses wonders in the 20th Century to create mass production factories and then added what seems like a huge new range of addition technologies: computers, networks and cell phones. 

As these new inventions have swept through our societies, they too have shown their good and bad sides. 

The Internet as a technology went were previous information communications technology could never go, but also has its own massive dark underbelly. A place were danger lurks. Computers have overturned many of the tedious jobs created in the industrial revolution, but replaced their physical aspects with intellectual ones. Cell phones broke the chains on where we could access computers, but chained the users back to an almost mindless subservience to their constant neediness. 

None of these things are bad, but than neither are they good. They just are part of our slow assimilation of technologies over the ages. 

To many it may seem like we are in a combinatorical explosion of new technologies, but really I don't think that is the case. Well, not directly. 

Somewhere I remember reading that it takes about twenty years for a technology to go from idea to adoption. That jives with what I've seen so far and it also makes sense in that that period is also roughly about 'generation'. 

One generation pushes the existing limits, but it takes a whole new one to really embrace something new. Collectively, we are slow to change.

If this adoption premise is valid, then the pace for inventions is basically independent from our current level of progress. It remains constant.

What I think has changed, particullary since the start of the industrial revolution, is the sheer number of people available to pursue new inventions. 

Changes to our ability to create machines enhanced our ability to produce more food, which in turn swelled our populations. Given that weapons like nukes dampened the nature of conflicts around the globe, we are experiencing the largest ever population for our species in any time in history (that we are aware of). 

Technology spawned this growth and as a result it freed up a larger segment of the population to pursue the quest for new technologies. It's a cycle, but likely not a sustainable one.

It's not that -- as I imagined when I was younger -- we are approaching the far reaches of understandable knowledge. We are far from that. We don't know nearly as much as we think we do and that extends right down to the core of what we know. 

Our current scientific approach helps refine what we learn, but we built it on rather shaky foundations. There is an obvious great deal of stuff to learn for practically every discipline out there and there is just a tonne of stuff that we kinda know that needs to be cleaned up and simplified. 

Healthcare, software, economics, weather, management; these are all things that we do optimistically, but the results are not nearly as predictable as we would like, or people claim. On those fronts our current suite of technologies certainly has a huge distance left to go. 

Each new little rung of better predictability -- better quality -- represents at least an exponential explosion of work and knowledge acquisition. For any technology, it takes a massively long time to stabilize and really integrate it into our civilizations. 

Controlling fire was exotic at one point, but now it is no longer so magical. Gradually we collectively absorbed the ability to get reliable usage from it and lessoned its negative side, or as in the case of firemen, at least we built up a better understand of how to deal with any problems rapidly.

For each new technology, such as software, it is a long road for us to travel before we achieve mastery. It will take generations of learning, experience and practice, before these technologies will simply become lost in the surroundings. They'll no longer be new, but we'll find better ways to leverage them for good, while minimizing the bad. This is the standard trajectory for all technologies dating right back to the first one -- which was probably just a stick used to poke stuff.

With this broader definition of technologies, because it extends so far back, it is somewhat easier to project forwards. 

If we have been gradually acquiring new technologies to allow us to manipulate our environment, it is likely that we have been chasing low hanging fruit. That is, we have been inventing technologies and integrating them roughly in the order that they were needed. 

Shelter might have been first, followed by fire then perhaps clothing. Maybe not, but it would not be unreasonable to assume that people tended to put their energies into their most significant problems at the moment; we do not generally have really good long-term vision, particullarly for things that go beyond our own life times. 

With that in mind, whether or not you believe in global warming, it has become rather obvious that our planet is not the nice, consistent stable environment that we used to dream that it was. 

It's rather volatile and possible easily influenced by the life forms trapsing all over it. 

That of course shows that the next major technological trend is probably going to be related to our controlling the planet's environment in the same way that clothing and shelter helped us deal with the fickle weather. 

To continue our progress, we'll need to make sure that the continual ice ages and heat waves don't throw us drastically off course. Any ability we gain that can help there is a technology by my earlier definition. 

As well, the space available on our planet is finite. 

Navigating outer space seemed easy in the science fiction world of last century, but in practice it does appear to be well beyond our current technological sophistication. 

We don't even have a clue how to create the base technologies like warp drives or anti-gravity, let alone keep a huge whack of complicated stuff like a space shuttle running reliably. 

We're talking a lot about space exploration but our current progress is more akin to our ancestors shoving out logs into the ocean to see if they float. It's a long way from there to their later mastery of crafting sailing ships and another massive leap to our state-of-the-art cruise liners. Between all of those is obviously a huge gulf, and one that we need to fill with many new technologies, great and small.

Given our short life spans, we have a tendency to put on blinders and look at the progress of the world across just a few decades. That incredibly tiny time horizon doesn't really do a fair job in laying our the importance, or lack of importance, in what is happening with us as a species. 

We're on a long-term trajectory to somewhere unknown, but we certainly have been acquiring lots of sporatic knowledge about where we have come from. 

Of course it will take generations to peice it all together and further generations to consolidate it into something rational, but we in our time period at least get to see the essence of where we have been and where we need to go. 

Our vehicles for getting there are the technologies that we have been acquiring over millennia. They are far from complete, far from well understood, but we should have faith that they form the core of our intellectual progress. 

They map out the many paths we have been taking. 

Technology is the manifestation of us applying our intellect, which is the current course set by evolution. It tried big and powerful, but failing that it is now trying 'dynamic'; an ability to adapt to one's surrounding much faster than gradual mutations ever could. 

Recycling

I was chatting with a friend the other day. We're both babysitting large systems (>350,000 lines) that have been developed by many many programmers over years. Large, disorganized mobs of programmers tend towards creating rather sporadic messes, with each new contributor going further off in their own unique direction as the choas ensues. As such, debugging even simple problems in that sort of wreckage is not unlike trying to make sense of a novel where everyone wrote their own paragraphs, in their own unique voice, with different tenses and with different character names, and now all the paragraphs are hopelessly intertwined into what is supposedly a story. Basically flipping between the different many approaches is intensely headache inducing even if they are somehow related.

Way back in late 2008, I did some writing about the idea of normalizing code in the same way that we normalize relational database schemas:

http://theprogrammersparadox.blogspot.com/2008/10/structure-of-elegance.html

http://theprogrammersparadox.blogspot.com/2008/10/revisiting-structure-of-elegance.html

http://theprogrammersparadox.blogspot.com/2008/11/code-normal-form.html

There are a few other discussions and papers out there, but the idea was never popular. That's strange given that being able to normalize a decrepit code base would be a huge boon to people like my friend and myself that have found ourselves stuck with somebody else's short sightedness.

What we could really use to make our lives better is a way to feed the whole source clump into an engine that will non-destructively clean it up in a consistent manner. It doesn't really matter how long it takes, it could be a week or even a month, just so long as in the end, the behaviour hasn't gotten worse and the code is now well-organized. It doesn't even matter anymore how much disk space it uses, just that it gets refactored nicely. Not just the style and formatting, but also the structure and perhaps the naming as well. If one could create a high-level intermediate representation around the statements, branches and loops, in the same way that symbolic algebra calculators like Maple and Mathematica manipulate mathematics, then it would just be straight forward processing to push and pull the lines matching any normalizing or simplification rule. 

Picking between the many names for variables holding the same type or instance of data would require stopping for human intervention, but that interactive phase would be far less time consuming than refactoring by hand or even with the current tool set that is available in most IDEs. And a reasonable structural representation would allow identifying not only duplicate code, but also code that was structurally similar yet contained a few different hard-coded parameters. That second case opens the door to automated generalization, which given most code out there, would be a huge boost in drastically reducing the code size. 

One could even apply meta-compiler type ideas to use the whole infrastructure to convert easily between languages. The code to intermediate representation could be split away from the representation to code part. That second half could be supplied with any number of processing rules and modern style guides so that most programmers who follow well-known styles could easily work on the revised anchient code base. 

Of course another benefit is that once the code was cleaned up, many bugs would become obvious. Non-symetric resource handling for instance, is a good example. If the code grabbed a resource but never released it, that might have been previously buried in speghetti, but once normalized it would be a glaring flaw. Threading problems would also be brought quickly to the surface.

This of course leads to the idea of code recycling. Why belt out new bug-riddled code, when this type of technology would allow us to reclaim past efforts without the drudery of having to unravel their mysteries? 

A smart normalizer might even leverage the structural understanding to effectively apply higher level concepts like design patterns. That's possible in that functions, methods, objects, etc. are in their essence just ways to slice and dice the endless series of instructions that we need to supply. With structure, we can shift the likely DAG-based representations around, changing where and how we insert those meta-level markers. We could even extract large computations buried with global variables into self-standing stateless engines. Just that capability alone would turbo charge many large projects.

With enough computing time -- and we have that -- we could even compute all of the 'data paths' through the code that would show how basically the same underlying data is broken apart, copied and recombined many times over, which is always an easy and early target when trying to optimize code. Once the intermediate representation is known and understood, the possibilities are endless.

There are at least trillions of lines of code out there, much of which has been decently vetted for usability. That stuff rusts constantly and our industry has shied away from learning to really utilize it. Instead, year after year, decade after decade, each new wave of programmers happily rewrites what has already been done hundreds of times before. Sure it's easier, and it's fun to solve the simple problems, but we're really not making any real progress by working this way. Computers are stupid, but they are patient and quite powerful, so it seems rather shortsighted for us to not be trying to leverage this for our own development work in the same way that we try to do it for our users. Code bases don't have to be stupid and ugly anymore, we have the resources now to change that, all we need to do is just put together what we know into a working set of tools. It's probably about time that we stop solving CS 101 problems and moved on to the more interesting stuff.

Requirements

The idea of specifying programs by defining a set of requirements goes way way back. I saw a reference from 1979, but it is probably a lot earlier. Requirements are the output from the analysis of a problem. They outline the boundaries of the solution. I've seen many different variations on them, from very formal to quite relaxed.

Most people focus on direct stakeholder requirements; those from the users, their management and the people paying the bill for the project. These may frame the usage of the system appropriately, but if taken as a full specification they can lead to rampant technological debt. The reason for this is that there are also implicit operational and development requirements that although unsaid, are necessary to maintain a stable and usable system. You can take shortcuts to avoid the work initially, but it always comes back to haunt the the project.

For this post I'll list out some of the general requirements that I know, and the importance of them. These don't really change from project to project, or for domain. I'll write them in rather absolute language, but some of these requirements are what I would call the gold or platinem versions, that is they are above the least-acceptable bar.

Operational Requirements

Software must be easily installable. If there is an existing version, then it must be upgradeable in a way that retains the existing data and configuration. The installation or upgrade should consist of a reasonably small number of very simple steps and any information that the install needs that can be obtained from the environment should automatically be filled in. In-house systems might choose to not to do a fresh install, but if they do, then they need another mechanism for setting up and synchronizing test versions. The installation should be repeatable and any upgrade should have a way to rollback to the earlier version. Most installs should support having multiple instances/version on the same machine, since that helps with deployments, demos, etc..

Having the ability to easily install or upgrade provides operations and the developers with ability to deal with problems and testing issues. If there is a huge amount of tech debt standing in the way, the limits force people into further destructive short-cuts. That is they don't set things up properly, so they get caught by surprise when the testing fails to catch what should have been an obvious problem. Since full installations occur so infrequently, people feel this is a great place to save time.

Software should handle all real world behaviours, it should not assume a perfect world. Internally it should expect and handle every type of error that can be generated from any sub-components or shared resources. If it is possible for the error to be generated, then there should be some consideration on how to handle that error properly. Across the system, error handling should be consistent, that is if one part of the system handles the error in a specific way, all parts should handle it in the same way. If there is a problem that affects the users, it should be possible for them to work around the issue, so if some specific functionality is unavailable, the whole system shouldn't be down as well. For any and all shared resources, the system should use the minimal amount and that usage should be understood and monitored/profiled before the software is considered releasable. That includes both memory and CPU usage, but it also exists for resources like databases, network communications and file systems. Growth should be understood and it should be in line with reasonable operational growth. 

Many systems out there work fine when everything is perfect, but are down right ornery when there are any bumps, even if they are minor. Getting the code to work on a good day is only a small part of writing a system. Getting it to fail gracefully is much harder, but often ignored as a invalid shortcut.

When a problem with software does occur, there should be sufficient information generated to properly diagnose the problem and zero in on a small part of the code base. If the problem is ongoing, there should not be too much information generated, it should not eat up the file system or hurt the network. It should just provide a reasonable amount of information initially and then advise on the ongoing state of the problem at reasonable intervals. Once the problem has been corrected it should be automatic or at least easy to get back to normal functionality. It should not require any significant effort.

Quirky systems often log excessively, which defeats the purpose of having a log since no one can monitor it properly. Logs are just another interface for the operations personal and programmers so they should be treated nicely. Some systems require extensive fiddling after an error to reset poorly written internal states. None of this is necessary and just adds an extra set of problems after the first one. It is important not to over-complicate the operational aspects by not addressing their existence or frequency.

If there is some major, unexpected problems then there should be a defined way of getting back to full functionality. A complete reboot of the machine should always work properly, there may also be faster, less sever options such as just restarting specific processes. Any of these corrective actions should be simple, well-tested and trustworthy, since they may be choosen in less than ideal circumstances. They should not make the problems worse, even if they do not fix them outright. As well, it should be possible to easily change any configuration and then do a full reboot to insure that that configuration is utilized. There may be less sever options, but again there should always be one big, single, easy route to getting everything back to a working state.

It is amazing how fiddly and difficult many of the systems out there are right now. Either correcting them wasn't considered or the approach was not focused. In the middle of a problem, there should always be a reliable hail mary pass that is tested and ready to be employed. If it is done early and tested occasionally, it is always there to provide operational confidence. Nothing is worse than a major problem being unintentionally followed by a long series of minor ones.

Development Requirements

Source code control systems have matured to the point where they are mandatory for any professional software development. They can be centralized or distributed, but they all provide strong tracking and organizational features such that they can be used to diagnose programming or procedural problems at very low cost. All independent components of a software system must also have a unique version number for every instance that has been released. The number should be easily identifiable at runtime and should be included with any and all diagnostic information.

When things are going well, source repos don't take much extra effort to use properly. When things are broken they are invaluable at pinpointing the source of the problems. They also work as implicit documentation and can help with understanding historic decisions. It would be crazy to build anything non-trivial without using one.

A software system needs to be organized in a manner that encapsulates its sub-parts into pieces that can be used to control the scope for testing and changes. All of the related code, configurations and static data must be placed together in a specific location that is consistent with the organization of the rest of the system. Changes to the system are scoped and the minimal number of tests is completed to verify their correctness. Consistency is required to allow programmers the ability to infer system wide    behavior from specific sub-sections of code.

The constant rate of change for hardware and software is sufficiently fast enough that any existing system that is no longer in active development starts to 'rust'. That is, after some number of years it has slipped so far behind its underlying technologies that it becomes nearly impossible to upgrade. As such, any system in active operation also needs to maintain some development effort as well. It doesn't have to be major extensions, but it always necessary to keep moving forward on the versions. Because of this it is important that a system be well-organized so that at very least, any changes to a sub-part of a system can be completed with the confidence that it won't affect the whole. This effectly encapsulates the related complexities away from the rest of the code. This allows any change to be correctly scoped so that the whole system does not need a full regression test. In this manner, it minimizes any ongoing work to a sub-part of the system. The core of a software architecture is this necessary system-wide organization. Beyond just rusting, most systems get built under sever enough time constraits that it takes a very long time and a large number of releases before the full breath of functionality has been implemented. This means that there are ongoing efforts to extend the functionality. Having a solid architecture reduces the amount of work required to extend the system and provides the means to limit the amount of testing necessary to validate the changes.

Finally

There are lots more implicit non-user requirements, but these are the ones that I commonly see violated on a regular basis. With continuously decreasing time expectations, it is understandable why so many people are looking for shortcuts, but these never come for free so there are always consequences that appear later. If these implicit requirements are correctly accounted for in the specifications of the system, the technical debt is contained so that the operations and ongoing development of the system is minimized. If these requirements are ignored, ever-increasing hurdles get introduced which compromise the ability to correctly mange the system and make it significantly harder to correct later.

Predictions

Since I've been following it, hardware innovations have always been the leading indicator for software direction. 

Mainframes were followed by minis, then micros, networks and servers, and now smart phones and tablets. Each new wave of hardware spawned in software another set of waves a few years later such as operating systems, better languages, data structures, GUIs, networks, libraries, client-server, object-oriented, the web, NUIs, distributed systems and mobile apps. These came as a response to the increased capability of the new hardware. Since it was all new territory, most of these new software technologies were redone from scratch. In this fashion we've built up billions of lines of code, in distinct multiple generations, managed by millions of programmers. 

In a very general sense, software has never had the chance to really mature because adapting to the next wave of hardware took precedence. This dates way back beyond me and might come loosely under the category of "the software crisis" from the 60s. An ongoing epidemic where each new generation of programmers ignores the knowledge of the previous ones because they were trying to keep up with the new demands brought on by advances in hardware. We don't get caught up in fixing the existing problems, we just move on to the next ones. 

The Internet of Things promises to continue this trend for yet another round. They'll be another huge wave of small hardware devices, followed by billions of lines of code to try and do something useful with all of these interfaces and the data collected.

The 'crisis' part of this evolution is that although software has innovated to catch up, the general quality of it has been sacrificed to keep pace. A big change in approach like Object Oriented programming did open the doors for us to build considerably larger systems, but each new generation compensated for the increased complexity by reducing aspects of the quality. 

There are of course, lots of counter examples of great modern software that is extremely well-built, but considering the billions of lines of code out there today, they just don't count for a sizeable volume. Most code is fragile. More importantly, our critical data is kept on or processed by older, more reliable technologies like mainframes, COBOL or even Fortran. That's partly because they are better debugged by now, but also because they were better engineered as well. Cruder hardware forced more concentration on the solutions, which resulted in the software being smaller and dependable. More thought went into the work, because there was more time available for it.

All of the new hardware coming out for the Internet of Things will once again hide this basic problem. Data was initially generated mostly by people, so it was small, prone to error and needed extensive editing. But it was highly structured and deep. With hardware generating the data the opposite will be true. It results in a mass amount of data that is extremely accurate, but also very shallow. Mostly it will be endless time-series of specific points in the real world, like a living room lamp. We'll get great volumes of stuff like ON 3:46pm, OFF 4:23pm, ON 7:23pm, and so on. In a sense, it is data that just 'outlines' some base information like the fact that someone came home at quarter to four for a while and then returned later for dinner. It doesn't say that directly, but it can be inferred.

The rush to utilize this new hardware input will once again obscure the fundamental problems with software development. We'll get an app to flicker the lights on and off for example, but beyond the basic tricks, the software side will only marginally improve our lives at the cost of making them once again more complex. Now to get the lights, you'll have to fumble around to find that lamp app buried within dozens of other ones. 

In fact this has been the ongoing trend for software since the begining. Computers haven't increased the 'quality' of our lives as much as they've could have and at the same time they've allowed us to amplify the complexities of our societies to a dangerous level. 

This is best seen in all of the security problems we've been having lately. It has become far too easy for people to get important data on anything they want, and it is frequently the case that they choose to use this data for means that are not good for either individuals or our societies. Identity theft, credit card fraud and illegal spying are all consequences of the software industries inability to reliably create large software systems. Since security isn't visible to the consumers, it is one of those isses that can be left for later.

Added to that, there are great questions that we could answer now about various behaviours of 'things' in our societies. We've collected the data, it is stored, but unfortunately it spans so many repositories that it is essentially unmineable. We converged on this 'siloed' version of our collective data because the problems we could have tackled are just too large for any single individual to comprehend them, and there hasn't been any market or financial benefits for organizations or individuals working together in the software industry to solve this. Thus we have very few standards, many many exceptions and great walls between our data.

With more hardware on the way, the industry will obviously continue on its current trajectory. We'll just build more hardcoded ways to view the new data, and gradually combine little bits of it together, here and there. The new features will keep people amused long enough that they forget about all of the underlying bad things that are happening. That will last us at least another decade at its present rate, but what then? What do we do when we have trillions of lines of code, erratically spread all over, generating data that makes petabytes look tiny?

At some point, hardware just isn't going to be able to cover up the sins of the software industry anymore. Software will start to change very slowly and there will be a great outcry for it to start living up to its promise. Of course by then, the collective mess that we've built will be well beyond any individual or group's comprehension. And worse, it will be nearly locked in stone because we'll not have the resources to chase through the tangled webs to unravel the fundumental problems. Still, once progress is no longer apparent, there will finally be a demand to correct what was so hastily thrown together.

I'd guess what happens at that point, but I think many people are just hoping we invent artificial intelligences by then and make it their problem. They'll have the patience to slowly unwind what we've built up over decades of haste. With a bit of luck we can avoid the consequences of our immaturity.

We could, of course, start working on research to find a solution to our growing problem. Most likely it would take decades and involve going all of the way back to the basic models of computation. There are a few groups out there with this type of goal, but there seems to be very little funding available to really put significant effort into it. It's not short-term enough to interest investors and if something great were discovered it wouldn't achieve widespread usage unless it was essentially 'shared' with the rest of the world. Those qualities are generally bad for profits. It would also be too long, deep and slow to make it a viable goal for movements like OpenSource. Any notoriety or payoff would come so late that interest would have long since waned. People would just give up and move onto something more topical. 

With all of that in mind, what we'll most likely see over the next decade is a new round of technology driven by the increase in hardware meant to capitalize on the Internet of Things wave. It will probably center around being distributed, and will need to handle large volumes of simple data. These waves usually come with a new round of languages, so any of the more interesting distributed ones might grow in popularity but it is more likely that the next set will be loosely based on what is out there right now. Like all of the other waves, it will ignore the ongoing problems with version changes and it may even degrade the representational quality of the data itself to save space, time or both. It will be harder to install, and it will inadvertently take more effort to keep it constantly upgraded, both these attributes have been getting worse lately. Of course its security will be abysmal and any serious logging or auditing will be absent (sacrificed because of space). Still people will flock to it, and the next generation of coders will pronounce the rest of us out-of-touch with modern software for not degrading our practices enough to keep up.

History will repeat itself at least once more. Some trends are just too strong to escape. On a relate note, if anyone has millions of spare dollars and a hankering to finally break this cycle, I've got enough crazy ideas to last decades...

Architecture

Writing a small program, a few thousand lines, is not particularly difficult once you've mastered logic and algorithms. You focus on a very specific part of the problem, work out the details and then pound it into shape. As the size of the program grows you might keep on with the same approach; decompose the problem into smaller ones, then dump each answer somewhere arbitrary in the code base. At some point however, the amount of code that is built up begins to become the problem in itself. 

With a small program, if you make some sloppy mistakes it is fairly easy to find and fix. A couple of days at most. So the code can be disorganized. It can be highly redundant. Each instance of the same solution can have its own unique approach. You can get away with a lot of really bad practices. It can be a mess.

As the code base gets larger, the cost of changing it grows -- probably exponential -- until it is a huge task. Somewhere around the medium range, say 40,000 to 60,000 lines of code, just dumping in code anywhere builds up enough disorganization that it hinders further development. It only gets worse as more code is added. A few hundred thousand disorganized lines is painful, millions of lines of that is totally unmanageable.

Disorganized code can make it very hard to correct problems. Finding a bug in a small program isn't too hard once you've mastered debugging, but in a big code base organization is required to let you quickly jump to the right area. Without organization, it could take weeks, months or even years to find the problem, you have to search everywhere. 

When there is enough functionality and enough complexity, fixing the problems is a mess too. It starts to resemble stuffing straw into an old sack. You stuff it in one hole, only to watch it fall out from others. The changes cause unexpected side effects, which cause more changes, then more side-effects, etc. It becomes an endless and mostly hopeless task made worse by impending deadlines. 

A big mess can also be hard to extend. You don't want to disturb the existing functionality, so you copy and paste bits of code from all over into a new location, but the new redundacies will start to contridict each other quickly causing their own special bugs. Instead of making things better, you are probably just making them worse.

Big disorganized code bases are unstable and suffer from all sorts of horrible operational issues and usually have annoying interface problems, not to mention endless security issues. In most cases they look as ugly from the outside as they do internally. 

As the size of any system grows, the organization of the code quickly becomes the most significant problem in the development. Fortunately there is a well-known solution.

Architecture, for software, is a combination of the underlying technology choices and the higher level organization that encapsulates the many peices of a big system from each other. Mostly, due to resource limitations, it is arrived at through a long-term vision that is then mediated via compromises and politics. It isn't necessary for small programs, but from medium to large and all the way through to huge programs, it becomes increasingly significant to the stability, quality and extendability of the system. 

A good architecture draws 'lines' through the code, providing the high level encapsulation between components. It makes it easy to attribute bugs to specific parts and it allows one to place a scope on the impact of any code changes. As well it defines exactly where common extentions should be made to the code and lays out a direction for how to add brand new peices. 

An architecture can be documented, but if the lines are clear and consistent in the code base an experienced programmer can pick it up directly from reading the source. It might help to understand why things are organized in a particular manner, but not knowing that doesn't prevent someone from following the pattern of the existing implementation. 

One way to know there is an good architecture in place is to guess where some existing functionality should be, then verify that it is actually there and only there. If that is true for most of the functionality, then it is well-organized in a manner that can be understood.

Besides all its other qualities, a good architecture makes it easier for refactoring and for code reuse. If you need to make changes there are a limited number of places to change, and they are all in a consistent location close together. This allows you to either enhance the behaviour across the system, or route new functionality through one single interface. This of course provides a strategy for building a massive system over years and years through an endless huge number of versions. Each new iteration usually involves first refactoring things, then extending them with consitent organization. Done well, the first part can even be non-destructive and quickly testable, saving time and increasing the confidence level for the new code. 

One frequent concern with extending an architecture in this manner is the possibility of creating too many intermediate layers, but again the depth of the layers is an organizational issue itself. And endless number is disorganized by definition, so this meta-problem needs similar treatment to the base organization. In fact, for millions of lines of code, this sort of upper level organizational issue repeats itself over and over again, getting a bit higher each time. Finding a reasonable multi-level organization for millions of anything is of course a non-trivial problem that lies at the heart of architecure.

Small programs don't need an architecture. It doesn't hurt, but it is not necessary. Big programs abolosutely must have one that is clean and consistent. Otherwise they become unmanageable balls of mud. Architecture is a different set of skills than programming, but an architect should know both. They should have a lot of experience with programming first. They also need to be able to grasp the larger context and help focus the work of the programmers to avoid re-implementations. Efficiency for ongoing development is driven directly by the organization of the code base. Extending a well-architected program is considerable faster than just tacking on stuff to a ball of mud. The coding may take longer, but all of the other costs are minimized. The cost of not having an architecture is a steep decline in development speed over the lifetime of a project. That generally shows as either lower productivity or lower quailty or both.

If you want to design and build big, complex systems than the only way to get them to that size and keep them moving forward is to have and enforce an architecture. Disorganization may seem faster to some people, but that's only a limited short-term perspective.