Is Open Source hurting Consulting?

In an earlier blog entry entitled Is Open Source hurting Computer Science? I looked into whether the glut of Open Source code was causing problems for software developers. In the comments, Alastair Revell correctly pointed out that my arguments were biased towards shrink-wrapped software. While that represents a significant chunk of the software market, it is by no means the majority of development.

As I was pondering how to respond, I realized that the circumstances for in-house development and consulting were far more complex than for packaged software. It was a dense enough subject to make for a good solid blog entry; a chance that shouldn't be wasted.

We can start by noting that there is a trend away from in-house development. That's not a surprise. For decades the software industry has been selling companies on developing their own custom solutions, only to have the majority of the projects fail; this has left many organizations weary when it comes to development. It is expensive and prone to failure.

I believe that it was Frederick P. Brooks who predicted in the Mythical Man Month that shrink-wrapped software would replace in-house development, arguing that it would bring the costs down. While he was right about moving away from internal projects, it does seem as if consulting has been the big winner, not commercial packages.

This shouldn't come as a surprise; most companies still feel that a unique process is a competitive advantage, and since software implements that process it too must be unique. The extra costs are worth it.

With the diminishing of in-house development, the key focus for this entry is on how significant volumes of free Open Source software are effecting the consulting industry. For that we need to first go back in time a bit.

Consulting has seen significant changes in the last couple of decades. At the end of the dot-com boom, staff was so hard to find that many programmers cashed in by becoming independent consultants. Staffing big projects was extremely difficult, so the rates were incredibly high. With the bust, the jobs disappeared and left a lot of people scrambling for work, often forced to finding it outside of software. In the turmoil, most of the independents and small companies disappeared; leaving an industry dominated only by very large companies.

It is at this point that we need to examine the rise of Open Source code. In a market of only big companies, what is the effect of lots of free code?

For a typical consulting project, access to large amounts of code would appear to be a good thing. Projects take forever, so it vital to get a quick version of the system up and running. This calms the fears and makes it easy to extend the project. A glut of code means that it is far easier to whack together a fast prototype and extend it to meet the requirements.

Quality problems with the code you might assume would be bad, but in fact extending the project beyond its original tenure is the life blood of consulting. An endless need for small fixes is the perfect situation. A smart lawyer and a polite client manager are all that is needed to turn a short-term implementation contract in to a long-term cash cow, milking it for all it is worth. Low quality means more work.

All this might be fine, if it were that individuals could startup consulting companies and gradually morph them into full fledge software companies. Fresh blood entering into the market helps keep the prices under control and drives innovation.

With all of the freely available code you'd think it should be easy for a small consultant to compete? However, the industry contains many significant obstacles.

Everybody wants the latest technology, a target that is changing too fast for most programmers to keep pace with. As well, many of the newer technologies are so eccentric that you need to have experts that specialize only in them in order to get them to work. A basic development project may cover ten or twenty such technologies; too much for any one person.

This means that the glut of code lends itself to being assembled by groups of developers not individuals. Given the basic instabilities of the software, projects made up entirely of independent consultants can be quite volatile. They need significant and patient management to keep them working together.

Past experience with software management drives many firms away from that, choosing instead, the big consulting companies that can design, manage, implement and often operate their projects for them. Of course big firms always have a preference for big "suppliers", making it hard for the little guy to get an opportunity. Little firms go for shrink-wrapped systems.

You'd think companies would have learned, after all many of them are stuck with these new, fancy, sexy but highly defective systems. These have been added to their roster along side of the fragile client/server systems of the nineties, and their ever reliable, but slow and hugely expensive mainframes; the real workhorses of most corporate IT. A trifecta of defective technologies.

CIOs in big companies are drawn toward the big consulting firms for their big new shiny projects. These consulting companies promise quick development and fast turn-around, but sadly the working problems linger on forever. Dragging out the contract is standard. It is just "another" period in the software industry where the big money users -- usually large corporations -- are getting shafted. Another of many that have already occurred.

With so much money to be made, consulting firms no doubt love Open Source. It and any quality problems it has only help the bottom line. Bugs are profitable for those that make money fixing them.

It would be no surprise then that consulting companies don't want to change the circumstances. Innovation does not help. It is good right now. With no new competition, this is unlikely to change until companies wise up.

In an off way this could also explain the increasing popularity of the newer lightweight methodologies. If you want fast results and low quality, then you certainly don't want a whole lot of process slowing things down and accidentally making it better do you? A methodology designed around slapping together quick and dirty prototypes with the least amount of planning fits well into the model of "built it quick and fix it forever". It is a scary thought.

In my previous blog entry I had reached a point where I felt that the glut of code was killing innovation and making it hard for small companies to get into the market. In this entry, I feel that the same glut also makes it hard for individual consultants and small consulting companies to get into the market, thus killing any competition or innovation. Also, related or not, in-house development is dying; it is gradually being replaced by external projects. As things are profitable right now, innovation is an unnecessary obstacle. Not unsurprisingly this implies that too much free code is causing significant problems in all corners of the the software industry. Too much of a good thing is never a good thing, it seems.