In those days, for the emerging smaller hardware, there was not a lot of software available. It was slow and expensive to build anything. For any non-software company, they existed as brick-and-mortar businesses. Software could help automate isolated parts of the company, but that was it.
Mostly, even over some medium horizons, buying an existing software product was far cheaper than building it. And that software was usually run in an independent silo, with minimal integrations, so it wasn’t hard to get it up and running.
If there was already an available product, it didn’t make sense to build it. Buying it was the default choice.
But so much has changed since then...
The biggest change is that many companies now exist in the digital realm way more than the physical one. All of the sales, communications, and management for most lines of business happen digitally. Many of the products and services are still physical, but overall most lines of business are a mix now.
Running a computer is also more complicated. In my youth, you would set up a server room with suitable power, cooling, and network connections. When you didn't have the space you would lease it. But the drop in hardware price caused an explosion, so the number of machines involved these days is astronomical (and often unnecessary).
This made operations chaotic and undesirable, so most software products are a service now. Someone else sets it up and runs it for you. It’s easier to get going, but you don’t have as much control over it.
With the increase in digital presence came a huge need for integrations. There are way more silos now, often hundreds or even thousands of them. There are specialist silos for every subproblem. When the silos were independent, integrations were rare. But now they all need each other's data; dependencies are necessary. So everything needs to be integrated into almost everything else.
When you bought software before, you could get some new hardware to host it, spin it up, and test it out. If it mostly worked as expected it went live. But these days, just running a new system at a vendor's site isn’t that useful. Being trapped in a silo cripples it. It isn’t really live until all of the major integrations are done. Silos made sense before, but they are a hazard now.
It is not in any vendor's best interest to standardize their software. It is a simple calculation. If it is standard, then it is nearly trivial for any customer to switch to someone else. If you run into any glitches and all of the customers leave, you are instantly done. So don’t use standards.
Integrating various non-standard SaaS silos with each other is an epic nightmare. The easiest way to do it is to copy the data everywhere. If you have a dozen silos that need a particular type of data, you make a dozen copies. Then desperately try to keep them in sync somehow.
To make it even worse, each integration team and vendor will choose to model the data moving around differently. So, it is endlessly translated into different formats, and some of those translations will lose valuable parts of the information. That wastes a lot of time and causes all sorts of grief.
So modern integration projects have become huge, expensive, and tricky.
It's counterintuitive, as you think you managed to avoid programming by buying everything, but now you end up having to do way more glue programming in order to connect it all together.
And so much of that ETL code is awful. It was rushed into existence by people with too little experience. You end up with masses of hopelessly intertwined spaghetti and endless operational alerts about warnings and errors, most of which are unfortunately ignored.
And that is the crux of the issue. If you buy everything now, then you’ll get lost while trying to get it to all work together properly, and this is a lot more costly than just having built it properly in the first place.
Some things you don’t want to build though. Sometimes because it's huge, but more often because it is so complex that the devs require specific knowledge and experience to build it reasonably. You can't just hire a whole bunch of kids, they’ll conjure up a mess instead of what you need, it won’t help.
For any group of programmers, there are absolute limits to what they can build. They rarely are self-aware of their own limits, but things won’t go well if you let them stray too far past their abilities.
You can assemble a strong group of developers to build exactly what you need, but if the work dries up they will dissipate and you will run into trouble keeping it functioning later. To keep good developers they need to always have good projects to work on.
Which is to say that if you need to build software, you need to set up a stable ‘dev shop’ with enough capacity to turn out and enhance the types of software you need. The dev shop is what you need to focus on. It should be able to attract new talent, and to always have enough interesting work to keep everyone motivated. Talent attracts talent, so if you get a couple of strong devs, you can grow the affair. You just have to make sure the environment stays reasonable.
If you do that, it fundamentally changes the original buy vs build dynamics.
You want to keep building enough stuff to ensure that the dev shop stays functional. Building, if you have the capacity and ability would now be the first choice. It is a longer-term goal though, as you don’t want all of your good developers to leave.
Then you want to build up and out from a few different starting points. The guidance is to minimize integrations first. They are ultimately more costly than the vendors, so you focus there.
You figure out which categories your shop can handle, then you consolidate all of the little fragmented silos into larger systems. Generalization is the key to keeping the costs in line. Software companies leverage their code for lots of companies; in-house projects need to leverage their code for lots of different problems.
The focus is not on speed though, rather it is on doing the best engineering that you can. Move slowly and carefully. Build up as much reusable kit as you can, model the data as properly as you can, and keep expanding out from the starting point slowly swallowing dozens of other products. But always keeping a close eye on both the dev shop and the operational capacity.
Obviously, the digital parts of existing lines of business would be first. You’d want to do this anyway, since just using the same vendors as everyone else has no competitive advantages. But to get those advantages back, the work you do has to be better and more relevant than the vendors, which means that you have to have strong technologists who really understand the business too.
Then funding isn’t by project, line of business, or budget. It is by dev shop. You set a solid foundation and build up capacity to implement better stuff and keep the funding stable as you grow it. A large organization can have a few different dev shops.
Outside of those areas of software growth, the old buy vs. build choice would remain, but as the starting points get larger they would end up eating stuff around the fringes so you’d need to factor that in as well.
The counterargument to all of this is that building software is seen as outside of the company’s vertical. But the modern reality is that as most businesses got deeper into the digital realm, they drifted ever closer to being applied software companies, than their original lines of business.
The classic examples are Google, a marketing company, and Amazon, a retailer. As applied software companies they thrived while their brick-and-mortar predecessors didn’t.
The general nature though is that software is not a vertical for any digital line of business, it is a part of it. Core. That, and the exploding integration costs means that reasonable software is necessary for scaling and stabilizing. Bad software makes everything unprofitable.
As software eats the world, this same fate will play out in all sorts of other industries, and the winning strategy is to accept that if you are heavily reliant on doing business in the digital realm, then you are already heavily reliant on building software.
Then it is far more effective to build some of the core stuff yourself, instead of just integrating generic vendor products. You’ll need to recruit stronger developers and make sure you can keep them, but if you do your capacity will grow. Then software development capability itself becomes a competitive advantage.
Mostly, even over some medium horizons, buying an existing software product was far cheaper than building it. And that software was usually run in an independent silo, with minimal integrations, so it wasn’t hard to get it up and running.
If there was already an available product, it didn’t make sense to build it. Buying it was the default choice.
But so much has changed since then...
The biggest change is that many companies now exist in the digital realm way more than the physical one. All of the sales, communications, and management for most lines of business happen digitally. Many of the products and services are still physical, but overall most lines of business are a mix now.
Running a computer is also more complicated. In my youth, you would set up a server room with suitable power, cooling, and network connections. When you didn't have the space you would lease it. But the drop in hardware price caused an explosion, so the number of machines involved these days is astronomical (and often unnecessary).
This made operations chaotic and undesirable, so most software products are a service now. Someone else sets it up and runs it for you. It’s easier to get going, but you don’t have as much control over it.
With the increase in digital presence came a huge need for integrations. There are way more silos now, often hundreds or even thousands of them. There are specialist silos for every subproblem. When the silos were independent, integrations were rare. But now they all need each other's data; dependencies are necessary. So everything needs to be integrated into almost everything else.
When you bought software before, you could get some new hardware to host it, spin it up, and test it out. If it mostly worked as expected it went live. But these days, just running a new system at a vendor's site isn’t that useful. Being trapped in a silo cripples it. It isn’t really live until all of the major integrations are done. Silos made sense before, but they are a hazard now.
It is not in any vendor's best interest to standardize their software. It is a simple calculation. If it is standard, then it is nearly trivial for any customer to switch to someone else. If you run into any glitches and all of the customers leave, you are instantly done. So don’t use standards.
Integrating various non-standard SaaS silos with each other is an epic nightmare. The easiest way to do it is to copy the data everywhere. If you have a dozen silos that need a particular type of data, you make a dozen copies. Then desperately try to keep them in sync somehow.
To make it even worse, each integration team and vendor will choose to model the data moving around differently. So, it is endlessly translated into different formats, and some of those translations will lose valuable parts of the information. That wastes a lot of time and causes all sorts of grief.
So modern integration projects have become huge, expensive, and tricky.
It's counterintuitive, as you think you managed to avoid programming by buying everything, but now you end up having to do way more glue programming in order to connect it all together.
And so much of that ETL code is awful. It was rushed into existence by people with too little experience. You end up with masses of hopelessly intertwined spaghetti and endless operational alerts about warnings and errors, most of which are unfortunately ignored.
And that is the crux of the issue. If you buy everything now, then you’ll get lost while trying to get it to all work together properly, and this is a lot more costly than just having built it properly in the first place.
Some things you don’t want to build though. Sometimes because it's huge, but more often because it is so complex that the devs require specific knowledge and experience to build it reasonably. You can't just hire a whole bunch of kids, they’ll conjure up a mess instead of what you need, it won’t help.
For any group of programmers, there are absolute limits to what they can build. They rarely are self-aware of their own limits, but things won’t go well if you let them stray too far past their abilities.
You can assemble a strong group of developers to build exactly what you need, but if the work dries up they will dissipate and you will run into trouble keeping it functioning later. To keep good developers they need to always have good projects to work on.
Which is to say that if you need to build software, you need to set up a stable ‘dev shop’ with enough capacity to turn out and enhance the types of software you need. The dev shop is what you need to focus on. It should be able to attract new talent, and to always have enough interesting work to keep everyone motivated. Talent attracts talent, so if you get a couple of strong devs, you can grow the affair. You just have to make sure the environment stays reasonable.
If you do that, it fundamentally changes the original buy vs build dynamics.
You want to keep building enough stuff to ensure that the dev shop stays functional. Building, if you have the capacity and ability would now be the first choice. It is a longer-term goal though, as you don’t want all of your good developers to leave.
Then you want to build up and out from a few different starting points. The guidance is to minimize integrations first. They are ultimately more costly than the vendors, so you focus there.
You figure out which categories your shop can handle, then you consolidate all of the little fragmented silos into larger systems. Generalization is the key to keeping the costs in line. Software companies leverage their code for lots of companies; in-house projects need to leverage their code for lots of different problems.
The focus is not on speed though, rather it is on doing the best engineering that you can. Move slowly and carefully. Build up as much reusable kit as you can, model the data as properly as you can, and keep expanding out from the starting point slowly swallowing dozens of other products. But always keeping a close eye on both the dev shop and the operational capacity.
Obviously, the digital parts of existing lines of business would be first. You’d want to do this anyway, since just using the same vendors as everyone else has no competitive advantages. But to get those advantages back, the work you do has to be better and more relevant than the vendors, which means that you have to have strong technologists who really understand the business too.
Then funding isn’t by project, line of business, or budget. It is by dev shop. You set a solid foundation and build up capacity to implement better stuff and keep the funding stable as you grow it. A large organization can have a few different dev shops.
Outside of those areas of software growth, the old buy vs. build choice would remain, but as the starting points get larger they would end up eating stuff around the fringes so you’d need to factor that in as well.
The counterargument to all of this is that building software is seen as outside of the company’s vertical. But the modern reality is that as most businesses got deeper into the digital realm, they drifted ever closer to being applied software companies, than their original lines of business.
The classic examples are Google, a marketing company, and Amazon, a retailer. As applied software companies they thrived while their brick-and-mortar predecessors didn’t.
The general nature though is that software is not a vertical for any digital line of business, it is a part of it. Core. That, and the exploding integration costs means that reasonable software is necessary for scaling and stabilizing. Bad software makes everything unprofitable.
As software eats the world, this same fate will play out in all sorts of other industries, and the winning strategy is to accept that if you are heavily reliant on doing business in the digital realm, then you are already heavily reliant on building software.
Then it is far more effective to build some of the core stuff yourself, instead of just integrating generic vendor products. You’ll need to recruit stronger developers and make sure you can keep them, but if you do your capacity will grow. Then software development capability itself becomes a competitive advantage.
No comments:
Post a Comment
Thanks for the Feedback!