Trivial solutions are great for solving trivial problems. But if you have a hard problem, then no combined set of trivial solutions will ever end up correctly solving it.
The mechanics of this are fairly easy to understand.
If you try to solve only a small subset of a big problem with a solution that targets just that subset, then it will spin off more problems, it is a form of fragmentation.
You’ve addressed the subset, but now that needs to interact with many of the other parts, and those are artificial complexity. They would not have been necessary if you had addressed the whole problem.
If you try to solve a hard problem, either one that is huge or one that is complex, with a lot of trivial solutions, there will be an endless stream of these unsolved fragments, and as you try to solve these, they will make it all rather perpetual. You can’t get a perpetual motion machine in our physical universe, but you can effectively spend nearly forever creating and patching up little holes in a misfitting solution.
If you want to solve a hard problem, then the solution itself cannot be trivial. It will not be simple, it will not be easy. Any desire or quest to get around this is hopeless.
“Things should be made as simple as possible, but no simpler” -- possibly Albert Einstein
The really important part of that misattributed above quote is at the end. That there is a notion of too simple.
The belief that one can get away with over-simplifying solutions is the underlying cause of so many software problems. It’s nice when software is simple and easy to understand, but if it isn’t the right solution, then it will cause trouble.
Yes, you can whack out a large number of trivialized silos into software. You can get these into active usage really quickly. But if they do not fit properly, the net effect is now worse than before. You’ve created all sorts of other problems that will accumulate to be worse than the original one. The software isn’t really helping, it’s just distracting everyone from really fixing the problem.
This is quite common in a lot of large companies. They have an incredible number of systems that spend more of their resources pushing data back and forth, than they do working with their intended users. And the more they move data around, the worse it gets. The quality issues spin off secondary systems trying to fix those data problems, but by then it is all just artificial complexity. The real underlying problems have been lost.
If any of the silos aren’t fully encapsulated, then either the partitioning is wrong, or the problem is hard and can’t be split up.
Thursday, December 7, 2023
Thursday, November 30, 2023
Observable on Steriods
A while back I wrote a small product that, unfortunately, had an early demise. It was great technically -- everyone who saw it loved it -- it’s just that it didn’t get any financial support.
The idea was simple. What if all of the data in an application was observable? Internally it was just thrown into a giant pool of data. So, it could be variables in a model, calculations, state, context, etc. Everything is just observable data.
Then for each piece of derived data, it would watch any of its dependencies. If it was a formula, it would see that any of the underlying values changed, recalculate, and then tell anyone watching it that it is now different.
The stock way of implementation observable in an object-orient paradigm is to keep a list of anyone watching, then issue an event, aka function call, to each. The fun part is that since any object can watch any other object, this is not just a tree or a dag, it can be a complete graph.
Once you get a graph involved, any event percolating through it can get caught in a cycle. To avoid this, I traded off space by having each event keep a dictionary of the other objects it had already visited. If an object gets notified of an event and it is already in that visited set, it just ignores the event. Cycles defeated.
So, now we have this massive pool of interconnected variables, and if we dumped some data into the pool, it would set off a lot of events. The leaf variables would just be updated and notify watchers, but the ones above would recalculate and then notify their watchers.
I did do some stuff to consolidate events. I don’t remember exactly, but I think I’d turn on pause, update a large number of variables, then unpause. While paused, events would be noted, but not issued. For any calculation with a lot of dependents, if one thing changed, I’d track the time and since it has already recalced, grabbing all of the child variables data, it would toss any events for other dependents that were earlier. So, for A+B+C, after the unpause, you’d be notified that A changed, and do the recalc, but then the time that B and C changes would be earlier than the recalc, so ignored. That cut down significantly on any event spikes.
Then finally, at the top, I added a whole bunch of widgets. Each one was wired to watch a single variable in the pool. As data poured into the pool, the widgets would update automatically. I was able to wire in some animations too, so that if a widget displayed a number, whenever it changed it would flash white, and then slowly return to its original color. But then I wired in tables and plotting widgets as well. The plot, for instance, is a composite widget, whose children, the points of the curve, are then they are all wired to different variables in the pool. So, the plot would change on the fly as things changed in the pool.
Now if this sounds like MVC, it basically is. I just didn’t care if the widgets were in one view or a whole bunch of them. They’d all update correctly either way. and the entire model was in the pool. And any of the interface context stuff was in the model so in the pool. And any of the user’s settings were in the model, so in the pool. In fact, every variable, anywhere in the system, was in the model, so in the pool. Thus the steroids designation. Anything that can vary in the code is an object and that is observable in the pool.
Since I effectively had matrices and formulas, it was a superset of a spreadsheet. A sort of disconnected one. A whole lot more powerful.
Because time was limited, my version had to wire up each object as code explicitly. But they didn’t do much other than inherit the mechanics, define the children to watch, and provide a function to calculate. It would have been not too hard to make all of that dynamic, and then provide an interface to create and edit new objects in the app itself. This would let someone create new objects on the fly and arrange them as needed.
It was easy to hook it up to other stuff as well. There was a stream of incoming data, so it was handled by a simple mapping between the data’s parameters and the pool objects. Get the next record in the stream, and update the pool as necessary. Also, keyboard and button events would dump stuff directly into the pool. I think some of the widgets even had two-way bindings, so the underlying pool variable changed when the user changed the widget and could percolate to everything else. I had some half cycles, where the widget displayed a value, it was two-way bound and as the user changed it, it triggered other changes in the pool, which updated stuff on the fly which would change the widget. I used that for cross-widget validations as well. The widgets change the meta information for each other.
I did my favorite form of widget binding, which is by name. I could have easily added scope to that but the stuff I was working with was simple enough that I didn’t have any naming collisions. I’ve seen structural-based binding sometimes, but they can be painful and rigid. The pool has no structure and the namespace is tiny because the objects were effectively hardwired to their dependencies. Extending it would need scope.
To pull it all together I had dynamic forms and set them to handle any type of widget, primitive or composite. I pulled a trick from my earlier work and expanded the concept of forms to include everything on the screen including menus and other recursive forms. As well, forms could be set to not be editable, which lets one do new, edit, and view all with the same code, which helps to save coding time and enforce consistency.
Then I put in some extremely complex calculations, hooked it up to a real-time feed, and added a whole bunch of screens. You could page around while the stream was live and change stuff on the fly, while it was continuously updating. Good fun.
It’s too bad it didn’t survive. That type of engine has the power to avoid a lot of tedious wiring. Had I been allowed to continue I would have wired in an interpreter, a pool browser, and a sophisticated form creation tool. That and some way to dynamically wire in new data streams would have been enough to compete with tools like Excel. If you could whip up a new table and fill it with formulas and live forms, it would let you craft complex calculation apps quickly.
The idea was simple. What if all of the data in an application was observable? Internally it was just thrown into a giant pool of data. So, it could be variables in a model, calculations, state, context, etc. Everything is just observable data.
Then for each piece of derived data, it would watch any of its dependencies. If it was a formula, it would see that any of the underlying values changed, recalculate, and then tell anyone watching it that it is now different.
The stock way of implementation observable in an object-orient paradigm is to keep a list of anyone watching, then issue an event, aka function call, to each. The fun part is that since any object can watch any other object, this is not just a tree or a dag, it can be a complete graph.
Once you get a graph involved, any event percolating through it can get caught in a cycle. To avoid this, I traded off space by having each event keep a dictionary of the other objects it had already visited. If an object gets notified of an event and it is already in that visited set, it just ignores the event. Cycles defeated.
So, now we have this massive pool of interconnected variables, and if we dumped some data into the pool, it would set off a lot of events. The leaf variables would just be updated and notify watchers, but the ones above would recalculate and then notify their watchers.
I did do some stuff to consolidate events. I don’t remember exactly, but I think I’d turn on pause, update a large number of variables, then unpause. While paused, events would be noted, but not issued. For any calculation with a lot of dependents, if one thing changed, I’d track the time and since it has already recalced, grabbing all of the child variables data, it would toss any events for other dependents that were earlier. So, for A+B+C, after the unpause, you’d be notified that A changed, and do the recalc, but then the time that B and C changes would be earlier than the recalc, so ignored. That cut down significantly on any event spikes.
Then finally, at the top, I added a whole bunch of widgets. Each one was wired to watch a single variable in the pool. As data poured into the pool, the widgets would update automatically. I was able to wire in some animations too, so that if a widget displayed a number, whenever it changed it would flash white, and then slowly return to its original color. But then I wired in tables and plotting widgets as well. The plot, for instance, is a composite widget, whose children, the points of the curve, are then they are all wired to different variables in the pool. So, the plot would change on the fly as things changed in the pool.
Now if this sounds like MVC, it basically is. I just didn’t care if the widgets were in one view or a whole bunch of them. They’d all update correctly either way. and the entire model was in the pool. And any of the interface context stuff was in the model so in the pool. And any of the user’s settings were in the model, so in the pool. In fact, every variable, anywhere in the system, was in the model, so in the pool. Thus the steroids designation. Anything that can vary in the code is an object and that is observable in the pool.
Since I effectively had matrices and formulas, it was a superset of a spreadsheet. A sort of disconnected one. A whole lot more powerful.
Because time was limited, my version had to wire up each object as code explicitly. But they didn’t do much other than inherit the mechanics, define the children to watch, and provide a function to calculate. It would have been not too hard to make all of that dynamic, and then provide an interface to create and edit new objects in the app itself. This would let someone create new objects on the fly and arrange them as needed.
It was easy to hook it up to other stuff as well. There was a stream of incoming data, so it was handled by a simple mapping between the data’s parameters and the pool objects. Get the next record in the stream, and update the pool as necessary. Also, keyboard and button events would dump stuff directly into the pool. I think some of the widgets even had two-way bindings, so the underlying pool variable changed when the user changed the widget and could percolate to everything else. I had some half cycles, where the widget displayed a value, it was two-way bound and as the user changed it, it triggered other changes in the pool, which updated stuff on the fly which would change the widget. I used that for cross-widget validations as well. The widgets change the meta information for each other.
I did my favorite form of widget binding, which is by name. I could have easily added scope to that but the stuff I was working with was simple enough that I didn’t have any naming collisions. I’ve seen structural-based binding sometimes, but they can be painful and rigid. The pool has no structure and the namespace is tiny because the objects were effectively hardwired to their dependencies. Extending it would need scope.
To pull it all together I had dynamic forms and set them to handle any type of widget, primitive or composite. I pulled a trick from my earlier work and expanded the concept of forms to include everything on the screen including menus and other recursive forms. As well, forms could be set to not be editable, which lets one do new, edit, and view all with the same code, which helps to save coding time and enforce consistency.
Then I put in some extremely complex calculations, hooked it up to a real-time feed, and added a whole bunch of screens. You could page around while the stream was live and change stuff on the fly, while it was continuously updating. Good fun.
It’s too bad it didn’t survive. That type of engine has the power to avoid a lot of tedious wiring. Had I been allowed to continue I would have wired in an interpreter, a pool browser, and a sophisticated form creation tool. That and some way to dynamically wire in new data streams would have been enough to compete with tools like Excel. If you could whip up a new table and fill it with formulas and live forms, it would let you craft complex calculation apps quickly.
Thursday, November 23, 2023
Bottom Up
The way most people view software is from the top down. They see the GUI and other interfaces, but they don’t have any real sense of what lies behind it.
The way that bugs are reported is from the top down. Usually, there is an interface problem somewhere the underlying cause may be deep in the mechanics.
The foundation of every system is the data it persists. If there isn’t a way to reliably keep the data around for a long time, it might be cute, but it certainly isn’t practical.
The best way to build software is from the bottom up. You lay down a consistent set of behaviors and then you build up more complicated behaviors on top of those. This leverages the common lower stuff, you don’t have to reduplicate the work again for everything on top.
The best way to extend a system is bottom-up, You start with getting the data into persistence, then into the core of the system, and then you work your way upwards until all of the interfaces reflect the changes.
The way to deal with bugs is top-down. But the trick is to keep going as low as you can. Make the lowest fixed as time will allow. Sometimes it is rushed, so you might fix a bunch of higher-level symptoms, but even if you do that you still have to schedule the proper lower-level ones soon.
The best way you screw up a large system is to let it get disorganized. The way you organize a system is from the bottom up, Looking at it top down will only confuse you.
Some people only want to see it one way or the other, as top-down or bottom-up, but clearly, that isn’t possible. When a system becomes large, the clash in perspectives becomes the root of a lot of the problems. Going in the wrong direction, against the grain, will result in hardship.
The way that bugs are reported is from the top down. Usually, there is an interface problem somewhere the underlying cause may be deep in the mechanics.
The foundation of every system is the data it persists. If there isn’t a way to reliably keep the data around for a long time, it might be cute, but it certainly isn’t practical.
The best way to build software is from the bottom up. You lay down a consistent set of behaviors and then you build up more complicated behaviors on top of those. This leverages the common lower stuff, you don’t have to reduplicate the work again for everything on top.
The best way to extend a system is bottom-up, You start with getting the data into persistence, then into the core of the system, and then you work your way upwards until all of the interfaces reflect the changes.
The way to deal with bugs is top-down. But the trick is to keep going as low as you can. Make the lowest fixed as time will allow. Sometimes it is rushed, so you might fix a bunch of higher-level symptoms, but even if you do that you still have to schedule the proper lower-level ones soon.
The best way you screw up a large system is to let it get disorganized. The way you organize a system is from the bottom up, Looking at it top down will only confuse you.
Some people only want to see it one way or the other, as top-down or bottom-up, but clearly, that isn’t possible. When a system becomes large, the clash in perspectives becomes the root of a lot of the problems. Going in the wrong direction, against the grain, will result in hardship.
Subscribe to:
Posts (Atom)