Observable on Steriods

A while back I wrote a small product that, unfortunately, had an early demise. It was great technically -- everyone who saw it loved it -- it’s just that it didn’t get any financial support.

The idea was simple. What if all of the data in an application was observable? Internally it was just thrown into a giant pool of data. So, it could be variables in a model, calculations, state, context, etc. Everything is just observable data.

Then for each piece of derived data, it would watch any of its dependencies. If it was a formula, it would see that any of the underlying values changed, recalculate, and then tell anyone watching it that it is now different.

The stock way of implementation observable in an object-orient paradigm is to keep a list of anyone watching, then issue an event, aka function call, to each. The fun part is that since any object can watch any other object, this is not just a tree or a dag, it can be a complete graph.

Once you get a graph involved, any event percolating through it can get caught in a cycle. To avoid this, I traded off space by having each event keep a dictionary of the other objects it had already visited. If an object gets notified of an event and it is already in that visited set, it just ignores the event. Cycles defeated.

So, now we have this massive pool of interconnected variables, and if we dumped some data into the pool, it would set off a lot of events. The leaf variables would just be updated and notify watchers, but the ones above would recalculate and then notify their watchers.

I did do some stuff to consolidate events. I don’t remember exactly, but I think I’d turn on pause, update a large number of variables, then unpause. While paused, events would be noted, but not issued. For any calculation with a lot of dependents, if one thing changed, I’d track the time and since it has already recalced, grabbing all of the child variables data, it would toss any events for other dependents that were earlier. So, for A+B+C, after the unpause, you’d be notified that A changed, and do the recalc, but then the time that B and C changes would be earlier than the recalc, so ignored. That cut down significantly on any event spikes.

Then finally, at the top, I added a whole bunch of widgets. Each one was wired to watch a single variable in the pool. As data poured into the pool, the widgets would update automatically. I was able to wire in some animations too, so that if a widget displayed a number, whenever it changed it would flash white, and then slowly return to its original color. But then I wired in tables and plotting widgets as well. The plot, for instance, is a composite widget, whose children, the points of the curve, are then they are all wired to different variables in the pool. So, the plot would change on the fly as things changed in the pool.

Now if this sounds like MVC, it basically is. I just didn’t care if the widgets were in one view or a whole bunch of them. They’d all update correctly either way. and the entire model was in the pool. And any of the interface context stuff was in the model so in the pool. And any of the user’s settings were in the model, so in the pool. In fact, every variable, anywhere in the system, was in the model, so in the pool. Thus the steroids designation. Anything that can vary in the code is an object and that is observable in the pool.

Since I effectively had matrices and formulas, it was a superset of a spreadsheet. A sort of disconnected one. A whole lot more powerful.

Because time was limited, my version had to wire up each object as code explicitly. But they didn’t do much other than inherit the mechanics, define the children to watch, and provide a function to calculate. It would have been not too hard to make all of that dynamic, and then provide an interface to create and edit new objects in the app itself. This would let someone create new objects on the fly and arrange them as needed.

It was easy to hook it up to other stuff as well. There was a stream of incoming data, so it was handled by a simple mapping between the data’s parameters and the pool objects. Get the next record in the stream, and update the pool as necessary. Also, keyboard and button events would dump stuff directly into the pool. I think some of the widgets even had two-way bindings, so the underlying pool variable changed when the user changed the widget and could percolate to everything else. I had some half cycles, where the widget displayed a value, it was two-way bound and as the user changed it, it triggered other changes in the pool, which updated stuff on the fly which would change the widget. I used that for cross-widget validations as well. The widgets change the meta information for each other.

I did my favorite form of widget binding, which is by name. I could have easily added scope to that but the stuff I was working with was simple enough that I didn’t have any naming collisions. I’ve seen structural-based binding sometimes, but they can be painful and rigid. The pool has no structure and the namespace is tiny because the objects were effectively hardwired to their dependencies. Extending it would need scope.

To pull it all together I had dynamic forms and set them to handle any type of widget, primitive or composite. I pulled a trick from my earlier work and expanded the concept of forms to include everything on the screen including menus and other recursive forms. As well, forms could be set to not be editable, which lets one do new, edit, and view all with the same code, which helps to save coding time and enforce consistency.

Then I put in some extremely complex calculations, hooked it up to a real-time feed, and added a whole bunch of screens. You could page around while the stream was live and change stuff on the fly, while it was continuously updating. Good fun.

It’s too bad it didn’t survive. That type of engine has the power to avoid a lot of tedious wiring. Had I been allowed to continue I would have wired in an interpreter, a pool browser, and a sophisticated form creation tool. That and some way to dynamically wire in new data streams would have been enough to compete with tools like Excel. If you could whip up a new table and fill it with formulas and live forms, it would let you craft complex calculation apps quickly.