Form Factor Free

I haven’t written any ‘crazy idea’ posts for months, so I figured I must be due. Over the years I’ve been playing around with various ‘dynamic’ behavior in the code as a way of maximizing reuse. The fundamental design has been to encapsulate the domain issues within a simple DSL, then drive both the interface and the database dynamically.

On the interface side, I’ve use declarative dynamic forms as the atomic primitive for the screen layouts. This allows me to dynamically alter the navigation as well as reduce the amount of code required to define a screen and to persistent user-constructed screens and workflows.

This type of paradigm is too expressive for basic relational database usage, so initially I built a key/value NoSQL-like (not distributed) database. For another attempt I wanted the external connectivity of an RDBMS, so I went with Hibernate and a long-skinny generic schema. The earlier attempt was significantly less code and easier to use, but the later attempt allowed for reporting and integration once I wrapped Hibernate with an OODB like interface.

Driven by a DSL, these systems have been very flexible, allowing the users to essentially move into the system and adapt it to their specific needs. The downside has been that the abstractions involved require fairly deep thinking about extending the systems. Most programmers prefer writing new code via brute force, so the speed of development is limited by finding people who won’t just hack madly around the existing code base, but are willing to read and reuse the infrastructure.

In thinking about these types of resource issues my feeling is that the kernel of any such architecture should be as small as possible and need very little modifications. Growing the system then is a matter of just inserting domain-specific algorithms and features into a predefined location in the architecture. That almost works, but in my last version I really ended up with three different places where the code had to be extended. With three different choices, I found that some programmers would pick the wrong location, slam their square peg into the round hole, and then try to compensate by shoving in lots of extra, artificial complexity. Choice it seems leads to people wanting to ‘creatively’ subvert the architecture.

My thinking these days (although it may be awhile before I get a chance to try it out) is that I want the extendability of the system to come down to one, and only one place. Taking away choice may sound mean, but I’ve always found it better to balance out programmer freedoms with system success. If too much freedom incurs a massive risk of failure, well… I’d rather the system really worked properly at the end of the day. It’s a happy user vs. a happy programmer tradeoff.

As well as encapsulating the extendability I got to thinking that the next wave of computing is going to take place on a nearly infinite number of form factors. That is, the screen size will vary from being watch size all the way up to wall size. It doesn’t make a lot of sense to write a huge number of nearly identical systems -- one for each size -- if we can enlist the computer to dynamically handle them for us.

ORMs and OODBs allow for the programmers to specify their internal data models, then have these drive the persistent storage structures. The slight wrinkle is that the persistent storage may be shared across several different applications, so it’s underlying model is likely a domain-driven ‘universal’ one instead of the various application specific models. Subsets and inherent context are likely the bulk of the differences.

Without worrying too much about the model differences, the other half of the dynamic equation is for that application model to directly drive the interface layout. Way back many companies tried to drive interfaces off relational schemas, but these systems proved too awkward and cumbersome to catch on. My sense is that the application modelling of the data needs to be driven heavily from the user/navigation side, rather than the storage side. That is, the application model reflects both the domain structure of the data, but also how the users want to manipulate that data.

If we can find an appropriate intermediate representation then the rest of it is easy. For each entity/datam in the model we attach a presentation template. To cope with the form factor free ability we attach links that handle both navigation and neighborhood relationships. When the user navigates to a screen, we get both the primary entities and the current form factor. From that we simply find anything in the neighborhood that fits in as well. Go to the user’s screen and if your screen is big enough, you’ll see all sorts of related information. Of course one has to deal with paging, dynamic navigation as well as widgets, validation and normal dynamic forms problems like cross-field validations/updates. The base problems in my earlier systems weren’t simple, but they weren’t really cutting edge either.

One time-saving possibility is that the screen construction happens at compile time rather than run time. Building the system would then produce many different components -- one for each form factor. It would be nicer to do this dynamically on-the-fly, but one always has to be weary of eating up too much CPU.

If it all worked as planned (it almost never does), extending the system is just extending the application data model. If you needed to add a new feature you’d start by integrating any new data into the model. New calculations would go in by adding new ‘derived’ entities which would be bound with calculations underneath. All of the presentation/navigation stuff would decorate the data, then all you’d need to do is just recompile, test and re-release. Changes that might normally take months could fall to weeks or days. The model intrinsically enforces any types of organization or conventions and can easily be reviewed by other programmers. With the extendability encapsulated, the base work would pay off in producing a system that could expand for years or decades without having clocked up much technical dept.

Time and Shortcuts

I haven't posted anything for a while now. This has been the longest gap I've had since I started blogging five years ago. It's not that I don't have any anything to say -- my hard drive is littered with half-finished posts -- but rather that I just haven't had the time or energy to get my thoughts down nicely.

Lately I've been wrapped up in a large complicated system that has a wide array of problems, none of which are new to me but it's unusal to see them all together in the same project. I like this because it is extremely challenging, but I definitely prefer projects to be well-organized and focus on engineering problems, not organizational ones. 

My sense over the last couple of decades is that software has shifted from being an intense search for well-refined answers to just a high-stress fight against allowing all of the earlier shortcuts to swamp the project. I find that unfortunate because the part of the job that I've always loved was that satisfaction from building something good. Just whipping together a barely functoning mess, I find depressing.

 

What I've noticed over the years is that there is a whole lot more progress made when the code is well-thought out and clean at all levels (from syntax to the interface). The increasingly rare elegant solution takes it one step higher. You get so much more out of the work, since there are so many more places to leverage it. But of course spending the time to get it right means coming out of the gate that much slower, and it seems that people are increasingly impaitent these days. They just want a quick bandaid whether or not that will make the probem worse. 

Getting a mess back under control means having to say 'no' often. It's not a popular word and saying it comes with a lot of angst. It does not help that there are so many silver bullet approaches floating about out there. Anyone with little software knowledge is easily fooled by the over abundance of snake charmers in our industry. It's easy to promise that the work will be fast and simple, but experience teaches us that the only way to get it done is hard work and a lot of deep thinking. Writing code isn't that hard, you can teach it to high school students rapidly, but writing industrial strength code is a completely different problem.

I'd love to take off some more time and write another book that just lists out the non-controversial best practices that we've learned over the last few decades -- a software 101 primer -- but given that my last effort sold a massive 56 copies and I'm a wage slave it's not very likely that I'll get a chance to do this anytime soon. The trick I think is to shy away from the pop philosophies and stick to what we know actually works. Software development is easy to talk about, easy to thoerize about, but what often really works in practice is counter-intuitive for people with little experience. That's not unusal for complex systems and a large development projects contains millions of moving parts (people, code, technology, requirements, data, etc) with odd shifting dependencies. You can't understand how to organize such a volitile set of relationships without first devling deep into actual experience and even then it's hard to structure the understanding and communicate it. 

What flows around the production of the code is always more complex than the code itself and highly influenced by its environment. A good process helps the work progress as rapidly as possible with high quality results, most modern methodologies don't do that. That's one of our industries most embarrassing secrets and it seems to be only getting worse.

Hopefully one of these days I'll catch my breath and get some of my half-finished posts completed. There are some good lessons learned buried in those posts, it just takes time and patience to convert them into some shareable.

Relationships

“Everything is relative in this world, where change alone endures.”

Leon Trotsky

A huge problem in software development is to create static, rigid models of a world constantly in flux. It’s easy to capture some of the relationships, but getting them all correct is an impossible task.

Often, in the rush, people hold the model constant and then overload parts of it to handle the change. Those types of hacks usually end badly. Screwed up data is computer can often be worse than no data. It can take longer to fix the problem then it would to just start over. But of course if you do that, all of the history is lost.

One way to handle the changing world is to make the meta-relationships dynamic. Binding the rules to the data gets pushed upward towards the users, they become responsible for enhancing the model. The abstractions to do this are complex, and it always takes longer to build than just belting out the static connections, but it is often worth adding this type of flexibility directly into the system. There are plenty of well-known examples such as DSLs, dynamic forms and generic databases. Technologies such as NoSQL and ORMs support this direction. Dynamic systems (not to be confused with the mathematical ‘dynamic programming’) open up the functionality to allow the users to extend it as the world turns. Scope creep ceases to be a problem for the developers, it becomes standard practice for the users.

Abstracting a model to accommodate reality without just letting all of the constraints run free is tricky. All data could be stored as unordered variable strings for instance, but the total lack of structure renders the data useless. There needs to be categorization and relationships to add value, but they need to exist at a higher level. The trick I’ve found over the years is to start very statically. For all domains there are well-known nouns and verbs that just don’t change. These form the basic pieces. Structurally as you model these pieces, the same type of meta-structures reappear often. We know for example that information can be decomposed into relational tables and linked together. We know that information can also be decomposed into data-structures (lists, trees, graphs, etc) and linked together. A model gets construction on these types of primitives, whose associations form patterns. If multiple specific models share the same structure, they can usually be combined, and with a little careful thought, named properly. Thus all of the different types of lists can just one set of lists, all of the trees can come together, etc. This lifts up the relationships by structural similarity into a considerable smaller set of common relationships. This generic set of models can then be tested against the known or expected corner-cases to see how flexible it will be. In this practice, ambiguity and scope changes just get built directly into the model. They become expected.

Often when enhancing the dynamic capabilities of a system there are critics who complain of over-engineering. Sometimes that is a valid issue, but only if the underlying model is undeniably static. There is a difference between ‘extreme’ and ‘impossible’ corner-cases, building for impossible is a waste of energy. Often times though, the general idea of abstraction and dynamic systems just scares people. They have trouble ‘seeing it’, so they assume it won’t work. From a development point of view that’s where encapsulation becomes really important. Abstractions need to be tightly wrapped in a black-box. From the outside the boxes are as static as any other piece of the system. This opens up the development to allow a wide range of people to work on the code, while still leveraging a sophisticated dynamic behavior.

I’ve often wondered about how abstract a system could go before it’s performance was completely degraded. There is a classic tradeoff involved. A generic schema in an RDBMS for example will ultimately have slower queries than a static 4th NF schema, and a slightly denormalized schema will perform even better. Still, in a big system, is losing a little bit of performance an acceptable cost for not having to wait for 4 months for a predictable code change to get done? I’ve always found it reasonable.

But it is possible to go way too far and cause massive performance problems. Generic relationships wash out the specifics and drive the code to being in NP-complete or worse. You can model any and everything with a graph, but the time to extract out the specifics is deadly and climbs at least exponentially with increases in scale. A fully generic model of everything just being a relationship between everything else is possible, but rather impractical at the moment. Somewhere down the line, some relationships have to be held static in order for the system to perform. Less is better, but some are always necessary.

Changing relationships between digital symbols mapped back to reality is the basis of all software development. These can be modeled with higher level primitives and merged together to avoid redundancies and cope with expected changes. These models drive the heart of our software systems, they are the food for the algorithmic functionality that helps users solve their problems. Cracks in these foundations propagate across the system and eventually disrupt the user’s ability to complete their tasks. From this perspective, a system is only as strong as its models of reality. It’s only as flexible as they allow. Compromise these relationships and all you get is unmanageable and unnecessary complexity that invalidates the usefulness of the system. Get them right and the rest is easy.