In Framing Work

"Wouldn't it have been better to allow the programmers to attach their own hash table directly into the form? " one of my fellow developers said gently.

"That way they can change the values in the hash table, and then update the form -- it's a more common approach to do it that way" he explained.

I was in the middle of discussing my latest piece of code, a complex generic form handing mechanism built on top of GWT.

It was a good question, in that while I was programming in Java I had clearly drifted away from the Java accepted paradigms, and into some else complete different.

Mostly that is a pretty bad idea. If a platform has an accepted way of handling a specific problem, you need pretty strong justification to choose to go at the code in a different way. It's always about consistency, and picking one well-known way of handling a problem then sticking to it is superior to filling the code with a million different ways to accomplish the same results. Simplicity and consistency are tied together.

However, in this case I was fairly happy with my results. I had deliberately chosen a different path, one that I was consistently applying in the upper layers of this code. It's intrinsic to the philosophy of the architecture. Consistency with the overall paradigm, is usually more important than getting caught up in an underlying language one. Besides, strictly speaking, only half of GWT is Java; the client half is Java-inspired JavaScript.

I chose this path deliberately because it matched my objectives, and I chose my objectives in order to maximize my resources. They are related.

The architecture can and often should reflect elements of the environment in which it is being developed. We are interested in being successful, and the definition of that changes depending on where we are located. A point easily over-looked.

Despite my confidence in the design, the question was still really interesting, because it forced me to think about the various pieces that combined to affect the architecture. I've been at this a long time and some things just become subliminal. I stop thinking about them. You know they work, but you never really move it into the foreground to examine it. Following this question through, leads into a lot of interesting issues. Things worth considering.


FRAMEWORKS AND LIBRARIES

The first issue comes from having a different perspective on code.

All programs have a control loop. That is the main flow of control; often one or more tight loops that continually go over and over again before they call out to the functionality. In a GUI an event loop is a common example. In some batch systems, there might be a pending queue of operations -- a finite control loop, but never-the-less the same concept. All programs have a control loop of some type, even if in a few batch cases it is trivial.

If you're looking at large pieces of code, we can classify them by whether or not they contain a control loop. This is already a common classification, it's just never really been formalized.

Thus, a 'library' of functionality definitely has no control loop. The code enters, and gets out quickly, heading back to the main event loop somewhere. You call a library to do something specific for you.

A 'framework', like Struts for thin clients, or Swing for thick ones definitely encapsulates a loop. Working with that type of code means attaching bits of functionality at various open places in the framework. Sort of like affixing pieces to the bottom of a large appliance. You hand over control to a framework for some indeterminate period of time.

The more normal definition of framework usually implicitly means that there is one and only one global one, but if you're following the control loop argument, there is no reason why we can't have many layers of frameworks. Frameworks within framework, and frameworks embedded in libraries. As libraries layer upwards -- you build on them -- frameworks layer downwards.

From this perspective, we can then assert that every component in a software system either contains a control loop of some type or it does not.

Thus every component is either a 'library' or a 'framework'. That type of breakdown is really convenient, in that we can now see the two as being the yin and yang of all software code. All large code blocks can be deconstructed into discrete components of either frameworks or libraries. This allows us to see the code as being one or the other.


GOALS

For a lot of code -- especially libraries -- amongst the main goals is the desire to allow the code to be re-used multiple times. We want to leverage it as much as possible.

This often leads to a large number of configurable parameters getting added, massively kicking up the complexity. A common side-effect is poor interaction between all of configurable pieces, forcing people using the technology to stick to well-known combinations of options, more or less invalidating the work that went into it.

The programmers then get a huge amount of freedom to utilize the library in any way possible. On this, they can build a lot of code. They also have the freedom to do things incorrectly or inconsistently. Freedom is a great idea until it becomes anarchy.

My goals for the this latest system are pretty much the opposite. I don't want to provide a huge number of small lower-level primitives to build with, I want to restrict the programmers where and however possible. They need just enough flexibility to get the work done easily, but not enough to make it inconsistent. The system itself should constrain them enough to enforce it's own consistency. I don't want open, I want simple and consistent, but I'll get back to that in a bit ...

We're taught to separate out the presentation layer from the model, but do we really understand what that means?

At some lower layer the system holds a huge amount of data that it is manipulating. Lots of the data is similar in many different ways. Ultimately we want to preserve consistency across the system when displaying the different types. In that sense, if we create a 'type' hierarchy to arrange the data, then we can choose reasonable consistent ways to display data whenever it is similar. The username data should always appear the same wherever it is located, so should groupname data.

The data in any system comes in the shape of a fixed series of regular types. The type and structure are almost always finite. Dynamic definitions are possible, but underneath the atomic pieces themselves are static. Everything in a computer is founded on discrete information. We need that in order to compile or run the code.

To these types we want to attach some presentation aspects. Username, for example, is the type. When reading or writing this data, the widgets, colors, fonts, etc. we use on the screen are the actual presentation. It's the way we ask for and display this data.

If the same data appears on twenty different screens, then it should be displayed the same on each screen. Of course data clumps together and different screens present different sub-contexts of the data, but almost all screens are composed of the same finite number of data elements. They switch around a lot but most systems really don't handle a massive number of different types of data, even if they do handle a massive amount of data. The two are very different.

If we create a model of the data, using a reasonable number of types, then for "presentation" all we have to do is define the specific clumps of data that appear together and then bind them to some presentation aspects.

As we are presenting the data, we don't need to know anything about it other than it's type. In a sense, all screens in every system can be completely generalized, except for the little bit of information (name, type and value) that get moved around.

Getting back to my goals, after having written a huge number of systems it has become obvious that for most, a large number of the screens in the system are simply there to create, edit or display the basic data.

The real "meat" of most systems is usually way less than 20% of the screens or functionality. The other 80% just need to be there to make the whole package complete. If you can create data, you need to edit it. If it's in the system you need to display it. All systems need a huge number of low frequency screens that support it, but are not really part of the core. Work that is time-consuming, but not particularly effective.

We can boil down a basic system into a lot of functionality, yet most of it revolves around simple screens that manipulate the data in the system.

One problem with growing systems is that once the functionality -- which started with a few hundred different features -- grows enough, the static hard-coded-ness of the screens becomes a problem. As the functionality grows into thousands of features, the programmers don't have the time or energy to re-organize the navigation to get at it. Thus, virtually all major systems get more and more features, but become more and more esoteric in their placement. And it's a huge and expensive risk to interfere with that degeneration. A gamble lost by many companies over the years.

For all programs, the difference between a few features and a few thousand is a completely different interface. Yet that work can't happen because it's too large. Thus the code-base becomes a disaster. Everybody tip-toes around the problems. And it just gets worse.

So, most of the functionality, in most systems is simple, boring and static. And it's organization/navigation represents an huge upcoming problem (technical debt) for the development, whether or not anyone has realized it. What to do?

Clearly we want to strip away absolutely everything unnecessary and while it's still static, make the remainder as "thin" as possible. We want just the the bare essence of the program in it's own specification.

To do this we can see all interactive programs as just being some series of screens, whether static or dynamic. There are always very complex ways to navigate from one location in the system to another. Thus an overly simplified model for most systems is for some type of navigation of some type of discrete (finite) screens. A screen might contain dynamic data, but the screen itself has a pre-determined layout so that it is consistent with the rest of the system.

Now, we know from the past, that any overly simplified model, like the above simply won't work. A great number of 4GLs where written in the 90s and then discarded trying to solve this exact problem. One of our most famous texts tells us explicitly that there is no such thing as a "silver bullet".

But I'd surmise that these early attempts failed because they wanted to solve 100% of the issue. They were looking for a one-size-fits-all solution. For my work, I'm only really interested in a fixed arrangement for the 80% of screens that are redundantly necessary, yet boring. The other 20% are SEP, as Douglas Adams pointed out in one of the Hitchhiker's Guide to the Galaxy books: Somebody Else's Problem.


ARCHITECTURE

In the definition of the navigation, and the definition of the screens, I really want to specify the absolute minimum "stuff" to map some types onto sets of data within screens.

Navigation is easy, it's really just various sets of locations in the system, and ways of triggering how to get there. Some are absolute, some are relative to the current location.

The sets of data are also easy, we can choose a form-based model, although being very generous so the forms can do all sorts of magical things. Well beyond a simple static one.

For flexibility we might have several 'compacted' screens, basically several simple main-frame like screens all appearing together to ease navigational issues. Web apps deal with condensed navigation well -- originally it was because of slow access -- so it's wise to make use of that where ever possible.

The forms simply need to bind some 'model' data to a specific presentation 'type'. A form needs be interactive, so it could just be a big display of some data. With this model in mind, all of the above is a purely presentational aspect.

Why not choose MVC like everyone else? Long time ago I built a really nice thick client MVC system, where I had a couple of models and the ability for the user to open up lots of different windows with different views on the same underlying models. If they change one aspect of a model in one window, it was automatically updated in another. That really nice, yet complicated trick is the strength of MVC, but that's not necessary in a web application. Since there is only on view associated with the underlying model, using a fancy auto-updating pattern is far too complex and far too much overkill when the goal is to simplify.

Still, although I wasn't headed for MVC, separating out the presentation, meant having a model anyways. And in my system, the presentation layer was in the client side of the architecture, and the model ended up in the server side. It fit naturally within the GWT client/server architecture.

The client become 100% responsible for presentation, while the server is only interested in creating a simplified model of the data. The client is the GUI, the server is the functionality.

Popular philosophy often tries to convince programmers that the system's internal model and the one in the database should be one great "unified" view of the data. That's crazy since, particularly with an RDBMS, the data can be shared across many applications. That's part of the strength of the technology. In that sense, the database should contain all of the data in some domain-specific 'universal' view. That is, the data should be in it's most natural sense relative to the domain from which it was gathered. For example someone pointed out that banks don't internally have things called saving accounts, just transaction logs. "Saving Accounts" are the client perspective on how their money exists in the bank, not the actual physical representation. A bank's central schema might then only contain the transactions.

On the other hand, all applications -- to be useful -- have some specific user-context that they are working in. The data in the application is some twisted specific subset of the universal data. Possibly with lots of state and transformations applied. It's context specific. As such, the application's internal model should most closely represent that view, not the universal one. An application to create and edit savings accounts, should have the concept of a saving account.

Why put universal logic into a specific application or specific data into a universal database? All applications have two competing models for their data. Lots of programmers fail to recognize this and spend a lot of time flip-flopping between them, a costly mistake.

Getting back to the architecture, the application specific model sits in the server side, while the presentation side is in the client.


MORE CHOICES

Another popular dictate is too not write a framework. But given my earlier definitions, an application can (and should) have a large number of smaller frameworks depending on the type of functionality it supports. Writing smaller frameworks is inevitable, in the same way that writing custom libraries is as well. It's just good form to build the system as reusable components.

My choice in building the form mechanism was between creating it as a library for everyone to use, or creating it as a framework and allowing some plugin functionality.

While the library idea is open and more flexible, my goals are to not allow the coders to go wild. Open and flexible go against the grain. I simply want the programmers to specify the absolute minimum for the navigation, screens and data subsets. As little as possible, so it can be refactored easily, and frequently.

A framework on the other hand is less code, and it can be easily controlled. The complex behaviors in the forms, as the system interacts, can be asked for by the programmers, but it's the framework itself that implements them. Yes, that is a huge restriction in that the programmers cannot build what the framework doesn't allow, but keep in mind that this restriction should only apply to the big redundant stuff. The harder 20% or less, will go directly to the screen and be really really ugly. It's just that it will also be unique, thus eliminating repeated code and data.

If the code is boiled down into nothing but its essence, it should be quick to re-arrange it. Not being repeated at all is the definition I put forth for six normal form. The highest state of programming consistency possible. Which means it's intrinsically consistent, the computer is enforcing the consistency of the screens, and where it's not, it's simply a binding to a type that can be easily changed.

The form definitions themselves are interesting too. Another popular design choice is to make everything declarative in an external language/format like XML. Frameworks like Struts have done this to a considerable degree, pushing huge amounts of the structural essence of the code, out of the code and into some secondary format.

Initially I was OK with these ideas, but overtime, in large systems they start to cause a huge amount of distributed information complexity.

A noble goal of development, known as Don't Repeat Yourself (DRY), is founded around the idea that redundancies should be eliminated because duplicate data falls out of sync. Often however, we have to have the same data, just in a couple of different formats, making it impossible to eliminate. If we can't get ride of duplicated yet related data, we certainly can work very hard to bring all of the related elements together in the same location. This type of encapsulation is critical.

The stripped out declarative ideas do the exact opposite. They distribute various related elements over a large number of configuration files, making it hard to get the full picture on what is happening. Object Oriented design is also guilty of this to some degree, but where it really counts, experience programmers will violate any other principles, if it helps to significantly increase the localization and encapsulation.

Virtually any set of ideas that center around breaking things down into small components and filing them away, start to fail spectacularly as the numbers of components rise. We can only cope with a small number of individual pieces, after which the inter-dependencies between those pieces scales up the complexity exponentially. Ten pieces might be fine, but a few hundred is a disaster. It's a common, yet often overlooked issue that crops up again and again with our technologies.

Getting back to the forms, my goal was that the representation be static, simple and completely encapsulating. The form is different from the data, but they both are simple finite descriptions of things. All of the forms and the data in the system can be reduced to textural representations. In this case I picked JSON for the format, because it was so much smaller than XML, doesn't always require the naming of the elements, and because JSON was easily convertible to JavaScript, where the client and the form framework are located.


SERVER SIDE AND BACK

In this design, the programmers have little control over the presentation, thus forcing them to use it consistently. On the back-end, the model is still reasonably flexible, however it is usually bound to a relational database.

The schema and any relatively discrete application-context transformation away from that schema have limited expressibility. You can get past that by giving the application context more state, but that's not the best approach to being simple, although sometimes it can't be helped.

Still, all you need to do is pass along some functionality indicator, assemble data from the database in the proper model, and then flatten it for transport to the front-end somehow. The expressibility of the front is actually flattened by definition anyways, so it's possible to put all of the data into some discrete flat container stored in JSON and pass it to the presentation layer. All that's needed is a simple way to bind the back-end data to some front-end presentation. A simple ASCII string for each name is sufficient.

Incoming data goes into a form, and then interacts with the user in some way. Most systems involved modifying data, so a reverse trip is necessary, as the data goes out from the form, over to the back-end and then is used to update the model. From there it is persisted back into the database.

The whole loop, database -> front-end -> database consists of some simple discrete and easily explainable transformations. Better described as

universal-model->app-model->container->presentation->container->app-model->universal-model

It's the type of thing that can be easily charted with a few ER diagrams and some rough sketches of the screens. The system needn't be any more complex than that.

Getting back to the form mechanics. The framework simply needs to accept a form definition, add some data and then do its thing. From time to time, there may be a need to call out and do some extra functionality, such as fetch more data, sort something or synchronize values between different form elements. But these hooks are small and tightly controlled.

As some point, either at the end or based on some set of events, the form gives control back to the screen, chucking up some data with it. Allowing the code to tie back to the database or move to some other point in the navigation.

The mechanism is simple. The programmers give the framework data, and at some point, when it's ready, they can get it back again. Until then, it's the framework's problem. They can now spend their time and energy working on more complex problems.

Aside from controlling the consistency, the design also helps with testing. With an inverted form library design there is a lot of flexibility, but lots to go wrong. With a black-box style form framework, the internal pathways are well-used and thus well-tested. The system is intrinsically consistent. It's utilizing the computer itself to keep the programmer from making a mess.


SUMMARY

There is more; generalizing this type of architecture involves a larger number of trade offs. Every choice we make in design is implicitly a trade off of some type.

If you boil down the domain problems, you most often find that the largest bulk of them are just about gathering together big data sets.

Mostly the applications to do this are well understood, even if we choose to make them technically more complex. Still, while writing a multi-million line application may seem impressive, it's bound to accumulate so much technical debt that it's future will become increasingly unstable.

The only way around this is to distinguish between what we have to say to implement some specific functionality and what is just extra noise added on top by one specific technology or another. In truth, many systems are more technologically complex then they are domain complex. That is a shame because it is almost never necessary. Even when it is, a great deal of the time, the technical complexity can be encapsulated away from the domain complexity. We can and should keep the two very separate.

Even after such a long discussion, there will still be a few people unconvinced by my explanation. Sure of my madness. The tendency to build things as libraries, because that's the way it's always been done, and because frameworks are currently considered verboten will be too strong for many programmers to resist.

We crave freedom, then strictly adhere to a subset of subjective principles, which is not always the best or even a rational choice. Getting the simplicity of the code under control so extensions to the system aren't inconsistent or tedious is a far more important goal than just about any other in programming. If it means breaking some nice sounding dogma, to get simple and expandable, then it is worth it. Code should be no more complex than it must be. We know this, but few really understand its true meaning. Or how hard it is to achieve in practice.

Developers shouldn't be afraid to build their own frameworks or libraries, or any other code if the underlying functionality is core to their system. That doesn't mean it should be done absolutely from scratch -- there is no sense in rediscovering anew all of the old problems again -- but if one can improve upon existing code, partially after reading and understanding it, then it's more than just a noble effort to go there. Dependencies always cause problems, waste time. It's just whether that effort in the long run is more or less than the design, development and testing of a new piece of code. If you really know what you are doing, then a lot more comes within your reach.