Friday, July 25, 2008

The Art of Our Understanding

Ah, the ongoing great battle between whether something is an 'art', a 'science' or 'engineering'. Geeks, more than anyone else, love to fight this one out.

I read Knuth's take, hiding over at Paul Graham's site:

http://www.paulgraham.com/knuth.html

I find that his definition for the word art is a huge let down. Art, in the way he writes about it, is really just the unstructured act of doing anything. If it ain't organized, then it must be art.

That, as history shows, is a reasonable use of the word, but it is depressingly cold. It is so because there are more dimensions to human existence than just being. We straddle these wonderful worlds of emotion and thought, sitting on the fence in between.

Our emotions form primitive reactions to the world around us. They are, in all sense, the very opposite of logical thought. They drive us to do crazy and irrational things.

It's not hard to guess that we're actually in the process of evolving from primal emotional creatures to higher forms of logical entities. These days we're caught in the middle. We can be rational and think our way out of the problems, but we can also succumb to crazy emotional responses. This leads to a duality in our perspective. The natural order of things, is an emotional sense, with its rules about the survival of the species. Our intellectual order, is about higher values such as justice and fair-play. Abstract concepts that seem out of place in the real world.

It is emotion, I truly believe, that when rendered into a tangible form sets some constructive act apart from the others. A building houses us, but a majestic one takes our breath away. A graphic designer makes pretty images, but a great painter endows them with that full sense of being. A singer might belt out the notes in tune, but a talented musician can bring tears to our eyes. Any person can stand up and say a dialog, but an actor makes us believe that they feeling it.

Again, and again the differences come down to how various people have crossed beyond just an act, and have found some indescribable way to add emotional context to their work. They touch us at our very core. They reach out to our primitive side.

And, to make it even more interesting, because it is some low-level reactive nature within us, it is virtually impossible to wrap it in a logical foundation. You can not say it, teach it, or pass it on in any concrete manner. It must be felt to be understood, and experienced to be copied. It is beyond our rational thought.

The problem with Knuth's practical definition for the word 'art', is that it does not distinguish between those things that just are, and those that have gone well beyond. It makes it reasonable to talk about the 'art of knitting a sweater' for instance. While that does pay homage to the skill required to actually do the work, it indirectly puts the sweater along side great classic works. Works that inspire emotion in ways that a sweater never could.

Personally, I think the term art, when it is applied in the narrower modern context should only refer to those things that connect with us on an emotional level. We need to distinguish between ordinary stuff, and those great works that enlighten us in some ethereal way. If we use one term for both, we will quickly lose all of our art in a sea of pretty pictures, and that would be sad.

There is something that should be appreciated in people using their skills to create things in an unstructured manner, but it is a rather common event. It happens all the time, all over the world. Artisans craft wares, artists create masterpieces. Art, at least in the sense that I mean it, is a rare and wondrous event. We lose some of ourselves when we fail to recognize that.

Wednesday, July 23, 2008

Meta Blogging Madness or Mania

I've been really good over the last year. Despite some temptations, I've pretty much stayed away from the usual meta-blogging topics. Sure, I've tilted occasionally, or even had to delete an unpublished post or two, but I figure with a record like that it's OK to take a quick break from my usual work and spew out a few of my feelings.

It's my one year anniversary for this blog, and my two and two-thirds anniversary for my whole blogging career. Barely nothing at all, but an incredibly long time for me.

My first blog musings, called "Building Blocks" are collected together at:

http://www.lulu.com/content/1054566

I started blogging in an attempt to improve my writing style. Doing everything backwards, I started first by writing a book on programming:

http://www.lulu.com/content/178872

I'd heavily promote the book -- it is full of great ideas and understanding -- but the writing is a little stiff, repetitive and messy. I'd spent most of my life avoiding writing, only to find myself accidentally drawn in to it. I just wanted to express my understanding of software construction.

I wrote it based on my raw experiences gained from releasing a complex product with virtually no resources. When you have barely nothing to work with, you learn to use it well. For all its weaknesses, the book does get right down into finding ways to manage the complexity of development.

I'm considering redoing the book for a second edition, but I don't want to put in a lot of effort if there isn't any real demand. The writing needs to stop getting in the way of the content.

That is ultimately why I started blogging; to learn how to become a better writer, so I could share the things I've learned.

Surprisingly I found that blogging was a lot harder than I had imagined. My grammar, and style are terrible, but working on that is the easy part. The biggest problems come from trying to find ways to express real knowledge, but in a way that is entertaining enough to attract readers. I'm always finding myself balancing between what I want to really say, and what people really want to hear.

You can't just spit out any old idea or concept. People don't want to slog through your inner deep thoughts if there isn't some incentive. Sadly, learning seems to rarely be the driving force. Most blogs, tend towards lighter topics or platitudes, because most of the readers prefer it that way. Blogging is a conversational form of entertainment.

Still, you don't have to spend long in software development to be embarrassed by the state of the industry. That, I think is one of the driving forces behind software developers combing through tonnes of blogs, hoping for that one significant revelation that will vault their abilities into super-stardom.

We know things aren't working, but we're clueless as to how to make things work better. At least those of us who still care, are; many have just tuned-out and gone back to flailing at their keyboards; aware, as always of that silently ticking count-down clock that will drive them from the industry for good.

Readers don't give you much credit for trying; falling short is always greeted with a great wall of silence. In many ways that can be the hardest part about blogging; even the rude insults are a offbeat form of complement. Silence is deafening.

At least they're taking you somewhat seriously. Seriously enough to be mad at you.

A friend of mine talked about how writing is such a lonely job. I guess because you are putting words on a sheet of paper, you're only communicating with your audience indirectly. There is a natural disconnect between you and the readers; you have no idea if they respect what you are saying or are just silently laughing at your foolishness.

It is getting past that fear -- of being the fool -- that I think is the hardest aspect of blogging. Not, that I don't internally dwell on my short comings, but few people really want to expose all of their weaknesses to the public. I want you to see me with the good looks of Brad Pitt, and the intelligence of Albert Einstein. After all, that is how I choose to see myself (staying away from mirrors really helps). My sense is that the more people that respect me, the more my ideas will get fair play. Each flaw just makes it harder for me to communicate.

Rambling is another great problem. I very much want to say everything, but time is always limited. It is far easier to write something long and boring then it is to distill all of one's thoughts down into some distinct set. Those technical writers that employ big ugly huge words on a mass scale throughout their work, mistakenly believe that it lends weight to what they are saying. Instead it often sinks it.

But sometimes even when you instinctively know something, finding a way to verbalize that, in a short, simple and straight-forward manner is exceptionally tough. I fail often.

There are times when I just want to stop. I start to wonder if there is really a point to this. Computer Science, I am quite aware, is a huge mess, but why should I be so egotistical as to think that even the smallest of things I've learned are of any use to world at large?

I'm driven forward again and again by continual examples of just how crude most people are going at it. I've not managed to find even a fraction of the answers, but what little I know so often seems miles ahead of what I see out there. So many people go at software with a reckless abandon that not unsurprisingly leads to serious complications. The biggest and best things I've learned are that it really isn't that hard after all, if you moderate your behavior and ideas. Extreme problems come from extreme behavior.

My biggest problem seems to be the middle-class gravity well. I am continually caught by those forces that pull me back into trying to support my lifestyle. You accept the house, the car, and the 'things', and suddenly you find that you have to maintain a well paying position in order to keep what you have.

At first it is fine, but it gradually becomes an incentive to not push one's limits. The safe jobs, often pay the best. Once you've maximized your programming salary, the next levels only come from moving higher up into management. Pretty soon, the bigger house, bigger car and more 'things' becomes the reason why you're not out risking everything at a startup, or holed up in your basement for a year writing something radical. You can't easily risk all of the things you've gained. Each year you have farther to fall.

But, I have pushed the envelop, and I've pushed my luck a few too many times already. Ages ago I thought up some really radical ideas -- things that would fundamentally alter everything we know about computers -- but I wanted to control how I approached them. Initially I felt that if I could break out of the middle class with some novel new startup, I could earn my freedom, allowing me to pursue my dreams.

I went at it for five years, but I lacked the horsepower to make it work. After crashing back to the planet, painfully, I tried again, this time looking for financing, but I'm not nearly charismatic enough to inspire confidence.

I often figured that the ultimate joke is that my ideas either won't work, or I'll just never get my chance. The universe is littered with lost, but great advancements. I've pretty much settled on dumping mine out onto the world one day once I've finally given up. Still, no one wants to be the guy who sold DOS to Bill Gates for an obscenely low price. Although I'd rather be that, then see Computer Science struggle on in it's current fashion for hundreds of years before someone finally gets it.

You don't have to be a genius to see the next step, you just need to have been standing in the right place at the right time. Eventually someone will find the same spot, they always do. Progression, at some speed is a natural by-product of our coming together, but that doesn't mean that someone like Albert Einstein didn't save us hundreds of years by making such a fundamental leap across the void.

Realistically I'm probably nowhere near smart enough to be able to actualize my concepts. Just having a good idea doesn't mean it will work. Just having an idea doesn't mean it is good. But it's not easy to convince oneself of their own short comings, after the idea -- good or bad, smart or foolish -- has taken root. Thus for all my self-doubt, I am still driven forward because it is too late to go back.

The side-effect of thinking you have a radical idea is that it changes your perspective on the rest of the industry. Knowing that there is some entirely different way for software to exist, drives the heart of my inquisitiveness. I don't think we've reached the state of the art, and because of this I am willing to look at everything with a critical eye. There is, even if I am barely correct, a huge distance for us to travel with our understandings. We've only started the journey, yet so many of us are digging in.

That frustration fuels my blog writings. I am seriously looking for something new. Something radical. Something that will lead us beyond our current threshold, and allow us to build the types of systems that utilize the fantastical potential we've created in computers.

In the craziest sense, it doesn't actually matter if any of my ideas are workable or not, it only matters that there is enough space left in software development for there to still be 'radical' ideas.

All of that brings me back to laboring away at my blog. I feel that I am balancing significantly deep content with trying to be entertaining. Well, at least that is what I am trying to do. It doesn't always work, and I can't always live up to my expectations. Still, sometimes we have to keep at it, even if we know that its not as strong as we would like. Blogging seems to come easily for some, while others are destined to wince with every forced paragraph.

With so many possibilities for software, it's hard not to overflow with ideas. Blogging might be casual entertainment, but it is also the right forum for us to explore the next round. We just need to make sure that today's blogging content really effects tomorrow's technologies. That way, it will be constructive, as well as entertainment.

Wednesday, July 16, 2008

Resources for Software Development

We, the software development community, are often the source of our own really bad software. Sure, you can blame the managers or you can rail against the underlying technologies, but in the end the most frequent, most destructive problems come from the attitudes of the people involved in the development.

Software, as in any endeavor that involves constructing physical things, requires a great deal of discipline. A mess by definition is repetitively doing things differently. All of those inconsistencies add up, and unless it's some type of creative writing or a painting, the result is big and ugly. Picture a house built in ten different styles, or a car made up of several different models. The 'essence' of a mess is inconsistency.

Software development projects are always big. Big projects require big teams. And big teams need to work together. If they don't work together, the result is a big mess.

I have seen enough projects where the teams are maintaining hundreds of thousand lines of repetitive, redundant and inconsistent code when less than a quarter of that should have been enough. It is easy to see the effects of that type of problem, as the team wastes more and more effort on weak testing strategies and chasing inconsistencies throughout the code. It is a black hole for resources. You can't fix the problems if you are too busy slapping on band-aides.

By its very nature, building a software tool is difficult, but not, as many programmers want to believe, because of mapping functionality onto specific computer language code. That part of development is relatively straight-forward, the difficulties come from analyzing the user problems and choosing the appropriate functionality to implement. Well, they also come from dealing with a messy code-base, but that wound is entirely self-inflicted.

The analysis problems are unavoidable, but ultimately they are not related to programming. Once a team has decided on how to 'solve' the problem, the rest of the effort should just be work. Nothing but work. A group of professionals using their collective knowledge to come together and assemble the appropriate instructions to implement some functionality on well-known data.

In the midst of this, there are always changes, but those small unavoidable shifts in understanding can be easily managed across the lifetime of a sophisticated software product. If you were close to begin with, the changes are never really significant.

The really huge problem with our software projects is our own tragic culture of uniqueness.

It is not, in any way, shape or form, that I want to take away from the human individuality. We are all individuals, and we shouldn't be forced to band together as helpless sheep or powerless ants. But, and here is a huge point: it is that very sense of individualism that is destroying so many projects. The blame, often fails on the guy down the hall, or the manager in charge, or someone else, but the 'problem' at the end of the day is the inconsistent mess created by the programmers that is not working correctly.

It is inevitable that if a team of developers flails away at their keyboard, without working in some harmony, the code will become a mess. Developers will leave, managers will change, these things are normal. If every programmer is silo'ed into their own over-protective code-possessive region of the project, the inevitable changes and shifts are going to bring pain. Quite possibly disaster.

A team that is not working together, that does not share all of their code, that does not follow similar standards, etc. is a defective team. One that is just waiting for trouble. And the effort of a defective team, not unsurprisingly, is always a weak and fragile project.

Yet, even as we all know this intrinsically, and so many of us have had this as a direct experience in our professional lives, we still bend towards the dark side. We don't want to be seen as just 'resources', or to be 'interchangeable', or even 'replaceable', but by not being these things we are sabotaging our own projects.

Presumably, if you get any good professional who follows their industry's best practices properly, they should be able to perform the job for which they were commissioned. There is variation to some degree, but ultimately the very definition of 'professional' means that everybody lives up to some recognizable standard; by its nature that means they can be replaced.

Your life shouldn't rest on the skills of your lawyer, your doctor or your accountant. Scientists, mathematicians and engineers shouldn't be able to set their disciplines back hundreds of years because of incorrect theorems or approaches. You should be able to hire a competent professional to accomplish a specific job for which they have knowledge and experience. And yes, these people, are 'resources', that are 'replaceable'.

There is nothing short of insanity, that would lead one to believe that 'programming' as a practice was anything other than a professional activity. Don't confuse vision, analysis or leadership with programming, those are wishy-washy irrational things, not unlike business acumen. Programming, itself is just the act of taking the desired functionality and encoding it. It is a task that can be accomplished by any well-trained professional programmer.

There should only be one good standard practice of doing this. Some variation, of course, but a team of programmers should come together, pick their conventions, and then stick with them. Thus, beyond all of the work and the team itself -- if it is functioning correctly -- should be a team of 'resources' dedicated to getting the job done correctly. Losing one member, aside from the morale or leadership aspects should not change the outcome of the team. That would be bad. That would be unprofessional.

In truth, although the overall direction might change somewhat, a different analyst, product manager or even visionary leader in a well-healed team shouldn't jeopardize the overall efforts either. All professionals bring with them some minimal level of functioning. That should be enough to keep the project moving forward, even if it suffers some change in direction.

Catastrophic problems occur, but they are individual issues. A bad hire is a bad hire, not proof that a discipline is too chaotic to be standardized. Or too radical to be repeatable.

I do believe that a good product needs a strong unique vision, not a committee, to steer its direction. Consistency starts at the top, but even great direction and leadership at the higher levels fall short if there is chaos in the ranks. Once a development path is forged, the implementation team needs to follow it, not wander around aimlessly exploring options.

Filling in a blank is quite different from re-interpreting the design. A product is only unified if its developers are; inconsistent code sucks; inconsistent interfaces suck.

Honestly, I don't want to work with any more 'irreplaceable' programmers. I've been there, and it is bad. You get some programmer that is so far out on the edge, that virtually nobody else can understand or fix their code. That is, by any definition, crappy code.

If your team can't live without a specific person, then your team is dysfunctional. It is that simple.

This is true at most levels. The best managers know how to make themselves a dispensable part of the process, it is the poor ones that are so controlling that they hide the details from everyone else. It is absolutely true for technical people as well. Those people that horde their knowledge, hiding in their cubicles, smug in their own superiority, are not the ones you really want in a well-functioning team. I'd gladly trade the best of them for a good team player any day.

It is intrinsic to the very definition of a well-functioning team, that its members are 'resources'. It is intrinsic to the very definition of good code that it is readable and can be maintained by any competent programmer. It is intrinsic to a software development project that everybody is working towards the same goal. These are not good ideas, they are requirements; things that need to be urgently fixed if not true.

You can blame management all you want for various failures, but if you're not willing to play nicely with the other developers in one big sand box with all of the toys, then you're probably more at fault than you could ever realize. Programmers never work alone.