In the land of the blind, the one-eyed man is king.
[In regione caecorum rex est luscus.]
Desiderius Erasmus, Adagia (III, IV, 96)
Dutch author, philosopher, & scholar (1466 - 1536)
When you first get into building software it is exciting. There is a definite thrill in gaining a new found ability to create things, watch them run, and see people start to use them. I’ve always felt that it was this ‘high’ that keeps me coming back -- again and again -- even though software development is so plagued by other annoyances.
But some days it can feel like I am dragging around a big cart behind me, filled with people kicking and screaming, while beating me on the back of the head as I am trying to focus on how to get us out of the hole and onto our destination as fast as possible. It can be a thankless job some days (when they’re not heaping false praise upon you for lesser accomplishments), but that is another story, for another day.
When you first start programming, of course you focus in on the code. It is on the small problems, which are fun and interesting. How do I manipulate this string? How to I can I call this library? How do I make this screen pretty? How do I store this data? And it is too easy in the midst of all of this to see the ‘code’ as the most important thing.
In my early years, I felt that all I needed was time. And with time I could get code, and with code I could solve any user’s problem.
But age and experience gradually creep up, and after many many experiences of seeing that the code itself didn’t save the ship, or make the user’s happy, or contribute to a career, you start to see software from a completely different perspective. It’s at that point that you awaken one day to the larger context that is swirling above the code.
There is this sudden rush as your other eye opens. You come to realize that the best algorithms, coded in the best ways, and tested with the best results, still won’t allow you to win if winning isn’t possible. You realize that any and all technologies -- no matter how slick or smooth -- are fraught with an endless series of hastily coded landmines just sitting there, waiting for you to take a fatal step. You come to realize that a poorly coded system that makes itself easy for the users has way more value then something super-complicated with buckets of functionality but is entirely cryptic, badly organized or untrustworthy. You come to understand that while the implementation is important, it’s not the ‘thing’; not the show, and not necessarily going to mean the difference between success and failure.
But most of all, you wake up to the fact that the ‘code’ itself is just a part of the puzzle, and that the algorithms, while fun to solve are the small problems, floating is a sea of large and massive ones.
I’ve watched more programmers than I care to admit retreat into the small problems and corner-cases to avoid the big ones. I’ve watched them throw all of their coding weight into extended blitzes hoping to save an already dead project. I’ve watched them burn through rolls of virtual duct tape in the faint hope that at some point they’ll get to the other side. I’ve watched them ignore the users, blame the management and generally try every trick in the book to avoid having to deal with the part of software development that they don’t like, or are afraid of. And I’ve never seen it work.
Programming is a part of software development. It is an important part and for many people it is the part that they are good at, enjoy and should focus on. But it is in no way the part that is going to carry the day. A well-built project is no less doomed than a poor one. The quality of code may become an issue at some point, particularly when it means that any new extensions are impossible to accomplish because the system has acquired way too much technical debt, but that type of premature death is just one of the many ways that our work dies, there are so many other hurdles that have to be jumped.
As is often the case, the larger issues have to be handled before the smaller ones can be successful. Architecture, design, process and consistency are necessary to keep the project sane. Empathy, politics, marketing and an understanding of the target market are necessary to make the software useful. Testing, packaging, installation and support are necessary to keep up the momentum and avoid getting bogged down by sloppy patches and dangerous work-arounds. And elegance, readability, simplicity and encapsulation are necessary to avoid painting one’s self into a corner at the later stages.
What algorithm you choose, and whether or not the you’ve applied a lot or a little micro-optimizations? These help, but at the end of the day they are just icing on the cake, or as is the case in many development shops, just icing on the floor.
Wednesday, March 9, 2011
Thursday, February 17, 2011
Big Data from Small Source Code
Editor’s Note: This is a guest post by Denton Gentry. He’s an experienced software developer and fellow blogger from California. This years New Year resolution for Denton was to cross-post to other blogs. He’s off to a good start. Read more from Denton at his blog Coding Relic.
"Nobody would be interested in this code, why open source it?"
"I'm not really looking for people to help out on this code, its just a simple little thing."
"I wrote this to learn the language, its not good enough for anyone to see."
I suspect that most developers are proud of the code they've written, but most of that code is owned by an employer and considered proprietary. We couldn't open source it even if we wanted to. Code written on one's own time and not related to the employer's business could be open sourced, but rationalizations such as these keep us from doing so.
I know I've used them.
The traditional reasons for open sourcing code revolved around collecting a community of like-minded developers to collaborate with. It is a lot of work to build such a project, and even then most such efforts fail to get traction and simply languish. Driving such a project would take a lot of time and effort, time not worth investing for something one doesn't feel truly passionate about.
The thing is, the world has changed.
When it comes to hiring, I'll take a Github commit log over a resume any day.5 Feb via web
jeresig
Very subtly over the last few years, we've transitioned to a new model for information on the Internet. Its a world of Big Data, where we can draw inferences from collections of data even if the individual pieces go mostly unexamined.
- Code search engines mean someone can reference your code to figure out arguments to a particular API, even if they don't use the code itself or care about the project it is a part of.
- Framework and platform developers can use statistics from crawlers to know what parts of their API are widely used versus what is not getting traction. This can inform decisions about deprecation or API evolution.
- Aggregate volumes of code can compare the rise and fall in popularity of different programming languages, useful information for developers looking to keep their skills current.
- Resumes are dry reading, and get stale. Code commits, blog posts, etc are a living resume of one's work and skills.
- Also, quite frankly you may be surprised at how many people benefit from a posting of code. Someone stuck on a problem will go through many pages of search engine results looking for an answer.
The question now should be if there are any reasons not to publish the source of personal coding projects. Is there a reason to keep it to private? Tools like GitHub and Stack Overflow have made programming into a social activity, with new opportunities for personal and professional advancement. The only cost in taking advantage of them is time, and even at that only a little time.
The question now of whether to publish the source revolves around internal factors, not external. "Am I proud of this code?" not "will anyone use this code?"
Tuesday, February 8, 2011
Top-down and Bottom-up
When I was younger, there was a lot of debate about how to design and build software. The prevailing theories were that you either started at the top and worked your way into the details, or that you started at the bottom and built up the pieces.
Software systems are tools used by people, so it is important to understand what their problems are and what is necessary to solve them. The only way to do this, is to see it from their perspective. If you understand what the user is trying to accomplish and how they are doing that, you can find the best design that simplifies their lives.
Functionality should not be arbitrary buried in dis-organized menus, it needs to be at their finger-tips, right when they need it, and out of the way the rest of the time.
Design then, only comes together if you’re looking down at the problem. You have to start at the 10,000 foot view and then wind your way through all of the steps necessary for someone to complete their work. Empathy for the users, and a deep understanding of both their environment and their goals is key to creating the tools that actually make their lives better.
But, from the other perspective, software is extremely slow and expensive to write. As the user expectations have increased, even the small systems of today are significantly larger then just a few decades ago. And the more we depend on underlying libraries, the faster the complexity increases. Even if we don’t have to write all of the code, each underlying dependency brings with it a unique set of problems that requires time to understand it and time to manage it properly. Declining standards, poor design, forgotten knowledge and sloppy release procedures in these external pieces don’t help.
Our only defense against spiraling complexity is to try isolate the work as much as possible. That is, highly redundant code that haphazardly calls the underlying components, anywhere, is the type of spaghetti that quickly burns through all of the development resources by wasting time chasing sloppy mistakes. A well-architect-ed system that correctly encapsulates all of its underlying parts allows for the scope of change to be controlled. Once a problem has been settled, it should no longer crop back up in arbitrary places. Each piece must fully encapsulates a specific section of the system. This is the only way to provide a solid foundation on which bigger and better functionality can be added without fear of setting off a chain reaction of cascading bugs.
This type of design can only be achieved by building upwards. Starting with the depths of the system, not unlike a real building, each layer is carefully designed, built and stacked. The lines between the layers need to be well-understood and carefully mapped out. This type of bottom-up construction insures that the whole does not become unstable as it gets pushed and extended. It can also insures that the momentum of the project doesn’t grind to halt because of exponential explosions in complexity. It requires a bit more effort and foresight, but it is the only way to build large systems that are dependable and can continue to grow as the needs of the users increase.
So, is it top-down or bottom-up? Really it is both. The design and requirements need to be seen from the top, but the construction needs to focus on building up solid and reliable pieces from the bottom. Foresight and experience are necessary to understand the scale of the system, but projects still need to be adaptable to changes in the environment, technologies or the user’s priorities. A solid foundation is necessary to insure that the system will survive for its maximum lifespan while continuing to grow.
Although our industry pretends to turn over every five years, by now it is clear that the life expectancy for big systems is ten to twenty years. And for some systems, they may be around far longer than that. With that understanding of our history, it seems even more reckless to hack together something without a design, apply duct-tape excessively as leaks appear, or just hope to catch the flaws in testing. Good software development demands a longer-term focus as well as seeing the whole process from both the ground and the aerial view.
Software systems are tools used by people, so it is important to understand what their problems are and what is necessary to solve them. The only way to do this, is to see it from their perspective. If you understand what the user is trying to accomplish and how they are doing that, you can find the best design that simplifies their lives.
Functionality should not be arbitrary buried in dis-organized menus, it needs to be at their finger-tips, right when they need it, and out of the way the rest of the time.
Design then, only comes together if you’re looking down at the problem. You have to start at the 10,000 foot view and then wind your way through all of the steps necessary for someone to complete their work. Empathy for the users, and a deep understanding of both their environment and their goals is key to creating the tools that actually make their lives better.
But, from the other perspective, software is extremely slow and expensive to write. As the user expectations have increased, even the small systems of today are significantly larger then just a few decades ago. And the more we depend on underlying libraries, the faster the complexity increases. Even if we don’t have to write all of the code, each underlying dependency brings with it a unique set of problems that requires time to understand it and time to manage it properly. Declining standards, poor design, forgotten knowledge and sloppy release procedures in these external pieces don’t help.
Our only defense against spiraling complexity is to try isolate the work as much as possible. That is, highly redundant code that haphazardly calls the underlying components, anywhere, is the type of spaghetti that quickly burns through all of the development resources by wasting time chasing sloppy mistakes. A well-architect-ed system that correctly encapsulates all of its underlying parts allows for the scope of change to be controlled. Once a problem has been settled, it should no longer crop back up in arbitrary places. Each piece must fully encapsulates a specific section of the system. This is the only way to provide a solid foundation on which bigger and better functionality can be added without fear of setting off a chain reaction of cascading bugs.
This type of design can only be achieved by building upwards. Starting with the depths of the system, not unlike a real building, each layer is carefully designed, built and stacked. The lines between the layers need to be well-understood and carefully mapped out. This type of bottom-up construction insures that the whole does not become unstable as it gets pushed and extended. It can also insures that the momentum of the project doesn’t grind to halt because of exponential explosions in complexity. It requires a bit more effort and foresight, but it is the only way to build large systems that are dependable and can continue to grow as the needs of the users increase.
So, is it top-down or bottom-up? Really it is both. The design and requirements need to be seen from the top, but the construction needs to focus on building up solid and reliable pieces from the bottom. Foresight and experience are necessary to understand the scale of the system, but projects still need to be adaptable to changes in the environment, technologies or the user’s priorities. A solid foundation is necessary to insure that the system will survive for its maximum lifespan while continuing to grow.
Although our industry pretends to turn over every five years, by now it is clear that the life expectancy for big systems is ten to twenty years. And for some systems, they may be around far longer than that. With that understanding of our history, it seems even more reckless to hack together something without a design, apply duct-tape excessively as leaks appear, or just hope to catch the flaws in testing. Good software development demands a longer-term focus as well as seeing the whole process from both the ground and the aerial view.
Subscribe to:
Posts (Atom)