tag:blogger.com,1999:blog-6104420435021904082.post5013039363151870358..comments2024-03-13T12:21:27.016-05:00Comments on The Programmer's Paradox: Software BlueprintsPaul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-6104420435021904082.post-15412895947169515482009-01-25T10:16:00.000-06:002009-01-25T10:16:00.000-06:00Hi Sam,Thanks for your comments. Your right about ...Hi Sam,<BR/><BR/>Thanks for your comments. Your right about developers constantly working in new and unknown domains. The problems would be less if the industry valued experience properly, we apprenticed on each new domain/technology or the developers themselves took their own inexperience into account. <BR/><BR/>Although the technology does change, after years of programming you do find that most of the time, most databases and applications are not really all that different from each other. For most systems we write less than 20% of the code that is actually difficult. The other 80% exists to hold it all into place. Relative to a specific technology and user domain, I've found that most of building a new system is routine (and more so if your development paradigm is brute force). <BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-34132284887733852732009-01-25T07:07:00.000-06:002009-01-25T07:07:00.000-06:00I could be wrong here (and perhaps I didn't see it...I could be wrong here (and perhaps I didn't see it in your argument), but it seems to me as if developers (and their schedules) suffer almost consistently from "I've never built that before, but I'll give it a shot" mentality. It's a part of the job description, so let me explain it by way of counterexample:<BR/><BR/><B>You would never hire a residential real estate developer for a skyscraper startup.</B> I think this should go without saying in software engineering, but it doesn't. As developers, we're *paid* to design and build the equivalent small straw huts, residential real estate developments, skyscrapers, roads, bridges, turngates and turnpikes, etc. It is a vastly different problem domain to go from residential housing development to skyscraper development to church development (and the requisite architecture needs of each). Also, skyscrapers can't be upgraded or structurally altered, whereas software undergoes such upgrades *all the time* (witness how many times Apple revs Mac OS X and the iPhone OS as an example).<BR/><BR/>Stuff comes up all the time in development of the individual component widgets that build *said* structures. We are in a perpetual conversation with the compiler and the libraries we use (some devs would use the term 'argument'). If I'm uncomfortable with things like recursion, design patterns, regular expressions, OOA/D/P, etc... well, it's going to be many long and painful days. Even if I'm comfortable with them, there are simply days where it's going to be a hard slog. There's no getting around that. The compiler's opinion, while not final, is conclusive on every build.<BR/><BR/>I like the organic garden growing metaphor the best - it shows the accretion of development as well as dealing with the terrain of the land (as developers compare their experience of developing for the Web vs. iPhone vs. Windows vs. pure UNIX).<BR/><BR/>Thanks for a great article.<BR/><BR/>Just twopence worth....Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-28671805216928296882008-05-14T21:27:00.000-05:002008-05-14T21:27:00.000-05:00Hi JS,I think we need more detail than a sketch, b...Hi JS,<BR/><BR/>I think we need more detail than a sketch, but less than 'all of it'. For that the medium might be UML (or sloppy UML -- not quite so pedantic), or anything that can convey the major details without getting bogged down in it's own complexity. Presenting it is one problem, but actually deciding what is in or out is probably harder.<BR/><BR/>Most blueprints I've seen (far too few), are essentially 2D representations that are precise about the core details, but say nothing about those things covered by conventions and standards, such as studs and electrical sockets (I could be wrong about that). In a way it doesn't matter in so far as it is both the minimal and maximal amount of information a professional needs to get the work completed in a specific fashion.<BR/><BR/>I like your proposal, the question is how to describe the system? Also, to what level of granularity does it need to go? What is interesting in construction is that 2D works for a 3D building. That says a lot about how much or how little you need to get 'it' without having to see all of it. <BR/><BR/>Besides pyramids, I think most modern building designs are essentially scaled up in this fashion. They seem to start with something existing and then extend it, rather than go all of the way back to first principles each time.<BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-26871627878252335772008-05-14T19:36:00.000-05:002008-05-14T19:36:00.000-05:00Hello Paul,Are you proposing then that we need som...Hello Paul,<BR/><BR/>Are you proposing then that we need something analogous to an architectural sketch? As far as this goes, where do the UML diagrams fall short? (Mind you, I hate UML diagrams.) You get to see the big picture of how the system objects fit together. Or is this too much detail?<BR/><BR/>Here's what I propose we do: identify the smallest possible software system that is still large enough that some sort of system design is meaningful and explore what the minimum necessary information to describe the system so that someone can understand how it works without having to resort to inspecting the code or reading through all the program specs. Once this process kind of works, it can be validated against other 'simple' software systems, and eventually scaled up. As that's how the pyramids were built.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-78754380032083317972008-05-14T10:04:00.000-05:002008-05-14T10:04:00.000-05:00Hi JS,That is an excellent comment. But for each o...Hi JS,<BR/><BR/>That is an excellent comment. But for each of the known solutions you mention there are reasons why they haven't had 'significant' traction in the software industry. For #1: not all programmers can (or should) design, and testing just means the code works, not that it is right. For #2, well documented code is easier to extend, but still it doesn't mean you've built the right thing. #3, formal verification is way too expensive, time-consuming, and not fun enough for most programmers. And #4, I assume is similar to #3, but also there are very well-defined specifications (overly-detailed blueprints), so it's easier to be formal. You need to verify against something.<BR/><BR/>The common thread is that all of these are 'post' design, ways in which you can assure that what you've built is what you expect. That's great, but our industry allows lots of bugs, so it's cheaper just to throw it out there and fix what breaks. It's a cost issue. <BR/><BR/>My desire for blueprints is driven by looking for a way to both validate the core aspects of the design "before" building it, and by leveraging my own design capabilities so I can build larger systems with what limited time I have. In that sense, the blueprints are only useful if they are detailed enough to validate the design, but not enough to actually generate the code (otherwise we are back to Jack Reeves' ideas). That's why, even thought I don't know what they should be, I figure that if I can create them "fairly quickly", and hand them to you, you should get a mostly 'complete' sense of what the system is doing, how it works and how you should build your piece to fit into it.<BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-12085303787665212352008-05-14T02:23:00.000-05:002008-05-14T02:23:00.000-05:00This doesn't make any sense. You talk here about ...This doesn't make any sense. You talk here about not learning from the past, but it seems you ignore a good deal of useful approaches that are known to work already. I can name several off hand: <BR/>1) Fred Brooks in the Mythical Man Month mentions two important ideas: a) having an overall design for which exactly one person is responsible and b) using a significant amount of test code to automatically validate that software continues to work as changes are made.<BR/><BR/>2) Don Knuth invented literate programming. The idea here is to treat code and the necessary documentation as one unit where the small amount of source code is embedded in a well written document designed to be read and understood by people. The code is extracted from the document by automated tools during the compilation stage.<BR/><BR/>3) Dijkstra of 'Goto considered harmful' fame promoted the use of logic and formal verification methods to prove the correctness of code. Pre, post and invariant conditions can do wonders for ensuring the correctness of code.<BR/><BR/>4) The aviation industry and others that need safety critical software produce code that just plane works. They use certified compilers along with formal verification methods to guarantee code works.<BR/><BR/>This is a problem with known solutions. <BR/><BR/>One other point: if you can build a blueprint for a software system, why not generate the code from it rather than reimplementing it in code?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-70142134055377195632008-05-12T10:46:00.000-05:002008-05-12T10:46:00.000-05:00Hi Andreas,Thanks for the comment. I'm not so sure...Hi Andreas,<BR/><BR/>Thanks for the comment. I'm not so sure that UNIX is more complex than a skyscraper. The elevators, electricity, plumbing and HVAC systems in skyscrapers are all phenomenally complex, and in the case of elevators at least, highly computerized. Just dealing with the weight of the structure, depending on soil conditions is hugely complicated. As well, watching a well-run project manage all of the machines and the hundreds of on-site contractors as it goes from a giant hole in the group to a final complexed structure is inspirational. They always look so organized when I peek through the fences, and there are few stories of accidents. We tend to overestimate the complexity of our own work -- because we know the details -- while underestimating that of other disciplines. Software is just far messier than it is inherently complex.<BR/><BR/>It's probably only fair to compare one 'instance' of a skyscraper to one 'release' of software. The design and construction crew 'move' on to the next building, while the software guys move on to the next iteration. The new blueprints build on the previous understandings. <BR/><BR/>There are certainly some huge differences, but I see that ability to 'verify' the core aspects of a design before committing to building it as being a crucial thing that is missing in software. Who wants to spend millions only to find out later that the system wasn't scalable? The 'real' cost of even the cheapest software (design, implementation, deployment, and management) is always very expensive, but programmer's hate to admit that.<BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-14939195744115231532008-05-12T03:42:00.000-05:002008-05-12T03:42:00.000-05:00The great flaw, I think, in this perspective is th...<I>The great flaw, I think, in this perspective is that a huge advantage of doing a blueprint is to check to insure the major bits are all correct and in place long before the work starts.</I><BR/><BR/>I think there is an misalignment here: 'The work' in skyscraper building is casting the ideas into concrete; in software it is firing up the compiler: Executing the design is just massively faster. This comes true in physical engineering as well; thanks to simulations the design verification goes much faster. (Admittedly that is not the same as the actual construction.)<BR/><BR/>But keep in mind that unix is massively more complex than a skyscraper; the skyscraper has many units of the same kind, and its blueprint is mainly bookkeeping of those (where are the outlets on each floor, where are the struts). Unix consists of a lot of different things; the closest physical analogy would actually be the machine floor of a factory where each machine has a different and separate design.<BR/><BR/>The bookkeeping also comes in the way of usability of blueprints. It is less work to change the blueprint for a moved outlet than to move the outlet itself, but it is often more work to update the blueprint/documentation than to do the corresponding change in the software itself.<BR/><BR/>If the software blueprint isn't automatically verifyable against the actual system, it is pretty unavoidable to have it fall out of sync. Even then I think the goal is not to have a separate blueprint but to make the code itself more blueprint-like.Andreas Kreyhttps://www.blogger.com/profile/18011171943440483489noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-31284364292608811762008-05-06T18:24:00.000-05:002008-05-06T18:24:00.000-05:00What is lacking in software development is an effe...What is lacking in software development is an effective modelling language, one that is neither to low level to be useful for planning nor too high level to lead to useless blueprints.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-91032983083918564042008-05-06T10:14:00.000-05:002008-05-06T10:14:00.000-05:00Hi Jack,Thanks for continuing the discussion. That...Hi Jack,<BR/><BR/>Thanks for continuing the discussion. That is a great point, but possibly one that lessons with time. We build fewer and fewer all new first-generation systems every year. More commonly, something already exists so we are just building a replacement or enhancement. We've built up (and ignored) a lot of knowledge and analysis over the last fifty years.<BR/><BR/>Even when we do create something from scratch, iteration can only take you so far, you can't start with an accounting system and iterate your way into a social networking site. Some things may grow and change, but software is not really that flexible. To me, if your 2.0 version tosses out 'all' of the functionality from your 1.0 version, then that is an analysis failure. Equivalent in a way to having built a skyscraper on the wrong site. Rebuilding it is probably a better (and cheaper) option than moving it.<BR/><BR/>Hubris or not, I usually don't start writing until I think I am close enough to the final solution. Sure, it changes, but hopefully not radically. My bigger problem is usually lacking the time and resources just to get the basics built. It is the changes due to resource-based compromises that so often haunt me later. I often know how to do it right, but lack the oppurtunity. Thus my desire to leverage myself.<BR/><BR/>Interestingly enough, I think 'how much blueprint is enough?' at least in its initial stages is probably subjective. For myself, I want to control the major attributes of the system (standards, code, appearance, functionality), but I also know its important to let the other programmers get some of themselves into the design as well. Kinda like a <A HREF="http://theprogrammersparadox.blogspot.com/2008/03/controlling-development-chaos.html" REL="nofollow">movie director</A>. <BR/><BR/>I'm not looking for a way to specify 100% of the design. I want the plot to remain intact, but the main actors should be allowed to add a bit of depth to the overall project. It's a bit more flexibility I think than construction would allow.<BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-49951760569969505592008-05-05T21:38:00.000-05:002008-05-05T21:38:00.000-05:00I think you may somewhat have mistaken my point, "...I think you may somewhat have mistaken my point, "over there." What you say about the desirability of a blueprint is all fine. My point, though, is that in actual practice we most often don't actually know what we want in the first place. We almost invariably think we do, but then we get it, and see we've left something out, or we really didn't want something else, or something in the environment has changed while we were heads-down slaving away.<BR/><BR/>This is not universally so. There are certainly cases where we know well enough in advance to justify the blueprint. And there are certainly cases where we're obligated to figure that out. I would certainly agree that systems with human safety concerns qualify -- train routing, fly-by-wire aircraft, stuff like that. The penalty for being wrong, here, is sufficient that we darned well better know what we want, or we've no business making the system in the first place. And if we change our minds, we need someone yanking our chain back to the process. And if the environment around us changes, shame on us for not controlling that variable.<BR/><BR/>Your exercise in drawing the fine line of "how much blueprint is enough" doesn't seem to reach practicability. "Enough so someone else can create the product," you say, but there are all these questions open as to how to know whether the thing so-produced actually is the thing you meant (leaving quite aside "realize you should have meant, now you see the outcome").<BR/><BR/>So, "over there," I championed evolutionary-design systems, like Agile and Open Source. That doesn't mean no thinking happens up-front, but it does mean we actively avoid the hubris of thinking we know it all before we've actually met it.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-76537225098250209512008-05-05T12:40:00.000-05:002008-05-05T12:40:00.000-05:00Hi Rob,Thanks for the comments. Your first questio...Hi Rob,<BR/><BR/>Thanks for the comments. Your first question might just be a chicken and egg type dynamic: a skyscraper has to be stable, so maybe they aren't allowing last minute changes? Or possibly, the design itself is just flexible enough to accommodate any renovations that people need to make?<BR/><BR/>I'd like to think that blueprints could work wherever major changes were being implemented. If they are useful they won't be a huge amount of extra work, and they'll pay for themselves by ensuring that the changes are reasonable.<BR/><BR/>On a related note, the office building next to me appears to be extending itself by at least three floors, if not more. If they screwed up their calculations and the thing comes tumbling down, I'm right at the base of it (so I really really hope changes and enhancements in big buildings don't destabilize the initial core work). <BR/><BR/><BR/>Paul.Paul W. Homerhttps://www.blogger.com/profile/02349253120538728302noreply@blogger.comtag:blogger.com,1999:blog-6104420435021904082.post-50110946647697006522008-05-05T11:55:00.000-05:002008-05-05T11:55:00.000-05:00Great article!I wonder if skyscrapers would be so ...Great article!<BR/><BR/>I wonder if skyscrapers would be so stable if there were constant enhancements and requirements changes made to them? I think it should be possible to blueprint the first "release" of an application, but the changes and enhancements made to that application by new teams of people who probably do not understand some of the underlying dependencies within the application will inevitably make the end result less stable.<BR/><BR/>I assume the same thing would happen to a skyscraper if a requirement arose to add another 10 floors to it without rebuilding it from the ground up, no?Anonymousnoreply@blogger.com