There are five basic stages of software development:
Analysis
Design
Coding
Testing
Deployment
These stages are the same whether or not the system is brand new or just undergoing the next round of development.
Analysis is all about gathering the basic facts around the problem and any other related information necessary for the solution. The keys to getting it right are to gather precise details and to organize them for later use. This not only includes the functionality, but also any data involved, environmental restrictions and any other systems to integrate with. This should also include some understanding of why the users need their functionality and how they are doing it currently. There are many ways to organize this type of analysis such as requirements or user stories. The analysis should not only be focused on the user's needs, but also the operational and developmental issues as well. Many projects fail due to missing or sloppy analysis.
Design is the act of taking all of the analysis, along with any existing functionality, and structuring this information in a consistent manner. The higher 'meta-levels' of the design form the architecture. A design can be expressed as diagrams, tables or descriptive text that lays out the various peices that need to be created. The depth of the design should be contingent on the experience and abilities of the available programmers. With more experienced coders, the design does not need to descend right down to the low-level details. Design is the creative part of software development, but it is best to prototype any questionable design elements first before relying on them. Also, designs for rather standard systems need only be rather standard themselves, and knowledge or research really help with being able to constrain the ideas to something feasible. The key to a good design is to tightly organize the work, while only spelling out the minimum of detail. The development environment has a major affect on the design and architecture, so it needs to be factored into the output. Extensions to the system should utilize the existing code base, rather than just add to technical debt by slapping new feature onto the side. A lack of design always results in a lack of an architecture, which enables disorganization, that eats up unexpected but necessary resources.
Coding should be the least exciting step in software development. There are lots of little problems to solve, but the work should be dividable and should all come together at the end because of the architecture and any earlier prototypes. Unfortunately, programmers frequently rush into coding too soon, so they get delayed by forgotten analysis and design issues, often called scope creep. Coding can also be the longest step in development, which is why it is so important to leverage 'reuse' effectively to save time wherever possible. Coding sometimes gets caught into a rut, where fixing one problems causes others. Again, this is a primary indicator that the architecure was poorly thoughtout or does not correctly match the development environment. There is some testing that occurs in the coding stage, specifically unit testing, but it is never a substitute for a full systems test. Starting the coding stage by doing large, non-destructive refactoring is an effective way of controlling technical debt and getting the development set on a stable foundation. A haphazard coding environment leads to fragile code which ends up wasting time in the next two stages.
Testing is at best probabilistic, in that you can't fully test what is usually an infinite number of possible inputs. It's also the stage that is most frequently reduced due to delays in coding. As such it is both important to focus the testing on finding obvious or embarrassing bugs, but also allowing for patches to be released in the deployment stage. Before testing has occurred, the source code should be frozen so that no accidental untested code leaks out. Most releases should be on the main trunk of development, leaving branches for bug fixes or longer term development. Automated testing is best, but for things like GUIs, manual testing may be more cost effective. A tremendous help is to have an established architecture that allows any chances to be properly scoped so that the tests can be focused only on the parts of the system that have changed recently. Frequent full regression tests should be performed when in doubt, or just occasionally to maintain quality.
Deployment is the most forgotten part of software development, so that frequently the installation or upgrade instructions are tediously long and over-complicated due to not having enough time available to properly automate them. For commercial systems it is normal to have many different versions all out in active use at the same time, so proper/unique version numbering is crucial to minimizing support. If the design and coding when well, support issues should be minimal, but again there are frequently ignored requirements, so many systems are very fragile to external unexpected events. As well as releases and upgrades, deployment always needs at least one super-fast way of issuing critical patches, to find unexpected bugs.
If the five stages are done as large sequential tasks, the process is considered to be waterfall. This works well for large well-defined projects since it maximizes the efficiencies, but does not support radically shifting specifications or projects that have skipped analysis and design. If the stages are done as a series of short iterations that allows maneuverability, but it is somewhat less efficient. A well running development shop can overlap a number of different iterations in different stages at the same time. That is one group is doing analysis for a new feature, the next is in design, another is coding up the previous design work, while another is testing the next release. The stages overlap nicely, but it takes some experience to keep the whole pipeline running smoothly. Projects should not be high stress, 'hurry up and wait' affairs but rather should just be progressing at a smooth, stable, reasonable pace. Panic development begets bad choices which decrease intelligence and efficiences. The total length of an iteration is dependent on the work being done. Larger changes need more time to get completed. Scheduling should take this and seasonal issues into account for any parallel iterations.
Most development is routine, and if accepted as such the process of building and extending large systems is enjoyable. High stress environments feed opportunities for short cuts which feed back into the viccious cycle causing poor quality and burnout. Quality is all about how well people attend to all of the little details, so silver bullets and shortcuts come at the cost of heavily reducing the quality of a system. Getting good programmers to work on bad systems is very difficult, so any initial quality problems usually continue throughout the lifetime of the project and shorten its lifespan.
Any process employed to help development should be one that actually helps the developers rather than hinders them. It is easy for a management consultants to draft up a plausible sounding process, but if the focus is not on the actual deliverables for each of the five steps, then it will mostly likely result in extra make-work and take away focus from the main objectives. Since each environment is essentially unique, any process should be built from the existing senior development knowledge already there, so that it is available to help the work rather than hinder it. Also, any sort of deliverables, particularly documentation should be finely scoped to its usage. That is, business documentation should be written to be read by business people and technical documentation should be written to be read by techies. Working backwards from a definitive goal for the deliverables is an effective way to see what has value and what is just make-work.
Poor quality, unfinished work, fragility and other problems are all direct signs that the overall development process has serious problems. Adding arbitrary rules or extra oversight just makes the problems worse. Prolonged bad process kills morale and is very tough to reverse since it deeply effects morale. A broken environment makes the work considerably more expensive and of lower quality. Setting up good engineering practices early makes a huge difference, but also watching for signs of a bad process and correcting it quickly can really help.
