Roles and Responsibilities

One thing I’ve noticed about the software industry is that there is no unified base of definitions or best practices. 

Pretty much, everywhere you go, all of the programmers in different domains or markets have their own unique view of what the consider is standard and proper. From a global perspective, this means that basically every programmer’s view of what is right, and what is wrong, is eclectic. With almost no commonality, this leads to a wide range of quality and interconnection issues within our industry. Basically, we are disorganized.

That is something that I suspect isn’t going to change for a very long time. Our industry isn’t mature enough yet to be able to standardize itself. Many people believe they know how to do it properly, but few even grasp the scope of the effort to align their ideas with others.

For this post, I thought I would just list out the various roles that I’ve encountered while developing software. I’ve worked on in-house, consulting and commercial projects for 11 different companies, in five different domains. Over the course of that last twenty-five years -- including my co-op student days -- I’ve worked alongside or discussed development issues with a huge number of people. What and how to build systems has always been a keen interest of mine. From that experience, spread over a wide scope of differing opinions and viewpoints, I’ve built up a set of roles that I have found useful. Given the lack of consensus in our industry, I’m sure that these roles and definitions will not be in wide agreement, however I think it’s important to have some breakdown of the different skill-sets one needs to develop systems ranging from small to massive, even if they have different titles, or are partitioned differently. No matter what we call things, the same focus, decisions, analysis, work, etc. have to be done underneath.

So here is my list:

Business Analyst, Analyst, Domain Expert

Software is about solving problems, to get it working right requires a vast amount of tiny detail. So it’s important to have somebody in the project that deeply understands the workflow issues, domain issues and details involved. Most domains are complicated enough that keeping up with them or digging into the depths is a full-time job, which needs as much focus and concentration as coding.

Data Modeler, Data Architect

For any system that is large, it will likely need to be connected with many other systems. To do this, there must be a ‘universal view’ of the data, not just an application specific one. The most popular method of sharing large data-sets is via a common relational database, so the underlying schema for this needs careful thought and consideration or it will be unusable across the whole set of systems. Another growing trend is sharing via standards such as XML. Good modeling requires trading off correctness for convenience, which for one system is tough enough, but for multiple becomes a real problem that requires significant work in order to manage properly.

Computer Programmer, Programmer, Coder

Programming is the act of taking a description of some behavior on a computer -- in more or less detail -- and creating working code. There is inherent ambiguity in the description, which has to be solidified in order to make the system stable. Programmers focus on taking these descriptions and producing source code which can then be built and run. In smaller projects, with easier domains, the descriptions might be very vague thus allowing the programmers to dip into the analyses aspect, but since they often don’t have the depth of expertise, this leads to a huge number of changes driven by faulty initial assumptions.

Systems Programmer

Some types of programming require strict discipline, deep understandings, and very rigid logical thinking. The code is very algorithmic in its nature. This includes building operating systems, parsers, protocols, optimized calculation engines, and many other non-visual components. This also requires a significant amount of research in order to avoid spending time re-inventing the underlying knowledge from scratch.

GUI Programmer, Web Programmer

Some types of programming are almost entirely visual. The construction of the code -- its rigor -- is far less important than its presentation. By its very nature, this code is extremely repetitive, but generally does not involves very deep loops or concepts like recursion. Most really good GUI programmers don’t structure their code nicely since they fiddle with it constantly in order to affect the way it looks when it runs. It’s all about appearance.

Applications Programmer

The bulk of code out there is basically Edit Loops. The system presents data, the user modifies it and then it is saved. This isn’t as glamorous as other types of programming, but it has its own challenges, most revolving around domain issues, quality and scheduling.

Database Programmer

Although it is changing, relational databases remain a fundamental technology in most development. Although they started from theory, over the decades they have become quirky collections of vendor-specific behaviors. As such they require an unusually large amount of specific knowledge, that if ignored generally means that the utility of a database is mitigated by improper usage. Use them correctly and they help, do it poorly and it only makes the problems worse. Thus to get the most out of a database there is a need to get a programmer that has specialized in a specific vendor’s version.

Toolsmith

In a big project, the programmers are entirely focused on meeting deadlines. But they too rely on lots of complex software in order to complete their jobs. Most of this software needs special handling and significant configuration in order to help, rather than harm. Examples include build scripts, source code control, integrated development environments, document templates, etc. Any of the tools that are used frequently, and need to be sharpened in order to remain effective. Small projects generally distribute this work to the programmers (or don’t do it), while big ones need to have one or more people to fill this role.

Senior Programmer, Technical Lead

Any group of programmers will be naturally independent enough that left on their own, each programmer will choose their own unique process, design, and conventions. Left unchecked, this quickly degenerates into a mess at the systems level. To avoid this, programmers are usually grouped into teams, and the teams are each lead by a senior programmer who has a lot of explicit experience with the specific type of development underway. If the lead doesn’t have experience, or can’t get the respect of the other programmers, then problems generally build up to a fatal level. The larger the project, the faster the build-up, so sometimes in small efforts, while it is there, it is not noticed particularly by the participants.

Software Architect

As the number of people necessary to get the work done increases, the need for leadership and communication also increases. For a large project, with a number of teams, someone has to take all of the high-level decisions and focus them onto a coherent set of choices. Without that, the pieces themselves may work, but collectively they will be defective or unstable. The bigger the project, the more common choices that have to be made. Avoiding making these choices, just means duplicate effort that is unlikely to be synchronized, thus leading to massive (possibly unsolvable) problems when it is all brought together. Today's users have huge expectations for the quality and sophistication of their systems, which are most often beyond the ability of a single person, or even one small team to complete.

Software Developer

Building the software is still a significant problem, but it sits within a larger one. To meet its goals, software not only has to exist, but it has to be constructed within an environment and sold often to another environment. There are a huge number of wetware issues implicitly involved in getting it out to the users, version after version. Someone who can take a project from a vague concept, all the way to multiple versions (including distribution, packaging and support issues) is a full software developer. That is, they understand not just the context of building the code, but also how it fits into the larger picture.

Project Manager

This often misunderstood role isn’t about issuing orders. A good project manager spends their days running around to all of the different people, making sure that they are not being blocked in their work. A project is running smoothly when all of its resources are running smoothly. A project manager doesn’t need to have been a programmer, but they do need a deep understanding of the technical issues involved at all levels of the project, otherwise, they won’t be able to properly asses the inter-dependencies. A big project is often a spaghetti network of smaller work items, that need to be done in the right order, to minimize the effort. Pre-planning, foresight, trade-offs, and technical understanding are important to make sure things progress smoothly.

Product Manager

Although a system is built to solve a set of concrete problems, there are often a large number of related problems floating around that could be solved as well. It seems like an easy issue, but particularly for commercial software, each choice comes with a series of very dire trade-offs. If one feature it built, it means that others are not. All projects required funding (budgets or revenue), and to keep this going they have to satisfy a large number of irrational conditions. Someone needs to have the foresight to seer the effort away from the dangers, and towards a constant series of wins that maintain the current momentum. Without direction or funding, the project dies.

Chief Information Officer (CIO)

In companies that don’t sell their code, but buy or build their own, the highest technical operations position is the CIO. Their job is to ensure that the data needed for the business, is the data collected and organized by the IT department. It’s a difficult job given that the industry is never forth-right about the limitations of their technologies, and it is constantly moving. Sometimes this position is filled by a non-technical business person, in which case it is really a monitoring/reporting role for the people doing the actual work underneath.

Chief Technology Officer CTO

In a company that produces commercial software, or software services, someone must direct the overall vision of the development effort. Like the CIO, this role is necessary to perform long-term strategy. Often times, this position goes to a non-technical person as monitoring/reporting role, but for smaller companies, it generally is held by the lead technical developer, thus it can be a leadership, working and visionary role. For someone interested in building massive systems, this is the best role in the company to have (although it can be very hectic and stressful).

Operator

This role has largely disappeared, but it was a very important one. Often the computer can’t make a valid intelligent choice, so the decision is passed back to a human. They choose to accept something or escalate the problem. It is unfortunate that this role is disappearing because it provides a way of injecting some dynamic intelligence into the system, reducing its instabilities.

Systems Administrator

In the operations department, someone must look after each of the machines. Modern computers contain a vast array of low-quality software that interacts very poorly. Modern practices have also focused on releasing software at a faster rate then it can be correctly written, so there is a never-ending stream of new versions, some of which need to be ignored. As well, they are a plethora of security risks and many ways in which people can accidentally damage their own machines.

Network Administrator

Interconnecting a large number of computers together requires an organized topology. In any large organization, there are always old machines dropping out, and new ones coming it. As well, because of lax security considerations on the part of almost all software vendors, there is an ever growing, massive collection of ways in which malicious programs can spread and cause very real, and expensive damage. Keeping up with this shifting landscape is a huge job.

Relational Database Manager

For massive databases, relational technologies need constant tweaks in order to prevent the performance from degrading. This means a lot of monitoring, collection of statistics, analysis and then implementing improvements that maintain the health of the database. Gradually, the technologies have gotten better at doing this for themselves, but for really large data-sets they are still insufficient, and the work requires careful thought and deep thinking.

Operations Manager

Someone has to run the operations department. Sometimes they are ex-programmers, and sometimes they just moved up in the operations department. Operations is a reactive environment that is always shifting. Also, many people working in the lower roles are not in love with their jobs, so they don’t tend to be as prudent about their efforts. Someone has to keep an eye on the whole affair and make sure it is running smoothly.

Pre-sales Support, Pre-sales Engineer

In hardware or software sales, it is unusual to find salespeople with more than a very shallow depth of the underlying products. However, they are extroverts and can sell it quite well. Technical people, on the other hand, tend towards being introverts and aren’t particularly successful as salespeople. In order to bridge this gap, most companies team up a gifted salesperson with a technical one. This works well for both sides since each can focus on what they do best. Pre-sales support is a great role for someone who loves technology but doesn’t want the rigors of hyper-focusing, day-after-day on a building effort. It’s a great paying career, and often a more comfortable choice than moving into operations.

Post-sales Support, Second-line Support

Software needs to be set up initially, and once the system is in place, someone has to deal with its inevitable problems. This is a problem-solving role, that can involve deep debugging skills. While this position is often associated with commercial products, it is often an in-house equivalent.

Helpdesk Support, Front-line Support

Front-line support deals directly with the users for the commonly repeating problems. Most of them are user-based issues, but often they are just known software bugs that can go on for years and years. If a problem isn’t repeating (first or second instance) it is passed to second-line support. If the second-line support is organized, by the time the problem has shown up for the third time, there are instructions on how to correctly deal with it.

Data Administrator

Modern dynamic systems often involve a lot of customization that can be handled by the user interface. This creates a new role whereby one or more of the users or domain experts is nominated to track and modify these configurations as needed. This is not to be confused with database administrators who are working at a much lower technical level. This is essentially a privileged user role.

User

The user is any person that is using the system. Some users are actively adding or updating data, while others are management-based users that are monitoring activity or mining the data. It is worth separating the two groups, and also distinguishing between frequent users, and causal users, since the interface needs of the different sub-roles are completely different. The best systems, need very little or no training for the users to get value from them, but that differs depending on how they use the system.


Titles that I don’t like:

Stakeholder

This is a relatively modern term for all of the people who have some stake in the project. Sometimes this group includes the programmers, but often it doesn’t (although they do have a stake in the project if they are intending to work on it for the next five years of their lives). Aside from the image I always get of a waiter holding up a tray of T-bones, I think the term includes too many people to be useful. Many of the stakeholders want unnecessary control over the efforts, based on their providing funding. But once successful, they won’t be involved in being a user when it goes live. Software projects are rife with complex people issues that often center around who is controlling the process, and it doesn’t help to seed control to people whose depth is too shallow to make reasonable decisions. In order to be successful, the right people with the right knowledge have to make the right choices. Missing that is fatal.

Software Engineer

Our industry is so distributed and so chaotic that it is unlikely that we could even legitimately call ourselves a craft, let alone engineering. The first of those involves passing the knowledge on from master to apprentice, something that happens rarely right now, while the second involves writing it all down in an organized and usable manner so that it can be applied deterministically. We’d have to get through one, before we can arrive at the other. Currently we can’t even get a consensus on the basic roles involved. 

Hacker

Hacking is what you do to get the prototype ready for a demo, or how you apply a temporary patch late at night. Once you get funding, or the next morning, you should take the work a little more seriously. Starting with a mess and hacking at it, only generates a bigger mess, one that will eventually exceed the abilities of all of the available resources, leading to a nasty downfall. Sometimes hacking is called for, but it should be used with the appropriate caution.