Whenever an angry customer complains about a product defect, someone is sure to ask “Why didn’t QA find it?” That’s a logical question based on the assumption that every product has a QA group whose job it is to create and run enough tests to find all of the defects in the product.
But so long as people believe that assumption and operate in that manner, we’re going to have quality problems. I don’t believe that “the QA group” is responsible for the quality of our products.
So if quality is not QA’s responsibility, whose is it? What should QA’s role be? And do we even need to have a QA group?
Quality is rightfully a hot topic at the moment, so I’ll address these and other topics related to quality here and in follow-on articles.
Most traditional software development organizations have a QA group whose job it is to write and execute tests, the purpose of which is to detect defects in the code written by the Software Engineering department. In a typical development process, the “product” moves along on a software assembly line from requirements to design to code to test and finally to packaging and shipment. The last significant step in that flow is “test” and that is always the responsibility of QA, who all too often operate in isolation, both by design and by necessity. (This is true even in many groups who are doing iterative development.)
Perhaps the most obvious reason that QA is not responsible for quality is that you can’t test quality into the code. Testing alone can never discover all of the defects in anything but the most trivial piece of code (not more than 60% according to Capers Jones), so QA is on mission impossible if they are expected to prevent the escape of any defects.
But QA typically operates under another very severe handicap, which is their isolation from the rest of the process.
In most development organizations, QA typically is organized as a distinct group, usually reporting to a QA manager. This is done for several reasons. Grouping people of a similar discipline, function or skill together is often done for management convenience. Presumably a “QA manager” is an expert in the job of QA and thus can best manage the work of QA people (and similar for engineers, and for analysts), including providing work direction and doing performance reviews. And some people believe that QA people must operate at some distance from engineers and analysts so they can be completely objective in their testing, uncontaminated by the biases they would surely develop if they were allowed to fraternize with those other functions.
Another reason that the QA group is isolated from the other development functions is that they are often operating at a different point on the development timeline. Since test is the last major step of a sequential development process, that’s where you normally will find QA working, while the analysts and engineers have moved on ahead.
So even if QA really were responsible for quality, forcing them to work in isolation at the end of the development cycle greatly reduces their effectiveness as a defense against quality issues because:
- Being the last major step in the process, they are under the greatest time pressure to achieve the GA date, exacerbated by the likelihood that upstream groups have had some inevitable schedule slips.
- The analysts and engineers feel little sense of responsibility for quality because their attention has turned elsewhere, and it’s QA’s job to come along aftwards and find the bugs in the work they completed awhile ago.
- It prevents QA people from participating in detailed discussion of the requirements and design that take place earlier in the cycle, thus reducing their understanding of the function whose quality they are trying to ensure.
- Allowing analysts and engineers to proceed ahead of QA inevitably leads to a mounting backlog of defects during the development cycle, whether discovered or (even worse) not discovered. This backlog represents an unknown quality liability.
- In general, it creates a handoff. See my posting on Handoffs to see why this is bad.
So what’s the solution, and who should be responsible for software quality? Answer: Everyone in the process must be fully responsible for quality – including the analysts and engineers – together with QA people. And furthermore, they need to be jointly and equally responsible, not merely “involved” by having performed their assigned job as part of the overall process. (“I wrote code according the the spec and it seems to work, so my job is done.”)
In an earlier post I discussed the use of cross-functional teams (CFTs) as an integral and essential part of the iterative development process. The use of real CFTs (where everyone on the team sits together, works together, and is jointly responsible together for satisfying the customer) is the strongest method I know of for ensuring that the product that emerges from the development process is of top quality, because it applies the greatest number of brains to achieving that goal, and makes everyone feel the responsibility that results from a feeling of ownership.
Of course we need to test our products, but testing is only one form of defense against quality issues. Only when all members of a CFT put their minds together will we maximize success. Working in that way, analysts might point out some functionality that needs special testing focus, and engineers can point out areas of the code that need special attention. But beyond that, it might be that the engineers themselves should write some of their own diagnostics or tests (the idea of test-driven development comes to mind), the analysts might be able to locate some customer data that would make particularly good input, the whole CFT might inspect the detailed requirements and design docs together, etc.
A few other comments:
- I didn’t forget about test automation, which is another extremely important tool for ensuring quality. I rank it second in importance, after the use of CFTs, because all the automation in the world still won’t do the job unless it is testing the right things. The collective brainpower of the CFT should be applied to devising the automation plan to ensure the best coverage.
- Quality is not only the absence of defects like crashes or incorrect results. Quality is when the product works the way the customer wants it to and expects it to. No amount of testing can ensure that, especially if done by a group that is separated from customers or even those who represent customers (such as analysts). So the QA function must be an integrated part of the CFT operation.
To return to my thesis: the QA group is not responsible for quality – everyone working on the development of a new product or feature is, jointly and equally. This is difficult to impossible if QA is organized as a separate function, if QA people are physically or temporally separated from the engineers and analysts, or if the engineers and analysts don’t take joint responsibility. The use of co-located CFTs who work together as teams from start to finish is the answer.
So what should QA’s role be? And do we even need to have a QA group? I’ll address these in a follow-up posting.
