Quantifying Quality
ROI
Any investment a company makes in additional
resources for testing and quality should yield clearly
measurable returns. As with anything, you get what you pay
for. The key to accountability in testing comes in knowing
what to expect as a measure of return for your investment. One
way to assess your company’s ROI for its investment in
testing is to view accountability as a set of levels. Each
level can be characterized by the business costs the company
is incurring. There are corresponding deliverables from the
test team that provide a measurement of assurance that the
investment is not wasted.
Level 1: Testing to catch and remove defects
The first level of investment and associated
return/accountability applies where an initial investment has
been made in building a Test Team that will off load
developers of test activities associated with debugging code.
This is one of the most common starting points for investing
in quality. This type of testing is also common to product
releases that are schedule and budget driven. In these
situations most senior managers realize the Test Team cannot
be held responsible for defects that reach the field. Rather
the team is asked to test efficiently and find the important
problems. Product organizations that operate at this level
usually limit quality assurance to testing. And, testing is in
place to help debug the product before it reaches the
customer. None-the-less, the Test Team can be held responsible
for four critical deliverables. These deliverables help your
project team understand just how much testing is happening and
help ensure efficient use of the test resources.
· Written Test
Cases. - Written Test Cases document, in detail, each test
that was performed on the product. Even if testing must happen
as quickly as possible and without prior planning, at the very
least, the testers should write the test cases as the testing
takes place. This allows a second test run to cover ground
faster and provides a good starting point for future testing.
These written test cases also demonstrate the amount and type
of testing the team has actually performed.
· Written Test
Plans. - Test Plans document the actual and desirable resource
needs for testing the product. They also describe what will be
tested and how the team goes about creating and executing the
tests. These plans can be very simple. The critical pieces of
a plan include people assignments, duration of testing for
each functional area being tested, and descriptions of special
test environments. This information provides a way of
measuring actual resource needs against estimated needs. At
the very least these plans form a start for estimating the
costs of a second pass of testing or a second release of the
product.
· Test Status
Reports. Status Reports document, at regular intervals, the
amount of testing performed. In their simplest terms, these
status reports should describe for each week and for each
functional area being tested how many Test Cases exist and how
many have been completed.
· Written Defect
Reports. Defect Reports should be generated for every bug,
enhancement request, and issue found by the test team. This
information provides insight into the quantity and nature of
defects remaining in the product at the time it is released.
This documentation ensures that defects are not lost on scraps
of paper and buried in emails between developers and testers.
These
deliverables provide transparency into the test process and
form the foundation for all other measures of return on the
investment being made in testing and quality. Without these
four types of documentation it becomes difficult to
objectively determine if the investment in testing is really
worthwhile.
Level 2: Ensuring complete test coverage
The second level of investment and associated
return/accountability reflects a commitment in the company to
thorough testing of product releases. This level of investment
occurs when a company is willing to invest in strong test
methodologies and is willing to hold up release of an offering
until after all appropriate testing has completed. This level
still views testing as primarily a debugging function. The
incremental cost here is found in the areas of test planning,
status reporting, reviewing plans, and additional staffing.
· The costs for
planning and status are usually associated with asking test
leaders and senior test personnel to spend additional time
planning tests, developing tests and reporting status in a
manner that can be acted upon by the project team. This
implies that a Test Team is truly a team with some level of
test project lead or management hierarchy. The planning and
status reporting activities incur a cost as they divert the
leader’s attention from actually performing tests.
· The costs
associated with reviewing derive from asking developers,
product managers and other personnel to participate in reviews
that contribute to test planning, status evaluation, test
scripting, and defect management. The involvement of the
cross-functional team at this test level implies that test
related responsibilities are now expanding beyond just the
Test Team.
· Staffing costs
derive from adding full time and temporary staff that are
necessary to accomplish special testing, or planned testing
within the desired schedules. Now that there are plans, the
project team can legitimately seek additional resources to
meet the test coverage and schedule milestones.
The deliverables outlined for Level 1 testing are
sufficient to determine that test related activities are
taking place with some level of forethought as to their
appropriateness. However, the Level 1 deliverables do not
provide assurance that testing is fully benefiting from
knowledge and skill found outside of the test organization.
Nor do the Level 1 deliverables provide a sound basis for
evaluating risks associated with schedules and quality. To
address these areas, the Test Teams should be held responsible
for all the first level deliverables and the following
additional items.
· Standard formats
for written Test Cases with required reviews by developers and
product managers. The Test Cases should not only reflect what
was tested. They should also reflect what should be tested.
The standard formats ensure that reviewers become familiar
with how to evaluate Test Cases. These formats also allow the
Test Team to enhance and extend information as the need for
improved Test Case documentation grows.
· Documented Test
Cycle plans for each base level or release. These plans
include the definition and execution of quick “verification
tests” and Test Cycle Completion Reports. The depth and
breadth of testing should change in response to different
stages of product development. Documented Test Cycles describe
the intended amount and type of testing for each stage. This
allows the project team to better understand and influence the
amount of planned testing.
· Formal Defect
Management Procedures that include documented processes for
assigning severities, priorities, responsibilities and status.
These procedures should include involvement of the
cross-functional team in reviewing defects. Many times the
decision to fix or defer defects is a business decision that
impacts the final features of a product and the call load on
support organizations. By managing a formal defect review
process, the Test Team ensures input from all affected
organizations.
· Graphical and
tabular Defect Status Reports that demonstrate trends in
arrival and closure rates. Defect arrival rates and closure
rates are critically important in assessing the likelihood of
releasing a product to a given schedule. These rates also
provide insight into the stability of the product development,
release and testing process.
Level 3: Introductory Quality Assurance
The third level of investment and associated
return/accountability applies when a company invests in more
efficient and effective testing and quality procedures. At
this stage, the company begins to look at quality assurance in
a more comprehensive fashion. However, testing is still
primarily focused on debugging the deliverable. The types of
costs at this level include investing in tools and procedures
that improve test coverage and test efficiency. Additional
costs involve increased project management of cross
organization processes and investing in automated test
development and management.
The investment in this area should be targeted at
facilitating a number of test related activities that improve
the speed with which tests are executed and improve the test
coverage. An important goal of the initiatives at this level
is to reduce the number of defects that are first found by
customers. This is an important measure of the effectiveness
of testing. The Test Team should actively pursue a number of
test improvement initiatives. Each of these has associated
costs and potential benefits.
· Implementation
of comprehensive Regression Testing. This includes advocacy
for and the development of an appropriate test automation
program. While an expensive endeavor, certainty in regression
testing is a key quality initiative for ensuring that defects
do not re-appear in future releases, and that today’s
changes do not break past features. Measuring the
implementation of regression testing is accomplished using
Test Plans and Test Completion Status reports that are
specific to this type of testing.
· Process and
technology tie-in for customer service issues with the Defect
Management Procedures. A tight coupling of customer reported
problems with defect management provides assurance that
customer complaints are receiving attention inside of the
development program. This also facilitates tracking of
customer reported defects within the design and implementation
process. This in turn provides Customer Service with tools for
assuring customers they are being heard and their problems are
being appropriately addressed.
· Use of
instrumented test environments to accurately determine the
level of test coverage. Instrumented testing provides insight
into how much of a deliverable is actually being tested by the
documented Test Cases. This information is important for
improving test coverage. It also provides insight into how
much redundancy exists in the test process. By understanding
what is covered by tests and where redundancy exists, the
project team can make informed tradeoffs regarding what tests
are absolutely necessary for any given change in the
deliverable. This in turn can lead to shorter test cycles
especially near the end of a project when last minute changes
are so important and also potentially risky.
· Creation and
management of dedicated test systems and test labs. Test labs
provide the test team with environments they can better tailor
for emulating customer environments. When evaluating the
return on the investment in test labs, there are three key
questions the Test Team should answer. 1) How will the test
team use the lab to speed up the execution of tests? 2) How
will the test team setup and use the lab to execute important
tests they could not execute without a lab? 3) If automated
tests are being used, how will the lab be used to reduce test
development time and increase test execution speed?
· Documented plans
and procedures for emulating customer environments and data.
The actual cost of establishing emulated customer environments
is often overlooked. Furthermore, a hidden cost involves
selecting the appropriate customer environments in the first
place. By requiring plans and then measuring progress against
these plans, you are more likely to end up with efficient and
effective emulations of customer environments. The planning
process also facilitates involvement of cross functional
groups like sales, product management, customer support, and
business line management. Each of these groups brings insights
into what is important and what can be ignored when developing
and maintaining emulated customer environments.
Beyond Testing: Quality Program Definition
and Management
As a company looks to move beyond testing and toward defect
prevention and true quality assurance, a number of investments
and associated returns can be realized. This is the area of
Quality Program Definition and Management. We will save this
discussion for a future technology note.
A company can use the levels described above to help it
develop and manage an investment strategy for testing and
quality assurance. These levels not only provide a framework
for discussing investments in testing, they also provide a
basis for determining how to measure the return on those
investments. In short, the company can measure, with some
clarity, how the test team is performing.
Many companies implement a Quality Assurance organization
that is really just a Test Team. This team serves the very
important purpose of helping the developers debug their releases.
When a company invests in test initiatives there should be
tangible deliverables and measurable results. The measures
described in this note provide a basis for evaluating further
investments in quality assurance and improved product development
methodologies.
|
