A Case Study in Test Management

Tauhida Parveen Scott Tilley George Gonzalez
Dept. of Computer Sciences Dept. of Computer Sciences Software Quality Management
Florida Institute of Technology Florida Institute of Technology Sabre Holdings Inc.
tparveen fit.edu stilley@c fit.edu s. @ george.gonzalez@sabre.com
ABSTRACT
Testing is an essential but often under-utilized area of software
engineering. A variety of software testing techniques have been
developed to effectively identify bugs in source code, yet these
techniques are not always fully employed in practice. There are
numerous reasons for this, including the difficulty in mastering
the complexity of managing all of the test cases for large-scale
projects. Test case management involves organizing testing
artifacts (e.g., requirements traceability data, test cases, and
expected results) in a systematic manner. To be successful, test
case management requires a high degree of discipline to
accommodate the large volume of artifacts under consideration.
This paper presents the results of a case study in centralizing test
artifacts in an industrial setting to aid better test management.
Several of the challenges in adopting this approach are discussed.
In response to these challenges, recommendations on how to
better leverage test case management are offered.
Keywords
Software testing, test management
1. INTRODUCTION
Software testing is a process to identify the correctness,
completeness, and quality of developed computer software. It is
an integral part of software engineering. It encompasses the
concepts of demonstrating the validity of software at each stage of
the development life cycle and the authenticity of the final system
with respect to the customer’s requirements [1]. However, in
application development projects, testing is often not given
enough resources, time, and priority until initial development is
completed. With competitive pressure and the increasing cost of
downtime, organizations have started to introduce testing at
earlier stages of software development while others are striving to
find effective testing strategies. Few organizations have
established the basis to measure the effectiveness of testing.
Without testing standards and a proper test management strategy,
the effectiveness of testing can not be measured or improved.
This paper focuses on the test management part of the testing
process.
Test management is a method of organizing test assets and
artifacts such as test requirements, test cases, and test results to
enable accessibility and reuse. This paper compares and contrasts
the theory of test case management against what commonly
happens in industry with respect to manual test cases. It describes
a case study of the authors experience in applying academic best
practices for software test case management in industry and the
outcome of such best practices. The paper also outlines an
approach to streamlining test case management and how to
enhance productivity through test case management.
The paper is organized as follows: Section 2 gives an overview of
testing and the common techniques that are followed in the testing
process. Section 3 describes a case study of the experiences of the
authors in implementing a test management strategy. Section 4
illustrates the impacts of establishing test management, the
challenges faced and some recommendations to aid in efficient
test management process. Section 5 summarizes the paper with
some direction for future research in this area.
2. SOFTWARE TESTING
Research shows that software quality consumes more than fifty
percent of the cost of the software and this number is even higher
for safety critical software [2]. One way to support quality
assurance is efficient software testing. Testing refers to the
execution of software with an input (known as the test case) and
observing the result of the input. The output data helps in
understanding the behavior of the system and provides data to
ensure the conformance to specification of the software.
The goal of testing is to efficiently identify any high impact
deviations from expected results, so that these deviations can be
corrected or prevented before the software is released. The value
of testing also encompasses goals such as the reduction of the
number of defects and increase development productivity through
continual feedback.
2.1 Testing techniques
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage, and that
copies bear this notice and the full citation on the first page. To copy
otherwise, to republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
ACMSE 2007, March 23-24, 2007, Winston-Salem, North Carolina, USA
©Copyright 2007 ACM 978-1-59593-629-5/07/0003...$5.00
Software testing is a collection of techniques used in the process
of verifying that a piece of software is free from technical errors.
Three well-known testing techniques that are commonly used in
the industry are black box, white box, and gray box testing [6].
Black box testing treats the system as if it were a “black box.”
This testing technique requires no knowledge of the internal code
of the system during the testing process and with little regard in

82

the internal logic of the system. Although black box testing is
designed to identify errors, the technique is also used to
demonstrate that the software meets functional requirements.
White box testing of software involves a closer examination of the
internal logic of the system. Logical paths through the software
are tested by providing test cases that use particular sets of
conditions and/or loops. The status of the system can be identified
at diverse points to establish if the expected status matches the
actual status.
Gray box testing refers to a technique that examines and can
manipulate the activity of back-end components to view the state
of the application during test case execution. An example of gray
box testing would be using database queries to seed data and
check results.
The testing process is not limited to the above-mentioned
techniques only. It includes testing the software at different levels
such as unit, system, integration, acceptance, regression, load,
performance, usability, and reliability testing. It also involves
writing well-described test cases, test plans, test results, and bug
reports.
2.2 Test automation
Testing can be manual or automated. Manual testing is performed
by testers who use manual interaction to carry out all the step by
step actions on the application and determine whether a particular
step was accomplished successfully or not.
Automated testing is assisted by tools that allow the software to
be tested repeatedly and quickly without the need for the presence
of a tester for every input, analysis, or evaluation. It is not always
possible to fully automate testing. Therefore, manual testing is
often an important part of almost any testing effort.
Test automation is also not an easy task and requires a greater up-
front investment than manual testing. It is expensive due to the
need of its specialized tools to support automation as well as the
need for specialized programming skills to develop test suites.
2.3 Testing in industry
The general theory of software testing is not directly usable in
industry. Commercial software testing is always under time
pressure that makes it hard to follow theoretical best practices of
software testing. Testing in practice is very dynamic, requires
reconfiguration of tests from existing test cases, or the
modification of basic assumptions from the existing test cases. In
theory, testing should be introduced in every step of the software
development cycle but in practice it is often introduced at the end
of the software development process. Limited time, budget, and
resources are put into the testing phase. There is often only one
tester for a whole software development team executing mostly
manual test cases and documenting the results. Therefore, a lot of
critical bugs that should be caught before deployment of the
product are never caught.
Since the overall testing process in software development projects
takes less priority, a sub-section of the testing process, test
management, is often not even considered to be an area that
should be paid attention to. Testing software applications requires
hundreds to thousands of unique test cases, and the ability to
efficiently manage the results that are produced from running
these test cases. The test cases are collectively known as the test
suite and can be manual or automated. With the test cases and the
test results, other artifacts such as test plans, bug reports, and
defect reports are also produced during the testing process. Often
these artifacts are not maintained due to lack of a proper test
management strategy and eventually become outdated and can not
be reused.
Test management is a method of organizing test assets and
artifacts such as test requirements, test plans, test cases, test
scripts, and test results to enable easy accessibility and
reusability. It is one of the many ways the testing process can be
made more efficient. There is a need to introduce test
management in the testing process, but often the amount of time
and effort that has to be put in implementing such a management
strategy may keep one from taking such a step.
3. A CASE STUDY
This case study describes the challenges that were faced when
introducing test management for the first time in an industrial
setting. The case study is based on experiences of the authors in
implementing a centralized test management strategy. It
emphasizes the need for a test management strategy with the right
resources and effective use of centralized control.
3.1 The project
The objective of the project was to gather all testing artifacts that
were available between different testing teams into one central
repository to streamline a more disciplined approach to test
management.
The company followed an agile software development
methodology [12]. Software development at this company
encompassed developing new products, upgrading of existing
products, and software customization. Development was
distributed among several offices around the world. Since
development was agile, testers were involved in the project from
the beginning and in every stage of the process. Every time
changes were made to the product by the developers, the resulting
build was sent to the testing team. The testing team executed
regression tests to make sure that all the tests cases that the
product passed before the changes were made, still worked after
the new changes have been added.
The regression tests were typically manual test cases in Microsoft
Excel files. The Excel files containing the test cases were saved in
the tester’s workstation and when needed by others, were
delivered via email. Every time a tester was assigned to a product,
the tester would make changes in the Excel files according to
what he thought suited him. The progress in the testing process
was handled by email from the testers stating a percentage count
of the test coverage. It was getting very hard for management as
well as offshore testers to have access to the test cases.
Predictably, a large percentage of test cases were lost over time
due to hardware failures, employee turnover, and simple changes
to test case formats. Other artifacts such as test plans and
automated test scripts were also maintained in this manner.
After managing the testing artifacts in the above-mentioned ad-
hoc manner, it was decided that a global test management strategy
needed to be in place. Therefore, a project to centralize the testing
artifacts started. The project had two major goals: (1) to manage
test cases globally so that the testers at various locations (in this
case Poland, India and the U.S.) could use the same test cases to
test the product; and (2) to minimize the loss of testing artifacts

83