6.1 Overview
When people discuss testing tools
they invariably think of automated testing tools and in particular
capture/replay tools. However, the market changes all the time and this module
is intended to give you a flavor of the many different types of testing tool
available. There is also a discussion about how to select and implement a
testing tool for your organization. Remember the golden rule, if you automate a
mess, you'll get automated chaos; choose tools wisely!
6.2 Objectives
After completing this module you
will be able to:
» Name up to thirteen different
types of testing tools.
» Explain which tools is in common
use today and why.
» Understand when test automation
tools are appropriate and when they are not.
» Describe in outline a tool
selection process.
6.3 Types of CAST tools
There are numerous types of
computer-aided software testing (CAST) tool and these are briefly described
below.
Requirements testing tools provide automated support for the
verification and validation of requirements models, such as consistency
checking and animation.
Static analysis tools provide information about the quality
of the software by examining the code, rather than buy running test cases
through the code. Static analysis tools usually give objective measurements of
various characteristics of the software, such as the cyclomatic complexity
measures and other quality metrics.
Test design tools generate test cases from a
specification that must normally be held in a CASE tool repository or from
formally specified requirements held in the tools itself. Some tools generate
test cases from an analysis of the code.
Test data preparation tools enable data to be selected from
existing databases or created, generated, manipulated and edited fro use in
tests. The most sophisticated tools can deal with a range of file and database
formats.
Character-based test running
tools provide
test capture and replay facilities for dumb-terminal based applications. The
tools simulate user-entered terminal keystrokes and capture screen responses
for later comparison. Test procedures are normally captured in a programmable
script language; data, test cases and expected results may be held in separate
test repositories. These tools are most often used to automate regression
testing.
GUI test running tools provide test capture and replay
facilities for WIMP interface based applications. The tools simulate mouse
movement, button clicks and keyboard inputs and can recognize GUI objects such
as windows, fields, buttons and other controls. Object states and bitmap images
can be captured for later comparison. Test procedures are normally captured in
a programmable script language; data, test cases and expected results may be
held in separate test repositories. These tools are most often used to automate
regression testing.
Test harnesses and drivers are used to execute software under
test, which may not have a user interface, or to run groups of existing
automated test scripts, which can be controlled by the tester. Some
commercially available tools exist, but custom-written programs also fall into
this category. Simulators are used to support tests where code or other systems
are either unavailable or impracticable to use (e.g. testing software to cope
with nuclear meltdowns).
Performance test tools have two main facilities: load generation
and test transaction measurement. Load generation is done either by driving
application using its user interface or by test drivers, which simulate load
generated by application on architecture. Records of numbers of transactions
executed are logged. Driving application using its user interface, response
time measurements are taken for selected transactions and these are logged.
Performance testing tools normally provide reports based on test logs, and
graphs of load against response times.
Dynamic analysis tools provide run-time information on state
of executing software. These tools are most commonly used to monitor
allocation, use and de-allocation of memory, flag memory leaks, unassigned
pointers, pointer arithmetic and other errors difficult to find 'statically'.
Debugging tools are mainly used by programmers to
reproduce bugs and investigate the state of programs. Debuggers enable
programmers to execute programs line by line, to halt program at any program
statement and to set and examine program variables.
Comparison tools are used. to detect differences between
actual results and expected results. Standalone comparison tools normally deal with
a range of file or database formats. Test running tools usually have built-in
comparators that deal with character screens, Gill objects or bitmap images.
These tools often have filtering or masking capabilities, whereby they can
'ignore' rows or columns of data or areas on screens.
Test management tools may have several capabilities. Test
ware management is concerned with creation, management and control of test
documentation, e.g. test plans, specifications, and results. Some tools support
project management aspects of testing, for example, scheduling of tests,
logging of results and management of incidents raised during testing. Incident
management tools may also have workflow-oriented facilities to track and
control allocation, correction and retesting of incidents. Most test management
tools provide extensive reporting and analysis facilities.
Coverage measurement (or analysis) tools provide objective
measures of structural test coverage when test are executed. Programs to be
tested are instrumented before compilation. Instrumentation code dynamically
captures coverage data in a log file without affecting functionality of program
under test. After execution, log file is analysed and coverage statistics
generated. Most tools provide statistics on most common coverage measures such
as statement or branch coverage.
6.4 Tool selection and
implementation
There are many test activities,
which can be automated, and test execution tools are not necessarily first or
only choice. Identify your test activities where tool support could be of
benefit and prioritize areas of most importance.
Fit with your test process may be
more important than choosing tool with most features in deciding whether you
need a tool, and which one you choose. Benefits of tools usually depend on a
systematic and disciplined test process. If testing is chaotic, tools may not
be useful and may hinder testing. You must have a good process now, or
recognize that your process must improve in parallel with tool implementation.
The ease by which CAST tools can be implemented might be called 'CAST
readiness’.
Tools may have interesting
features, but may not necessarily be available on your platforms, e.g., 'works
on 15 flavors of Unix, but not yours...’ Some tools, e.g. performance testing tools,
require their own hardware, so cost of procuring this hardware should be a
consideration in your cost benefit analysis. If you already have tools, you may
need to consider level and usefulness of integration with other tools, e.g.,
you may want test execution tool to integrate with your existing test
management tool (or vice versa). Some vendors offer integrated toolkits, e.g.
test execution, test management, performance-testing bundles. Integration
between some tools may bring major benefits, in other cases; level of
integration is cosmetic only.
Once automation requirements are
agreed, selection process has four stages:
Creation of a candidate tool
shortlist.
Arrange demos.
Evaluation(s) of selected tool(s).
Review and select tool.
Before making a commitment to
implementing the tool across all projects, a pilot project is usually
undertaken to ensure the benefits of using the tool can actually be achieved.
The objectives of the pilot are to gain some experience in use of the tools,
identify changes in the test process required and assess the actual costs and
benefits of implementation. Roll out of the tool should be based on a
successful result from the evaluation of the pilot. Roll -out normally requires
strong commitment from tool users and new projects, as there is an initial
overhead in using any tool in new projects.
Exercise
Incident management system
List some of your requirements for
an incident management system.
6.5 Summary
In module six you have learnt that
in particular you can now:
Understand there are many different
types of testing tool to support the test process.
Understand what CAST stands for.
Understand that you must have a
mature test process before embarking on test automation.
Know why you must define
requirements for a tool prior to
purchasing one.
For More Testing Techniques:
Flipkart:
http://www.flipkart.com/advanced-test-strategy-istqb-foundation-questions-answers-included/p/itmdp9yzkgedxghz?pid=9781482812220
Amazon:
http://www.amazon.com/Advanced-Test-Strategy-Foundation--Questions-ebook/dp/B00FKS462K/
No comments:
Post a Comment