Certified Tester

Advanced Level Overview

Version 2012

International Software Testing Qualifications Board

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®).

Advanced Level Working Group: Mike Smith (Chair), Bernard Homès (Vice Chair), Syllabus Coordinators Graham Bath, Rex Black, Judy McKay 2012

Digital book(s) (epub and mobi) produced by Booknook.biz under the direction of Rex Black and Lois Kostroski of the ASTQB.

Revision History

Version

Date

Remarks

0.1

October 4th 2011

Initial version:

0.2

October 5th 2011

TA edits from Judy McKay

0.3

October 19th 2011

WG review comments

0.4

14 DEC 2011

Incorporation of NB review comments

0.5

16 DEC 2011

Incorporation of review comments in San Francisco

Alpha

17 FEB 2012

Updates for Alpha release

Beta

6 APR 2012

Incorporation of comments from NBs received on time from Alpha release.

Beta

8 June 2012

Beta release for NB review after copy edit

Release Candidate

28 JULY 2012

Addition of explanation regarding standards and ISTQB web site additions

GA

19 OCT 2012

General availability release

Table of Contents

Revision History

Acknowlegements

E-book Search Instructions

1.    Introduction to the Advanced Level

1.1    Career Paths for Testers

1.2    Intended Audience

1.3    Learning Objectives

1.4    Entry Requirements

1.5    Structure and Course Duration

1.6    Handling of Standards

1.7    Keeping It Current

2.    Overview of Advanced Level Syllabi

2.1    Advanced Level: Test Manager (TM)

2.1.1     Business Outcomes

2.1.2     Content

2.2    Advanced Level: Test Analyst (TA)

2.2.1     Business Outcomes

2.2.2     Content

2.3    Advanced Level: Technical Test Analyst (TTA)

2.3.1     Business Outcomes

2.3.2     Content

3.    Appendix: Main Changes in the 2012 Syllabi

4.    Abbreviations

5.    References

5.1    Trademarks

5.2    Documents and Web Sites

Acknowledgements

This document was produced by a core team from the International Software Testing Qualifications Board Advanced Level Working Group - Graham Bath, Mike Smith, Rex Black, Judy McKay; 2010-2012.

The core team thanks the review team and all National Boards for their suggestions and input.

At the time the Advanced Level Syllabus was completed the Advanced Level Working Group had the following membership (alphabetical order):

Graham Bath, Rex Black, Maria Clara Choucair, Debra Friedenberg, Bernard Homès (Vice Chair), Paul Jorgensen, Judy McKay, Jamie Mitchell, Thomas Mueller, Klaus Olsen, Kenji Onishi, Meile Posthuma, Eric Riou du Cosquer, Jan Sabak, Hans Schaefer, Mike Smith (Chair), Geoff Thompson, Erik van Veenendaal, Tsuyoshi Yumoto.

The following persons participated in the reviewing, commenting and balloting of this document (alphabetical order):

Rex Black, Thomas Donner, Bernard Homès, Judy McKay, Rasluca Popescu, Jan Sabak, Mike Smith

This document was formally approved for release by the General Assembly of ISTQB® on October 19th, 2012.

E-book Search Instructions

Search Instructions

For the purposes of facilitating reference and review of the important terms used throughout, this page will instruct you in how to use the various search functions on the leading e-reading devices so that you may find Glossary definitions with ease.

How To Search in Kindle e-ink Devices

On a Kindle e-ink device, to look up a word in the dictionary, use the 5-way key to navigate to the specific word you wish to search.

1. Simply highlight the word, and the dictionary definition will appear at the bottom of the Kindle screen.

2. For a more wide-ranging search, start typing the word you wish to look up while the book is open on the screen.

3. When you’ve finished typing the word in the rounded box that will pop up, use the 5-way controller to move to the right. When the “find” button turns black, simply click it and it will search within the current book.

4. For more options, move to the right one more time. You’ll then see 5 options: “my items,” “store,” “google,” “wikipedia” and “dictionary.” Press the desired option to execute the search.

Please note that Kindle is fairly picky, so try to type accurately and competely, e.g., “fired” instead of “fire,” if “fired” is the word you’re seeking. A list of the search results will be returned, and you can navigate to the desired result. Simply then press the “back” toggle, once or twice, (beneath the 5-way button) to return to where you were reading.

How To Search in Kindle Fire Tablet Devices

1. On the Kindle Fire tablet, with the book open, simply highlight the word you wish to search.

2. A dialogue box will pop up, with a dictionary definition, a Note option, a Highlight option and a Search option.

3. Press “Search” and you’ll be asked “Search in Book,” “Search in Wikipedia” or “Search the Web.”

4. Pick the search you want’in this case, for the glossary definitions, you’d choose “Search in Book,” and a chapter-by-chapter list of search hits will be returned.

5. Select the item you want, (in many cases, simply the search result will give you the answer you seek), and you’ll be taken to the result, with your searched term highlighted in yellow.

When done reading, tap the page lightly, and when the bottom menu bar pops up, tap the BACK arrow to return to where you were reading.

How To Search on the Nook and NookColor Tablet Devices

On the Nook and NookColor tablets, with the book open, simply highlight the word you wish to search (using touch).

1. A dialogue box will pop up, with a Highlight option, a Notes option, a Share option (if enabled), a “Look Up” option and a “find” option.

2. Press “Lookup” for a dictionary search (you can also search Google and Wikipedia from this interface.)

3. For an in-book search, simply press “Find.”

4. A list of every occurrence of the term, by chapter, will pop-up in the lower-left-hand corner.

5. Select the choice to which you want to navigate with a thumb-press.

After reviewing the search result, press the double « button (occasionally you may need to press it twice) to return to the location at which you initiated the search.

How To Search on the iPad iBooks application

In the iBooks application, with the book open, simply highlight the word you wish to search (using touch).

1. A dialogue box will pop up, with a Define, Highlight, Note and Search option, with the word you identified highighted in blue.

2. Press “Search” for an in-book search.

3. A list of every occurence of the term, by page number, will pop-up in the upper-right-hand corner.

4. Select the choice to which you want to navigate with a thumb-press.

After reviewing the search result, look down at the lower-left-hand corner of the page, where you’ll see, in very small text, beneath the progress bar, text that says “Back to page X.” Simply press that text to return to the location at which you initiated the search.

How To Search on the Kobo device

Back Button

There is no Back button as such on the Kobo Touch. There is one hardware button that will take you from anywhere to the Kobo Home Page which is a collection of thumbnails of the last five books you’ve been reading. At this page, if you touch the cover image for a particular book, you will be positioned at the last page you read to.

How to Search on the Kobo Touch

To search for a word or phrase in the book:

1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.

2. Touch the open book to pop up a menu with five items, the third of which is “Search in Book.”

3. Touch “Search in Book” and an input box appears at the top of the screen. Underneath is a touchable list of the last six words or phrases that you searched for. On the bottom half of the screen is a keyboard.

4. Type the word or phrase you want to search for with the keyboard and touch “Go” in the lower right corner.

5. You will be presented with a list of phrases from the book text where the word or phrase occurs, organized by the section of the book where they occur.

To find a definition for a word:

1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.

2. Touch the open book to pop up a menu with five items, the fourth of which is “Definition.”

3. Touch “Definition” and an input box appears at the top of the screen. On the bottom half of the screen is a keyboard.

4. Type the word for which you want a definition with the keyboard and touch “Go” in the lower right corner.

5. You will be presented with a list of definitions.

1. Introduction to the Advanced Level

This overview document is intended for anyone with an interest in the ISTQB Advanced Level who wants a high-level introduction to its leading principles and an overview of the individual Advanced Level syllabi.

The Advanced Level certifications will be examined with three major role descriptions in mind, each representing basic responsibilities and expectations within an organization. In any organization, responsibilities and associated tasks may be split between different individuals or covered by a single individual.

The following Advanced syllabi are defined:

- Test Manager

- Test Analyst

- Technical Test Analyst

In this document each Advanced Level syllabus is summarized and associated Business Outcomes are stated. These Business Outcomes provide a specific statement of what can be expected from a person who achieves Advanced Level Certification in a particular subject area, (e.g., an Advanced Test Manager), and will outline the benefits for companies who are considering the development of specific testing skills at this level.

For stakeholders who are already familiar with or use the 2007 version of the Advanced Level Syllabus, a summary of principal changes is provided in the appendix.

1.1 Career Paths for Testers

Building on the Foundation Level, the Advanced Level supports the definition of career paths for professional testers. A person with the Advanced Level Certificate has extended the broad understanding of testing acquired at the Foundation Level to enable the role of Test Manager or Test Analyst to be performed. Test Analyst skills may be further extended into the role of Technical Test Analyst.

The Advanced Level establishes a platform from which further skills and knowledge may be acquired at the Expert Level. After achieving experience as a Test Manager, for example, a person may choose to develop their testing career further by acquiring Expert Level certifications in the subjects of test management and improving the test process.

People possessing an ISTQB Advanced Level Certificate may use the Certified Tester Advanced Level acronyms CTAL-TM, CTAL-TA, and CTAL-TTA, according to the subject for which they are certified. If all three certifications are acquired, the CTAL-Full acronym may be used.

1.2 Intended Audience

The Advanced Level qualification is aimed at people who have achieved an advanced point in their careers in software testing. This includes people in roles such as Testers, Test Analysts, Test Engineers, Test Consultants, Test Managers, User Acceptance Testers and Software Developers.

This Advanced Level qualification is also appropriate for anyone who wants a deeper understanding of software testing, such as Project Managers, Quality Managers, Software Development Managers, Business Analysts, IT Directors and Management Consultants.

1.3 Learning Objectives

In general the Foundation Level syllabus and all parts of the specific Advanced Level syllabus are examinable at a K1 level, i.e. the candidate will recognize, remember and recall terms and concepts stated in the Foundation and the specific Advanced Level syllabus.

The relevant Learning Objectives at K2, K3 and K4 levels are provided at the beginning of each chapter within each particular Advanced Level Syllabus.

1.4 Entry Requirements

To be able to obtain an Advanced Level certification, candidates must hold the Foundation Certificate and satisfy the board which examines them that they have sufficient practical experience to be considered Advanced Level qualified. Refer to the relevant Exam Board and/or National Board to check the specific criteria used to evaluate practical experience.

1.5 Structure and Course Duration

The Advanced Level consists of three separate syllabi. Each syllabus is self-contained; there are no shared or common elements.

The syllabi must be taught in the following minimum number of days:

Syllabus

Days

Test Manager

5

Test Analyst

4

Technical Test Analyst

3

The following figure shows the structure of the Advanced Level and its relationship to the Foundation and Expert Levels. Note that at the time of writing the Advanced Level Test Analyst certification has no direct dependency link to an Expert Level certification. This may change in the future as new syllabi are introduced at the Expert Level.

1.6 Handling of Standards

Standards (IEEE, ISO, etc.) are referenced in these syllabi. The purpose of these references is to provide a framework (as in the references to ISO 9126/ISO 25000 regarding quality characteristics) or to provide a source of additional information if desired by the reader. Please note that only the items from these standards that are referenced specifically in the syllabi are eligible for examination. The standards documents themselves are not intended for examination and are included only for reference.

1.7 Keeping It Current

The software industry changes rapidly. To deal with these changes and to provide the stakeholders with access to relevant and current information, the ISTQB working groups have created links on the www.istqb.org web site which refer to supporting documents, changes to standards and new occurrences in the industry. This information is not examinable under these syllabi.

2. Overview of Advanced Level Syllabi

2.1 Advanced Level: Test Manager (TM)

2.1.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Test Manager certification.

An Advanced Test Manager can…

TM1

Manage a testing project by implementing the mission, goals and testing processes established for the testing organization.

TM2

Organize and lead risk identification and risk analysis sessions and use the results of such sessions for test estimation, planning, monitoring and control.

TM3

Create and implement test plans consistent with organizational policies and test strategies.

TM4

Continuously monitor and control the test activities to achieve project objectives.

TM5

Assess and report relevant and timely test status to project stakeholders.

TM6

Identify skills and resource gaps in their test team and participate in sourcing adequate resources.

TM7

Identify and plan necessary skills development within their test team.

TM8

Propose a business case for test activities which outlines the costs and benefits expected.

TM9

Ensure proper communication within the test team and with other project stakeholders.

TM10

Participate in and lead test process improvement initiatives.

In general, an Advanced Test Manager is expected to have acquired the necessary skills to enable further development at the Expert Level in the areas of test management and improving the test process.

2.1.2 Content

Chapter 1: Testing Process

- The Test Manager’s main activities are covered for each step in the fundamental test process. Emphasis is placed on the tasks of Test Planning, Monitoring and Control. In addition, the Test Manager learns how to implement a project retrospective in order to validate processes and discover areas to improve.

Chapter 2: Test Management

- This is the core chapter of the syllabus and accounts for over two days of training out of the five days course.

- The Test Manager should understand how to define test management tasks according to the context of a project. This involves taking all stakeholder needs into account and adjusting test activities to the software lifecycle model.

- Risk-based testing (RBT) for test prioritization and effort allocation is covered in detail. Emphasis is placed on the Test Manager’s tasks in identifying, analyzing and assessing risks for the product and the project. The Test Manager learns specific techniques and understands how to take stakeholder views into account when assessing risk levels and when defining the set of test activities needed to mitigate the risks. By performing appropriate risk control activities, the Test Manager learns how to determine residual risks and can report them to project stakeholders such that informed release decisions can be made.

- The Test Manager needs to know how to evaluate different types of test documentation and tailor them to meet project and organization needs. This includes an understanding of relevant standards.

- Estimation of testing effort using specific metrics and based on experience is covered. Techniques required for estimating are covered and the Test Manager should learn how to use available data to communicate the value of testing.

- Test Managers should have an understanding for distributed, outsourced and insourced forms of test organizations.

Chapter 3: Reviews

- The activities of the Test Manager focus on defining an appropriate review plan and setting up the review to achieve the best results. The Test Manager should learn how to use metrics to optimize the review results and to show return on investment.

- The Test Manager should understand how to lead a review team through a formal review.

Chapter 4: Defect Management

- The Test Manager should know how to set up a defect lifecycle tailored for the adopted software lifecycle.

- The Test Manager should understand the skills to acquire in analyzing defect reports with the objective of evaluating the capability of the testing and software development processes.

Chapter 5: Improving the Test Process

- The Test Manager should be aware of the generic steps for conducting a standard test process improvement initiative and how to create a test improvement plan based on these steps.

- Information is provided on the test process improvement models TMMi, TPI Next, CTP and STEP.

Chapter 6: Test Tools and Automation

- The Test Manager should be able to plan and implement the selection of different types of tools (including open-source and custom-built tools) such that risks, costs, benefits and opportunities are adequately considered.

- An understanding of using tools for metric collection and evaluation is acquired.

Chapter 7: People Skills - Team Composition

- The Test Manager should understand how to assess the availability of soft skills along with the technical, business domain and testing skills required for a testing team, and can define a growth plan for skill development.

- The Test Manager should understand the relevant team motivation and communication skills required.

2.2 Advanced Level: Test Analyst (TA)

2.2.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Test Analyst certification.

An Advanced Test Analyst can…

TA1

Perform the appropriate testing activities based on the software development lifecycle being used.

TA2

Determine the proper prioritization of the testing activities based on the information provided by the risk analysis.

TA3

Select and apply appropriate testing techniques to ensure that tests provide an adequate level of confidence, based on defined coverage criteria.

TA4

Provide the appropriate level of documentation relevant to the testing activities.

TA5

Determine the appropriate types of functional testing to be performed.

TA6

Assume responsibility for the usability testing for a given project.

TA7

Effectively participate in formal and informal reviews with stakeholders, applying knowledge of typical mistakes made in work products.

TA8

Design and implement a defect classification scheme.

TA9

Apply tools to support an efficient testing process.

2.2.2 Content

Chapter 1: Testing Process

- A Test Analyst should understand the importance of involvement throughout the test process, with particular focus on the role and contribution of the Test Analyst, and how those align with the roles of the Test Manager and Technical Test Analyst.

- The Test Analyst’s tasks of test analysis and design are described. This includes determining the appropriate uses for concrete and logical test cases as well as defining the pre-conditions and criteria used for starting test execution and determining the status of test completion criteria. The impact of different lifecycle models is a central aspect to these tasks.

Chapter 2: Test Management: Responsibilities for the Test Analyst

- A Test Analyst should understand the purpose and expected contribution toward the project metrics.

- A Test Analyst should understand how to prioritize on the basis of risks and can schedule business domain-based tests appropriately. This includes understanding the risk assessment impact on test case selection, test coverage and test data tasks.

Chapter 3: Test Techniques

- Testing techniques belong to the core competencies of the Test Analyst. Specification-based and defect- and experience-based techniques are covered.

- The specification-based techniques introduced at the Foundation level are developed further. These include equivalence partitioning, boundary value analysis, decision tables, state transition testing and use case testing.

- Additional specification-based techniques introduced include classification tree testing, use of orthogonal arrays, pairwise testing, domain analysis and user stories.

- Defect-based techniques, exploratory testing and the use of defect taxonomies are additional techniques covered in the area of defect- and experience-based techniques.

- A Test Analyst should understand how to select the best technique for a given testing situation as well as how to effectively mix techniques to achieve the best testing result.

Chapter 4: Testing Software Quality Characteristics

- The specific software quality characteristics that apply to the Test Analyst are covered in this section. These include the functional areas of accuracy, suitability and interoperability as well as the non-functional areas of usability and accessibility.

- A Test Analyst should understand how to approach these areas and the testing techniques that can be applied.

Chapter 5: Reviews

- The activities of the Test Analyst focus on using checklists to identify defects in use cases and in requirements specifications from a tester’s perspective. In addition, the Test Analyst learns how to present problems found in a review meeting.

- Several sample checklists are supplied to help guide review sessions for various work products.

Chapter 6: Defect Management

- Test Analysts should understand how to define defect classification values to be used in a defect management system and how to apply this classification to defects found.

- A discussion of the importance of capturing, refining and using root cause information for process improvement is included in this section. Test Analyst should know how to perform preliminary root cause analysis to help provide good classification information for each reported defect.

Chapter 7: Test Tools

- This short chapter focuses on the tools and automation issues which are relevant to the Test Analyst. This includes building awareness of business process modeling tools and knowledge of the interactions between the tools typically used by a Test Analyst.

2.3 Advanced Level: Technical Test Analyst (TTA)

2.3.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Technical Test Analyst certification.

An Advanced Technical Test Analyst can…

TTA1

Recognize and classify the typical risks associated with the performance, security, reliability, portability and maintainability of software systems.

TTA2

Create test plans which detail the planning, design and execution of tests for mitigating performance, security, reliability, portability and maintainability risks.

TTA3

Select and apply appropriate structural design techniques to ensure that tests provide an adequate level of confidence, based on code coverage and design coverage.

TTA4

Effectively participate in technical reviews with developers and software architects applying knowledge of typical mistakes made in code and architecture.

TTA5

Recognize risks in code and software architecture and create test plan elements to mitigate those risks through dynamic analysis.

TTA6

Propose improvements to the security, maintainability and testability of code by applying static analysis.

TTA7

Outline the costs and benefits to be expected from introducing particular types of test automation.

TTA8

Select appropriate tools to automate technical testing tasks.

TTA9

Understand the technical issues and concepts in applying test automation.

2.3.2 Content

Chapter 1: The Technical Test Analyst’s Tasks in Risk-Based Testing

- A Technical Test Analyst should understand how to identify, assess and mitigate technical risks.

Chapter 2: Structure-Based Testing

- Structural testing techniques belong to the core competencies of the Technical Test Analyst.

- This section builds on the Foundation techniques of statement and decision coverage.

- The structure-based techniques covered are condition testing, decision/condition testing, modified condition/decision coverage (MC/DC), multiple condition testing, basis path testing and API coverage.

- In general, Technical Test Analysts should understand how to choose appropriate structural test technique(s).

Chapter 3: Analytical Techniques

- Technical Test Analysts should understand how to apply static analysis to detect potential security, maintainability and testability defects in code.

- The planning of dynamic analysis to mitigate risks in code and software architecture is covered.

Chapter 4: Quality Characteristics for Technical Testing

- A Technical Test Analyst should understand how to design high-level test cases for security, performance and reliability quality attributes and to support the Test Manager in creating test strategies to mitigate the identified risks.

- A Technical Test Analyst should understand how to include coverage of maintainability, portability and resource utilization quality attributes in a testing strategy.

Chapter 5: Reviews

- The activities of the Technical Test Analyst focus on using checklists to identify defects in code and architecture.

Chapter 6 Test Tools and Automation

- This major chapter focuses on the tools and automation issues which are relevant to Technical Test Analysts.

- Several tools are covered, including those used for web-based testing, for supporting model-based testing, for fault seeding and fault injection, for unit testing and the build process and for performance testing.

- A Technical Test Analyst should be able to recognize common technical issues that cause high failure rates in automation projects and to appreciate different automation techniques.

- Specific issues resulting from the use of open-source and custom-built tools are covered.

3. Appendix: Main Changes in the 2012 Syllabi

Introduction

In this 2012 version of the Advanced Level syllabi, feedback was considered from stakeholders in the market such as training providers and attendees.

Improved Scoping

A principal objective of the new Advanced Level Syllabi is to ensure that there is no overlap between any current ISTQB syllabi.

The Advanced Syllabi add to the Foundation Level and fit neatly with the topics managed at the Expert Level (Test Management, Improving the Test Process, Test Automation, etc.).

In addition, the consistency and clear scoping between the three Advanced Level syllabi has been substantially improved.

Clearer Organization of the Syllabus

The 2012 Advanced Level Syllabi have been created as three stand-alone syllabi with a single overview document (this document).

This structure makes it clearer for everyone (students, training providers and exam creation teams) to understand what is expected and what is to be covered. This is also consistent with the structure of the Expert Level. The individual Advanced syllabi may evolve separately if needed.

Reduced Duration of TA and TTA Syllabi

Stakeholder feedback has been carefully considered in setting the minimum course duration. As a result of careful scoping and a policy of “no overlaps”, the duration of the Test Analyst syllabus has been reduced from five to four days and the duration of the Technical Test Analyst has been reduced from five to three days.

Business Outcomes Now Form the Basis for the Advanced syllabi

Each Business Outcome provides a statement of what can be expected from a person who achieves the Advanced Level in the particular subject area, (e.g. an Advanced Test Manager). The BOs are listed in this document.

Business Outcomes are specifically directed to the business needs of industry and will particularly benefit businesses who are considering investing in developing the skills of their staff at the Advanced level.

The Learning Objectives of each Advanced Syllabus implement (and are traceable to) the Business Outcomes.

Improved Learning Objectives

Learning Objectives have been improved by removing potential misinterpretations and by splitting certain “compound” learning objectives into individual parts.

Learning Objectives are now uniquely numbered and placed at the start of each syllabus chapter, in line with Foundation and Expert Level syllabi.

The following example demonstrates these improvements:

2007 syllabus:

(K3) Use the algorithms “Control flow analysis”, “Data flow analysis” to verify if code has not any control or data flow anomaly

2012 syllabus (TTA):

TTA-3.2.1 (K3) Use control flow analysis to detect if code has any control flow anomalies

TTA-3.2.2 (K3) Use data flow analysis to detect if code has any data flow anomalies

Changes to the Test Management Syllabus

The principal subjects covered remain the same. However, the content has been improved.

Redundancies with FL have been removed (e.g. reviews).

The Expert Level syllabi “Test Management“ and “Improving the Test Process“ are now available. Scoping and alignment with these syllabi has taken place.

The defect management chapter is no longer based on IEEE-1044 and focuses more on setting up a defect management lifecycle and using defect data for process improvement.

Changes to the Test Analyst Syllabus

The major concentration of this syllabus remains on the testing techniques and test process.

Domain analysis and user stories are new to the specification-based techniques section.

Content has been significantly revised to align the scope with Foundation, other Advanced Level syllabi and Expert Level Test Automation.

The Test Management and Tools chapters are relatively short and cover only the specific issues relating to the Test Analyst.

The defect management chapter is no longer based on IEEE-1044 and focuses more on defect categorization and performing initial root cause analysis of defects.

Changes to the Technical Test Analyst Syllabus

A significant refocus on the technical aspects of testing has been completed. As a result it is now expected that candidates must be able to read and understand pseudo-code.

The basic aspects of testing and the testing process are now covered in other syllabi.

The test management chapter is relatively short and covers only the specific issues relating to the Technical Test Analyst.

Testing techniques remains a significant part of the syllabus and accounts for approximately one third of the training time. Six techniques are covered, including basis path testing and API coverage. The LCSAJ technique has been removed from the syllabus.

Content has been significantly reduced due to scoping with the other syllabi (in particular Test Analyst).

4. Abbreviations

Abbreviation

Meaning

BO

Business Outcome

ISTQB

International Software Testing Qualifications Board

LO

Learning Objective

TA

Test Analyst

TM

Test Manager

TTA

Technical Test Analyst

5. References

5.1 Trademarks

The following registered trademarks and service marks are used in this document:

ISTQB® is a registered trademark of the International Software Testing Qualifications Board

5.2 Documents and Web Sites

Identifier

Reference

[ISTQB-Web]

Web site of the International Software Testing Qualifications Board. Refer to this website for the latest ISTQB Glossary and syllabi. (www.istqb.org)

Certified Tester

Advanced Level Syllabus Test Analyst

Version 2012

International Software Testing Qualifications Board

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®).

Advanced Level Test Analyst Working Group: Judy McKay (Chair), Mike Smith, Erik Van Veenendaal; 2010-2012.

Revision History

Version

Date

Remarks

ISEB v1.1

04 SEP 01

ISEB Practitioner Syllabus

ISTQB 1.2E

SEP03

ISTQB Advanced Level Syllabus from EOQ-SG

V2007

12 OCT 07

Certified Tester Advanced Level syllabus version 2007

D100626

26 JUN 10

Incorporation of changes as accepted in 2009, separation of chapters for the separate modules

D101227

27 DEC 10

Acceptance of changes to format and corrections that have no impact on the meaning of the sentences.

D2011

23 OCT 11

Change to split syllabus, re-worked LOs and text changes to match LOs. Addition of BOs.

Alpha 2012

09 MAR 12

Incorporation of all comments from NBs received from October release.

Beta 2012

07 APR 12

Incorporation of all comments from NBs received from the Alpha release.

Beta 2012

07 APR 12

Beta Version submitted to GA

Beta 2012

08 JUN 12

Copy edited version released to NBs

Beta 2012

27 JUN 12

EWG and Glossary comments incorporated

RC 2012

15 AUG 12

Release candidate version - final NB edits included

GA 2012

19 OCT 12

Final edits and cleanup for GA release

Table of Contents

Revision History

Acknowledgements

0. Introduction to this Syllabus

0.1 Purpose of this Document

0.2 Overview

0.3 Examinable Learning Objectives

1. Testing Process - 300 mins.

1.1 Introduction

1.2 Testing in the Software Development Lifecycle

1.3 Test Planning, Monitoring and Control

1.3.1 Test Planning

1.3.2 Test Monitoring and Control

1.4 Test Analysis

1.5 Test Design

1.5.1 Concrete and Logical Test Cases

1.5.2 Creation of Test Cases

1.6 Test Implementation

1.7 Test Execution

1.8 Evaluating Exit Criteria and Reporting

1.9 Test Closure Activities

2. Test Management: Responsibilities for the Test Analyst - 90 mins

2.1 Introduction

2.2 Test Progress Monitoring and Control

2.3 Distributed, Outsourced and Insourced Testing

2.4 The Test Analyst’s Tasks in Risk-Based Testing

2.4.1 Overview

2.4.2 Risk Identification

2.4.3 Risk Assessment

2.4.4 Risk Mitigation

3.Testing Test Techniques - 825 mins.

3.1 Introduction

3.2 Specification-Based Techniques

3.2.1 Equivalence Partitioning

3.2.2 Boundary Value Analysis

3.2.3 Decision Tables

3.2.4 Cause-Effect Graphing

3.2.5 State Transition Testing

3.2.6 Combinatorial Testing Techniques

3.2.7 Use Case Testing

3.2.8 User Story Testing

3.2.9 Domain Analysis

3.2.10 Combining Techniques

3.3 Defect-Based Techniques

3.3.1 Using Defect-Based Techniques

3.3.2 Defect Taxonomies

3.4 Experience-Based Techniques

3.4.1 Error Guessing

3.4.2 Checklist-Based Testing

3.4.3 Exploratory Testing

3.4.4 Applying the Best Technique

4. Testing Software Quality Characteristics - 120 mins.

4.1 Introduction

4.2 Quality Characteristics for Business Domain Testing

4.2.1 Accuracy Testing

4.2.2 Suitability Testing

4.2.3 Interoperability Testing

4.2.4 Usability Testing

4.2.5 Accessibility Testing

5. Reviews - 165 mins

5.1 Introduction

5.2 Using Checklists in Reviews

6. Defect Management – 120 mins

6.1 Introduction

6.2 When Can a Defect be Detected?

6.3 Defect Report Fields

6.4 Defect Classification

6.5 Root Cause Analysis

7. Test Tools - 45 mins.

7.1 Introduction

7.2 Test Tools and Automation

7.2.1 Test Design Tools

7.2.2 Test Data Preparation Tools

7.2.3 Automated Test Execution Tools

8. References

8.1 Standards

8.2 ISTQB Documents

8.3 Books

8.4 Other References

9. Index

Acknowledgements

This document was produced by a core team from the International Software Testing Qualifications Board Advanced Level Working Group - Advanced Test Analyst: Judy McKay (Chair), Mike Smith, Erik van Veenendaal.

The core team thanks the review team and the National Boards for their suggestions and input.

At the time the Advanced Level Syllabus was completed the Advanced Level Working Group had the following membership (alphabetical order):

Graham Bath, Rex Black, Maria Clara Choucair, Debra Friedenberg, Bernard Homès (Vice Chair), Paul Jorgensen, Judy McKay, Jamie Mitchell, Thomas Mueller, Klaus Olsen, Kenji Onishi, Meile Posthuma, Eric Riou du Cosquer, Jan Sabak, Hans Schaefer, Mike Smith (Chair), Geoff Thompson, Erik van Veenendaal, Tsuyoshi Yumoto.

The following persons participated in the reviewing, commenting and balloting of this syllabus:

Graham Bath, Arne Becher, Rex Black, Piet de Roo, Frans Dijkman, Mats Grindal, Kobi Halperin, Bernard Homès, Maria Jönsson, Junfei Ma, Eli Margolin, Rik Marselis, Don Mills, Gary Mogyorodi, Stefan Mohacsi, Reto Mueller, Thomas Mueller, Ingvar Nordstrom, Tal Pe’er, Raluca Madalina Popescu, Stuart Reid, Jan Sabak, Hans Schaefer, Marco Sogliani, Yaron Tsubery, Hans Weiberg, Paul Weymouth, Chris van Bael, Jurian van der Laar, Stephanie van Dijk, Erik van Veenendaal, Wenqiang Zheng, Debi Zylbermann.

This document was formally released by the General Assembly of the ISTQB® on October 19th, 2012.

0. Introduction to this Syllabus

0.1 Purpose of this Document

This syllabus forms the basis for the International Software Testing Qualification at the Advanced Level for the Test Analyst. The ISTQB® provides this syllabus as follows:

1. To National Boards, to translate into their local language and to accredit training providers. National Boards may adapt the syllabus to their particular language needs and modify the references to adapt to their local publications.

2. To Exam Boards, to derive examination questions in their local language adapted to the learning objectives for each syllabus.

3. To training providers, to produce courseware and determine appropriate teaching methods.

4. To certification candidates, to prepare for the exam (as part of a training course or independently).

5. To the international software and systems engineering community, to advance the profession of software and systems testing, and as a basis for books and articles.

The ISTQB® may allow other entities to use this syllabus for other purposes, provided they seek and obtain prior written permission.

0.2 Overview

The Advanced Level is comprised of three separate syllabi:

- Test Manager

- Test Analyst

- Technical Test Analyst

The Advanced Level Overview document [ISTQB_AL_OVIEW] includes the following information:

- Business Outcomes for each syllabus

- Summary for each syllabus

- Relationships among the syllabi

- Description of cognitive levels (K-levels)

- Appendices

0.3 Examinable Learning Objectives

The Learning Objectives support the Business Outcomes and are used to create the examination for achieving the Advanced Test Analyst Certification. In general all parts of this syllabus are examinable at a K1 level. That is, the candidate will recognize, remember and recall a term or concept. The learning objectives at K2, K3 and K4 levels are shown at the beginning of the pertinent chapter.

1. Testing Process - 300 mins.

Keywords

concrete test case, exit criteria, high-level test case, logical test case, low-level test case, test control, test design, test execution, test implementation, test planning

Learning Objectives for Testing Process

1.2 Testing in the Software Development Lifecycle

TA-1.2.1

(K2) Explain how and why the timing and level of involvement for the Test Analyst varies when working with different lifecycle models

1.3 Test Monitoring, Planning and Control

TA-1.3.1

(K2) Summarize the activities performed by the Test Analyst in support of planning and controlling the testing

1.4 Test Analysis

TA-1.4.1

(K4) Analyze a given scenario, including a project description and lifecycle model, to determine appropriate tasks for the Test Analyst during the analysis and design phases

1.5 Test Design

TA-1.5.1

(K2) Explain why test conditions should be understood by the stakeholders

TA-1.5.2

(K4) Analyze a project scenario to determine the most appropriate use for low-level (concrete) and high-level (logical) test cases

1.6 Test Implementation

TA-1.6.1

(K2) Describe the typical exit criteria for test analysis and test design and explain how meeting those criteria affect the test implementation effort.

1.7 Test Execution

TA-1.7.1

(K3) For a given scenario, determine the steps and considerations that should be taken when executing tests

1.8 Evaluating Exit Criteria and Reporting

TA-1.8.1

(K2) Explain why accurate test case execution status information is important

1.9 Test Closure Activities

TA-1.9.1

(K2) Provide examples of work products that should be delivered by the Test Analyst during test closure activities

1.1 Introduction

In the ISTQB® Foundation Level syllabus, the fundamental test process was described as including the following activities:

- Planning, monitoring and control

- Analysis and design

- Implementation and execution

- Evaluating exit criteria and reporting

- Test closure activities

At the Advanced Level, some of these activities are considered separately in order to provide additional refinement and optimization of the processes, to better fit the software development lifecycle, and to facilitate effective test monitoring and control. The activities at this level are considered as follows:

- Planning, monitoring and control

- Analysis

- Design

- Implementation

- Execution

- Evaluating exit criteria and reporting

- Test closure activities

These activities can be implemented sequentially or some can be implemented in parallel, e.g., design could be performed in parallel with implementation (e.g., exploratory testing). Determining the right tests and test cases, designing them and executing them are the primary areas of concentration for the Test Analyst. While it is important to understand the other steps in the test process, the majority of the Test Analyst’s work usually is done during the analysis, design, implementation and execution activities of the testing project.

Advanced testers face a number of challenges when introducing the different testing aspects described in this syllabus into the context of their own organizations, teams and tasks. It is important to consider the different software development lifecycles as well as the type of system being tested as these factors can influence the approach to testing.

1.2 Testing in the Software Development Lifecycle

The long-term lifecycle approach to testing should be considered and defined as part of the testing strategy. The moment of involvement for the Test Analyst is different for the various lifecycles and the amount of involvement, time required, information available and expectations can be quite varied as well. Because the testing processes do not occur in isolation, the Test Analyst must be aware of the points where information may be supplied to the other related organizational areas such as:

- Requirements engineering and management - requirements reviews

- Project management - schedule input

- Configuration and change management - build verification testing, version control

- Software development - anticipating what is coming, and when

- Software maintenance - defect management, turnaround time (i.e., time from defect discovery to defect resolution)

- Technical support - accurate documentation for workarounds

- Production of technical documentation (e.g., database design specifications) - input to these documents as well as technical review of the documents

Testing activities must be aligned with the chosen software development lifecycle model whose nature may be sequential, iterative, or incremental. For example, in the sequential V-model, the ISTQB® fundamental test process applied to the system test level could align as follows:

- System test planning occurs concurrently with project planning, and test control continues until system test execution and closure are complete.

- System test analysis and design occur concurrently with requirements specification, system and architectural (high-level) design specification, and component (low-level) design specification.

- System test environment (e.g., test beds, test rig) implementation might start during system design, though the bulk of it typically would occur concurrently with coding and component test, with work on system test implementation activities stretching often until just days before the start of system test execution.

- System test execution begins when the system test entry criteria are all met (or waived), which typically means that at least component testing and often also component integration testing are complete. System test execution continues until the system test exit criteria are met.

- Evaluation of system test exit criteria and reporting of system test results occur throughout system test execution, generally with greater frequency and urgency as project deadlines approach.

- System test closure activities occur after the system test exit criteria are met and system test execution is declared complete, though they can sometimes be delayed until after acceptance testing is over and all project activities are finished.

Iterative and incremental models may not follow the same order of tasks and may exclude some tasks. For example, an iterative model may utilize a reduced set of the standard test processes for each iteration. Analysis and design, implementation and execution, and evaluation and reporting may be conducted for each iteration, whereas planning is done at the beginning of the project and the closure reporting is done at the end. In an Agile project, it is common to use a less formalized process and a much closer working relationship that allows changes to occur more easily within the project. Because Agile is a “light weight” process, there is less comprehensive test documentation in favor of having a more rapid method of communication such as daily “stand up” meetings (called “stand up” because they are very quick, usually 10-15 minutes, so no one needs to sit down and everyone stays engaged).

Agile projects, out of all the lifecycle models, require the earliest involvement from the Test Analyst. The Test Analyst should expect to be involved from the initiation of the project, working with the developers as they do their initial architecture and design work. Reviews may not be formalized but are continuous as the software evolves. Involvement is expected to be throughout the project and the Test Analyst should be available to the team. Because of this immersion, members of Agile teams are usually dedicated to single projects and are fully involved in all aspects of the project.

Iterative/incremental models range from the Agile approach, where there is an expectation for change as the software evolves, to iterative/incremental development models that exist within a V-model (sometimes called embedded iterative). In the case with an embedded iterative model, the Test Analyst should expect to be involved in the standard planning and design aspects, but would then move to a more interactive role as the software is developed, tested, changed and deployed.

Whatever the software development lifecycle being used, it is important for the Test Analyst to understand the expectations for involvement as well as the timing of that involvement. There are many hybrid models in use, such as the iterative within a V-model noted above. The Test Analyst often must determine the most effective role and work toward that rather than depending on the definition of a set model to indicate the proper moment of involvement.

1.3 Test Planning, Monitoring and Control

This section focuses on the processes of planning, monitoring and controlling testing.

1.3.1 Test Planning

Test planning for the most part occurs at the initiation of the test effort and involves the identification and planning of all of the activities and resources required to meet the mission and objectives identified in the test strategy. During test planning it is important for the Test Analyst, working with the Test Manager, to consider and plan for the following:

- Be sure the test plans are not limited to functional testing. All types of testing should be considered in the test plan and scheduled accordingly. For example, in addition to functional testing, the Test Analyst may be responsible for usability testing. That type of testing must also be covered in a test plan document.

- Review the test estimates with the Test Manager and ensure adequate time is budgeted for the procurement and validation of the testing environment.

- Plan for configuration testing. If multiple types of processors, operating systems, virtual machines, browsers, and various peripherals can be combined into many possible configurations, plan to apply testing techniques that will provide adequate coverage of these combinations.

- Plan to test the documentation. Users are provided with the software and with documentation. The documentation must be accurate to be effective. The Test Analyst must allocate time to verify the documentation and may need to work with the technical writing staff to help prepare data to be used for screen shots and video clips.

- Plan to test the installation procedures. Installation procedures, as well as backup and restore procedures, must be tested sufficiently. These procedures can be more critical than the software; if the software cannot be installed, it will not be used at all. This can be difficult to plan since the Test Analyst is often doing the initial testing on a system that has been pre-configured without the final installation processes in place.

- Plan the testing to align with the software lifecycle. Sequential execution of tasks does not fit into most schedules. Many tasks often need to be performed (at least partly) concurrently. The Test Analyst must be aware of the selected lifecycle and the expectations for involvement during the design, development and implementation of the software. This also includes allocating time for confirmation and regression testing.

- Allow adequate time for identifying and analyzing risks with the cross-functional team. Although usually not responsible for organizing the risk management sessions, the Test Analyst should expect to be involved actively in these activities.

Complex relationships may exist among the test basis, test conditions and test cases such that many-to-many relationships may exist among these work products. These need to be understood to enable test planning and control to be effectively implemented. The Test Analyst is usually the best person to determine these relationships and to work to separate dependencies as much as possible.

1.3.2 Test Monitoring and Control

While test monitoring and control is usually the job of the Test Manager, the Test Analyst contributes the measurements that make the control possible.

A variety of quantitative data should be gathered throughout the software development lifecycle (e.g., percentage of planning activities completed, percentage of coverage attained, number of test cases that have passed, failed). In each case a baseline (i.e., reference standard) must be defined and then progress tracked with relation to this baseline. While the Test Manager will be concerned with compiling and reporting the summarized metric information, the Test Analyst gathers the information for each metric. Each test case that is completed, each defect report that is written, each milestone that is achieved will roll up into the overall project metrics. It is important that the information entered into the various tracking tools be as accurate as possible so the metrics reflect reality.

Accurate metrics allow managers to manage a project (monitor) and to initiate changes as needed (control). For example, a high number of defects being reported from one area of the software may indicate that additional testing effort is needed in that area. Requirements and risk coverage information (traceability) may be used to prioritize remaining work and to allocate resources. Root cause information is used to determine areas for process improvement. If the data that is recorded is accurate, the project can be controlled and accurate status information can be reported to the stakeholders. Future projects can be planned more effectively when the planning considers data gathered from past projects. There are myriad uses for accurate data. It is part of the Test Analyst’s job to ensure that the data is accurate, timely and objective.

1.4 Test Analysis

During test planning, the scope of the testing project is defined. The Test Analyst uses this scope definition to:

- Analyze the test basis

- Identify the test conditions

In order for the Test Analyst to proceed effectively with test analysis, the following entry criteria should be met:

- There is a document describing the test object that can serve as the test basis

- This document has passed review with reasonable results and has been updated as needed after the review

- There is a reasonable budget and schedule available to accomplish the remaining testing work for this test object

Test conditions are typically identified by analysis of the test basis and the test objectives. In some situations, where documentation may be old or non-existent, the test conditions may be identified by talking to relevant stakeholders (e.g., in workshops or during sprint planning). These conditions are then used to determine what to test, using test design techniques identified within the test strategy and/or the test plan.

While test conditions are usually specific to the item being tested, there are some standard considerations for the Test Analyst.

- It is usually advisable to define test conditions at differing levels of detail. Initially, high-level conditions are identified to define general targets for testing, such as “functionality of screen x”. Subsequently, more detailed conditions are identified as the basis of specific test cases, such as “screen x rejects an account number that is one digit short of the correct length”. Using this type of hierarchical approach to defining test conditions can help to ensure the coverage is sufficient for the high-level items.

- If product risks have been defined, then the test conditions that will be necessary to address each product risk must be identified and traced back to that risk item.

At the conclusion of the test analysis activities, the Test Analyst should know what specific tests must be designed in order to meet the needs of the assigned areas of the test project.

1.5 Test Design

Still adhering to the scope determined during test planning, the test process continues as the Test Analyst designs the tests which will be implemented and executed. The process of test design includes the following activities:

- Determine in which test areas low-level (concrete) or high-level (logical) test cases are most appropriate

- Determine the test case design technique(s) that provide the necessary test coverage

- Create test cases that exercise the identified test conditions

Prioritization criteria identified during risk analysis and test planning should be applied throughout the process, from analysis and design to implementation and execution.

Depending on the types of tests being designed, one of the entry criteria for test design may be the availability of tools that will be used during the design work.

When designing tests, it is important to remember the following:

- Some test items are better addressed by defining only the test conditions rather than going further into defining scripted tests. In this case, the test conditions should be defined to be used as a guide for the unscripted testing.

- The pass/fail criteria should be clearly identified.

- Tests should be designed to be understandable by other testers, not just the author. If the author is not the person who executes the test, other testers will need to read and understand previously specified tests in order to understand the test objectives and the relative importance of the test.

- Tests must also be understandable by other stakeholders such as developers, who will review the tests, and auditors, who may have to approve the tests.

- Tests should be designed to cover all the interactions of the software with the actors (e.g., end users, other systems), not just the interactions that occur through the user-visible interface. Inter-process communications, batch execution and other interrupts also interact with the software and can contain defects so the Test Analyst must design tests to mitigate these risks.

- Tests should be designed to test the interfaces between the various test objects.

1.5.1 Concrete and Logical Test Cases

One of the jobs of the Test Analyst is to determine the best types of test cases for a given situation. Concrete test cases provide all the specific information and procedures needed for the tester to execute the test case (including any data requirements) and verify the results. Concrete test cases are useful when the requirements are well-defined, when the testing staff is less experienced and when external verification of the tests, such as audits, is required. Concrete test cases provide excellent reproducibility (i.e., another tester will get the same results), but may also require a significant amount of maintenance effort and tend to limit tester ingenuity during execution.

Logical test cases provide guidelines for what should be tested, but allow the Test Analyst to vary the actual data or even the procedure that is followed when executing the test. Logical test cases may provide better coverage than concrete test cases because they will vary somewhat each time they are executed. This also leads to a loss in reproducibility. Logical test cases are best used when the requirements are not well-defined, when the Test Analyst who will be executing the test is experienced with both testing and the product, and when formal documentation is not required (e.g., no audits will be conducted). Logical test cases may be defined early in the requirements process when the requirements are not yet well-defined. These test cases may be used to develop concrete test cases when the requirements become more defined and stable. In this case, the test case creation is done sequentially, flowing from logical to concrete with only the concrete test cases used for execution.

1.5.2 Creation of Test Cases

Test cases are designed by the stepwise elaboration and refinement of the identified test conditions using test design techniques (see Chapter 3) identified in the test strategy and/or the test plan. The test cases should be repeatable, verifiable and traceable back to the test basis (e.g., requirements) as dictated by the test strategy that is being used.

Test case design includes the identification of the following:

- Objective

- Preconditions, such as either project or localized test environment requirements and the plans for their delivery, state of the system, etc.

- Test data requirements (both input data for the test case as well as data that must exist in the system for the test case to be executed)

- Expected results

- Post-conditions, such as affected data, state of the system, triggers for subsequent processing, etc.

The level of detail of the test cases, which impacts both the cost to develop and the level of repeatability during execution, should be defined prior to actually creating the test cases. Less detail in the test case allows the Test Analyst more flexibility when executing the test case and provides an opportunity to investigate potentially interesting areas. Less detail, however, also tends to lead to less reproducibility.

A particular challenge is often the definition of the expected result of a test. Computing this manually is often tedious and error-prone; if possible, it is preferable to find or create an automated test oracle. In identifying the expected result, testers are concerned not only with outputs on the screen, but also with data and environmental post-conditions. If the test basis is clearly defined, identifying the correct result, theoretically, should be simple. However, test bases are often vague, contradictory, lacking coverage of key areas, or missing entirely. In such cases, a Test Analyst must have, or have access to, subject matter expertise. Also, even where the test basis is well-specified, complex interactions of complex stimuli and responses can make the definition of the expected results difficult; therefore, a test oracle is essential. Test case execution without any way to determine correctness of results has a very low added value or benefit, often generating spurious failure reports or false confidence in the system.

The activities described above may be applied to all test levels, though the test basis will vary. For example, user acceptance tests may be based primarily on the requirements specification, use cases and defined business processes, while component tests may be based primarily on low-level design specifications, user stories and the code itself. It is important to remember that these activities occur throughout all the test levels although the target of the test may vary. For example, functional testing at the unit level is designed to ensure that a particular component provides the functionality as specified in the detailed design for that component. Functional testing at the integration level is verifying that components interact together and provide functionality through their interaction. At the system level, end to end functionality should be a target of the testing. When analyzing and designing tests, it is important to remember the target level for the test as well as the objective of the test. This helps to determine the level of detail required as well as any tools that may be needed (e.g., drivers and stubs at the component test level).

During the development of test conditions and test cases, some amount of documentation is typically created, resulting in test work products. In practice the extent to which test work products are documented varies considerably. This can be affected by any of the following:

- Project risks (what must/must not be documented)

- The “value added” which the documentation brings to the project

- Standards to be followed and/or regulations to be met

- Lifecycle model used (e.g., an Agile approach aims for “just enough” documentation)

- The requirement for traceability from the test basis through test analysis and design

Depending on the scope of the testing, test analysis and design address the quality characteristics for the test object(s). The ISO 25000 standard [ISO25000] (which is replacing ISO 9126) provides a useful reference. When testing hardware/software systems, additional characteristics may apply.

The processes of test analysis and test design may be enhanced by intertwining them with reviews and static analysis. In fact, conducting the test analysis and test design are often a form of static testing because problems may be found in the basis documents during this process. Test analysis and test design based on the requirements specification is an excellent way to prepare for a requirements review meeting. Reading the requirements to use them for creating tests requires understanding the requirement and being able to determine a way to assess fulfillment of the requirement. This activity often uncovers requirements that are not clear, are untestable or do not have defined acceptance criteria. Similarly, test work products such as test cases, risk analyses, and test plans should be subjected to reviews.

Some projects, such as those following an Agile lifecycle, may have only minimally documented requirements. These are sometimes in the form of “user stories” which describe small but demonstrable bits of functionality. A user story should include a definition of the acceptance criteria. If the software is able to demonstrate that it has fulfilled the acceptance criteria, it is usually considered to be ready for integration with the other completed functionality or may already have been integrated in order to demonstrate its functionality.

During test design the required detailed test infrastructure requirements may be defined, although in practice these may not be finalized until test implementation. It must be remembered that test infrastructure includes more than test objects and testware. For example the infrastructure requirements may include rooms, equipment, personnel, software, tools, peripherals, communications equipment, user authorizations, and all other items required to run the tests.

The exit criteria for test analysis and test design will vary depending on the project parameters, but all items discussed in these two sections should be considered for inclusion in the defined exit criteria. It is important that the criteria be measurable and ensure that all the information and preparation required for the subsequent steps have been provided.

1.6 Test Implementation

Test implementation is the fulfillment of the test design. This includes creating automated tests, organizing tests (both manual and automated) into execution order, finalizing test data and test environments, and forming a test execution schedule, including resource allocation, to enable test case execution to begin. This also includes checking against explicit and implicit entry criteria for the test level in question and ensuring that the exit criteria for the previous steps in the process have been met. If the exit criteria have been skipped, either for the test level or for a step in the test process, the implementation effort is likely to be affected with delayed schedules, insufficient quality and unexpected extra effort. It is important to ensure that all exit criteria have been met prior to starting the test implementation effort.

When determining the execution order, there may be many considerations. In some cases, it may make sense to organize the test cases into test suites (i.e., groups of test cases). This can help organize the testing so that related test cases are executed together. If a risk-based testing strategy is being used, risk priority order may dictate the execution order for the test cases. There may be other factors that determine order such as the availability of the right people, equipment, data and the functionality to be tested. It is not unusual for code to be released in sections and the test effort has to be coordinated with the order in which the software becomes available for test. Particularly in incremental lifecycle models, it is important for the Test Analyst to coordinate with the development team to ensure that the software will be released for testing in a testable order. During test implementation, Test Analysts should finalize and confirm the order in which manual and automated tests are to be run, carefully checking for constraints that might require tests to be run in a particular order. Dependencies must be documented and checked.

The level of detail and associated complexity for work done during test implementation may be influenced by the detail of the test cases and test conditions. In some cases regulatory rules apply, and tests should provide evidence of compliance to applicable standards such as the United States Federal Aviation Administration’s DO-178B/ED 12B [RTCA DO-178B/ED-12B].

As specified above, test data is needed for testing, and in some cases these sets of data can be quite large. During implementation, Test Analysts create input and environment data to load into databases and other such repositories. Test Analysts also create data to be used with data-driven automation tests as well as for manual testing.

Test implementation is also concerned with the test environment(s). During this stage the environment(s) should be fully set up and verified prior to test execution. A “fit for purpose” test environment is essential, i.e., the test environment should be capable of enabling the exposure of the defects present during controlled testing, operate normally when failures are not occurring, and adequately replicate, if required, the production or end-user environment for higher levels of testing. Test environment changes may be necessary during test execution depending on unanticipated changes, test results or other considerations. If environment changes do occur during execution, it is important to assess the impact of the changes to tests that have already been run.

During test implementation, testers must ensure that those responsible for the creation and maintenance of the test environment are known and available, and that all the testware and test support tools and associated processes are ready for use. This includes configuration management, defect management, and test logging and management. In addition, Test Analysts must verify the procedures that gather data for exit criteria evaluation and test results reporting.

It is wise to use a balanced approach to test implementation as determined during test planning. For example, risk-based analytical test strategies are often blended with dynamic test strategies. In this case, some percentage of the test implementation effort is allocated to testing which does not follow predetermined scripts (unscripted).

Unscripted testing should not be ad hoc or aimless as this can be unpredictable in duration and coverage unless time boxed and chartered. Over the years, testers have developed a variety of experience-based techniques, such as attacks, error guessing [Myers79], and exploratory testing. Test analysis, test design, and test implementation still occur, but they occur primarily during test execution. When following such dynamic test strategies, the results of each test influence the analysis, design, and implementation of the subsequent tests. While these strategies are lightweight and often effective at finding defects, there are some drawbacks. These techniques require expertise from the Test Analyst, duration can be difficult to predict, coverage can be difficult to track and repeatability can be lost without good documentation or tool support.

1.7 Test Execution

Test execution begins once the test object is delivered and the entry criteria to test execution are satisfied (or waived). Tests should be executed according to the plan determined during test implementation, but the Test Analyst should have adequate time to ensure coverage of additional interesting test scenarios and behaviors that are observed during testing (any failure detected during such deviations should be described including the variations from the scripted test case that are necessary to reproduce the failure). This integration of scripted and unscripted (e.g., exploratory) testing techniques helps to guard against test escapes due to gaps in scripted coverage and to circumvent the pesticide paradox.

At the heart of the test execution activity is the comparison of actual results with expected results. Test Analysts must bring attention and focus to these tasks, otherwise all the work of designing and implementing the test can be wasted when failures are missed (false-negative result) or correct behavior is misclassified as incorrect (false-positive result). If the expected and actual results do not match, an incident has occurred. Incidents must be carefully scrutinized to determine the cause (which might or might not be a defect in the test object) and to gather data to assist with the resolution of the incident (see Chapter 6 for further details on defect management).

When a failure is identified, the test documentation (test specification, test case, etc.) should be carefully evaluated to ensure correctness. A test document can be incorrect for a number of reasons. If it is incorrect, it should be corrected and the test should be re-run. Since changes in the test basis and the test object can render a test case incorrect even after the test has been run successfully many times, testers should remain aware of the possibility that the observed results could be due to an incorrect test.

During test execution, test results must be logged appropriately. Tests which were run but for which results were not logged may have to be repeated to identify the correct result, leading to inefficiency and delays. (Note that adequate logging can address the coverage and repeatability concerns associated with test techniques such as exploratory testing.) Since the test object, testware, and test environments may all be evolving, logging should identify the specific versions tested as well as specific environment configurations. Test logging provides a chronological record of relevant details about the execution of tests.

Results logging applies both to individual tests and to activities and events. Each test should be uniquely identified and its status logged as test execution proceeds. Any events that affect the test execution should be logged. Sufficient information should be logged to measure test coverage and document reasons for delays and interruptions in testing. In addition, information must be logged to support test control, test progress reporting, measurement of exit criteria, and test process improvement.

Logging varies depending on the level of testing and the strategy. For example, if automated component testing is occurring, the automated tests should produce most of the logging information. If manual testing is occurring, the Test Analyst will log the information regarding the test execution, often into a test management tool that will track the test execution information. In some cases, as with test implementation, the amount of test execution information that is logged is influenced by regulatory or audit requirements.

In some cases, users or customers may participate in test execution. This can serve as a way to build their confidence in the system, though that presumes that the tests find few defects. Such an assumption is often invalid in early test levels, but might be valid during acceptance test.

The following are some specific areas that should be considered during test execution:

- Notice and explore “irrelevant” oddities. Observations or results that may seem irrelevant are often indicators for defects that (like icebergs) are lurking beneath the surface.

- Check that the product is not doing what it is not supposed to do. Checking that the product does what it is supposed to do is a normal focus of testing, but the Test Analyst must also be sure the product is not misbehaving by doing something it should not be doing (for example, additional undesired functions).

- Build the test suite and expect it to grow and change over time. The code will evolve and additional tests will need to be implemented to cover these new functionalities, as well as to check for regressions in other areas of the software. Gaps in testing are often discovered during execution. Building the test suite is a continuous process.

- Take notes for the next testing effort. The testing tasks do not end when the software is provided to the user or distributed to the market. A new version or release of the software will most likely be produced, so knowledge should be stored and transferred to the testers responsible for the next testing effort.

- Do not expect to rerun all manual tests. It is unrealistic to expect that all manual tests will be rerun. If a problem is suspected, the Test Analyst should investigate it and note it rather than assume it will be caught in a subsequent execution of the test cases.

- Mine the data in the defect tracking tool for additional test cases. Consider creating test cases for defects that were discovered during unscripted or exploratory testing and add them to the regression test suite.

- Find the defects before regression testing. Time is often limited for regression testing and finding failures during regression testing can result in schedule delays. Regression tests generally do not find a large proportion of the defects, mostly because they are tests which have already been run (e.g., for a previous version of same software), and defects should have been detected in those previous runs. This does not mean that regression tests should be eliminated altogether, only that the effectiveness of regression tests, in terms of the capacity to detect new defects, is lower than other tests.

1.8 Evaluating Exit Criteria and Reporting

From the point of view of the test process, test progress monitoring entails ensuring the collection of proper information to support the reporting requirements. This includes measuring progress towards completion. When the exit criteria are defined in the planning stages, there may be a breakdown of “must” and “should” criteria. For example, the criteria might state that there “must be no open Priority 1 or Priority 2 bugs” and there “should be 95% pass rate across all test cases”. In this case, a failure to meet the “must” criteria should cause the exit criteria to fail whereas a 93% pass rate could still allow the project to proceed to the next level. The exit criteria must be clearly defined so they can be objectively assessed.

The Test Analyst is responsible for supplying the information that is used by the Test Manager to evaluate progress toward meeting the exit criteria and for ensuring that the data is accurate. If, for example, the test management system provides the following status codes for test case completion:

- Passed

- Failed

- Passed with exception

then the Test Analyst must be very clear on what each of these means and must apply that status consistently. Does “passed with exception” mean that a defect was found but it is not affecting the functionality of the system? What about a usability defect that causes the user to be confused? If the pass rate is a “must” exit criterion, counting a test case as “failed” rather than “passed with exception” becomes a critical factor. There must also be consideration for test cases that are marked as “failed” but the cause of the failure is not a defect (e.g., the test environment was improperly configured). If there is any confusion on the metrics being tracked or the usage of the status values, the Test Analyst must clarify with the Test Manager so the information can be tracked accurately and consistently throughout the project.

It is not unusual for the Test Analyst to be asked for a status report during the testing cycles as well as to contribute to the final report at the end of the testing. This may require gathering metrics from the defect and test management systems as well as assessing the overall coverage and progress. The Test Analyst should be able to use the reporting tools and be able to provide the requested information for the Test Manager to extract the information needed.

1.9 Test Closure Activities

Once test execution is determined to be complete, the key outputs from the testing effort should be captured and either passed to the relevant person or archived. Collectively, these are test closure activities. The Test Analyst should expect to be involved in delivering work products to those who will need them. For example, known defects deferred or accepted should be communicated to those who will use and support the use of the system. Tests and test environments should be given to those responsible for maintenance testing. Another work product may be a regression test set (either automated or manual). Information about test work products must be clearly documented, including appropriate links, and appropriate access rights must be granted.

The Test Analyst should also expect to participate in retrospective meetings (“lessons learned”) where important lessons (both from within the testing project and across the whole software development lifecycle) can be documented and plans established to reinforce the “good” and eliminate, or at least control, the “bad”. The Test Analyst is a knowledgeable source of information for these meetings and must participate if valid process improvement information is to be gathered. If only the Test Manager will attend the meeting, the Test Analyst must convey the pertinent information to the Test Manager so an accurate picture of the project is presented.

Archiving results, logs, reports, and other documents and work products in the configuration management system must also be done. This task often falls to the Test Analyst and is an important closure activity, particularly if a future project will require the use of this information.

While the Test Manager usually determines what information should be archived, the Test Analyst should also think about what information would be needed if the project were to be started up again at a future time. Thinking about this information at the end of a project can save months of effort when the project is started up again at a later time or with another team.

2. Test Management: Responsibilities for the Test Analyst - 90 mins.

Keywords

product risk, risk analysis, risk identification, risk level, risk management, risk mitigation, risk-based testing, test monitoring, test strategy

Learning Objectives for Test Management: Responsibilities for the Test Analyst

2.2 Test Progress Monitoring and Control

TA-2.2.1

(K2) Explain the types of information that must be tracked during testing to enable adequate monitoring and controlling of the project

2.3 Distributed, Outsourced and Insourced Testing

TA-2.3.1

(K2) Provide examples of good communication practices when working in a 24-hour testing environment

2.4 The Test Analyst’s Tasks in Risk-Based Testing

TA-2.4.1

(K3) For a given project situation, participate in risk identification, perform risk assessment and propose appropriate risk mitigation

2.1 Introduction

While there are many areas in which the Test Analyst interacts with and supplies data for the Test Manager, this section concentrates on the specific areas of the testing process in which the Test Analyst is a major contributor. It is expected that the Test Manager will seek the information needed from the Test Analyst.

2.2 Test Progress Monitoring and Control

There are five primary dimensions in which test progress is monitored:

- Product (quality) risks

- Defects

- Tests

- Coverage

- Confidence

Product risks, defects, tests, and coverage can be and often are measured and reported in specific ways during the project or operation by the Test Analyst. Confidence, though measurable through surveys, is usually reported subjectively. Gathering the information needed to support these metrics is part of the Test Analyst’s daily work. It is important to remember that the accuracy of this data is critical as inaccurate data will create inaccurate trending information and may lead to inaccurate conclusions. At its worst, inaccurate data will result in incorrect management decisions and damage to the credibility of the test team.

When using a risk-based testing approach, the Test Analyst should be tracking:

- Which risks have been mitigated by testing

- Which risks are considered to be unmitigated

Tracking risk mitigation is often done with a tool that also tracks test completion (e.g., test management tools). This requires that the identified risks are mapped to the test conditions which are mapped to the test cases that will mitigate the risks if the test cases are executed and passed. In this way, the risk mitigation information is updated automatically as the test cases are updated. This can be done for both manual and automated tests.

Defect tracking is usually done via a defect tracking tool. As defects are recorded, particular classification information about each defect is recorded as well. This information is used to produce trends and graphs that indicate the progress of testing and the quality of the software. Classification information is discussed in more detail in the Defect Management chapter. The lifecycle may affect the amount of defect documentation that is recorded and the methods used to record the information.

As the testing is conducted, test case status information should be recorded. This is usually done via a test management tool but can be done by manual means if needed. Test case information can include:

- Test case creation status (e.g., designed, reviewed)

- Test case execution status (e.g., passed, failed, blocked, skipped)

- Test case execution information (e.g., date and time, tester name, data used)

- Test case execution artifacts (e.g., screen shots, accompanying logs)

As with the identified risk items, the test cases should be mapped to the requirements items they test. It is important for the Test Analyst to remember that if test case A is mapped to requirement A, and it is the only test case mapped to that requirement, then when test case A is executed and passes, requirement A will be considered to be fulfilled. This may or may not be correct. In many cases, more test cases are needed to thoroughly test a requirement, but because of limited time, only a subset of those tests is actually created. For example, if 20 test cases were needed to thoroughly test the implementation of a requirement, but only 10 were created and run, then the requirements coverage information will indicate 100% coverage when in fact only 50% coverage was achieved. Accurate tracking of the coverage as well as tracking the reviewed status of the requirements themselves can be used as a confidence measure.

The amount (and level of detail) of information to be recorded depends on several factors, including the software development lifecycle model. For example, in Agile projects typically less status information will be recorded due to the close interaction of the team and more face-to-face communication.

2.3 Distributed, Outsourced and Insourced Testing

In many cases, not all of the test effort is carried out by a single test team, composed of fellow employees of the rest of the project team, at a single and same location as the rest of the project team. If the test effort occurs at multiple locations, that test effort may be called distributed. If it occurs at a single location it may be called centralized. If the test effort is carried out at one or more locations by people who are not fellow employees of the rest of the project team and who are not co-located with the project team, that test effort may be called outsourced. If the test effort is carried out by people who are co-located with the project team but who are not fellow employees, that test effort may be called insourced.

When working in a project in which some of the test team is spread across multiple locations or even across multiple companies, the Test Analyst must pay special attention to effective communication and information transfer. Some organizations work on a “24 hour testing” model in which the team in one time zone is expected to hand off the work to the team in another time zone to allow testing to continue around the clock. This requires special planning on the part of the Test Analyst who will hand off and receive work. Good planning is important to understand responsibilities, but it is vital to ensure that the proper information is available.

When verbal communication is not available, written communication must suffice. This means that email, status reports and effective use of the test management and defect tracking tools must be employed. If the test management tool allows tests to be assigned to individuals, it can also work as a scheduling tool and an easy way to transfer work between people. Defects that are accurately recorded can be routed to co-workers for follow-up as needed. Effective use of these communication systems is vital for an organization that cannot rely on daily personal interaction.

2.4 The Test Analyst’s Tasks in Risk-Based Testing

2.4.1 Overview

The Test Manager often has overall responsibility for establishing and managing a risk-based testing strategy. The Test Manager usually will request the involvement of the Test Analyst to ensure the risk-based approach is implemented correctly.

The Test Analyst should be actively involved in the following risk-based testing tasks:

- Risk identification

- Risk assessment

- Risk mitigation

These tasks are performed iteratively throughout the project lifecycle to deal with emerging risks, changing priorities and to regularly evaluate and communicate risk status.

Test Analysts should work within the risk-based testing framework established by the Test Manager for the project. They should contribute their knowledge of the business domain risks that are inherent in the project such as risks related to safety, business and economic concerns, and political factors.

2.4.2 Risk Identification

By calling on the broadest possible sample of stakeholders, the risk identification process is most likely to detect the largest possible number of significant risks. Because Test Analysts often possess unique knowledge regarding the particular business domain of the system under test, they are particularly well-suited for conducting expert interviews with the domain experts and users, conducting independent assessments, using and facilitating the use of risk templates, conducting risk workshops, conducting brainstorming sessions with potential and current users, defining testing checklists and calling on past experience with similar systems or projects. In particular, the Test Analyst should work closely with the users and other domain experts to determine the areas of business risk that should be addressed during testing. The Test Analyst also can be particularly helpful in identifying the potential effects of risk on the users and stakeholders.

Sample risks that might be identified in a project include:

- Accuracy issues with the software functionality, e.g., incorrect calculations

- Usability issues, e.g., insufficient keyboard shortcuts

- Learnability issues, e.g., lack of instructions for the user at key decision points

Considerations regarding testing the specific quality characteristics are covered in Chapter 4 of this syllabus.

2.4.3 Risk Assessment

While risk identification is about identifying as many pertinent risks as possible, risk assessment is the study of these identified risks. Specifically, categorizing each risk and determining the likelihood and impact associated with each risk.

Determining the level of risk typically involves assessing, for each risk item, the likelihood of occurrence and the impact upon occurrence. The likelihood of occurrence is usually interpreted as the likelihood that the potential problem can exist in the system under test and will be observed when the system is in production. In other words, it arises from technical risk. The Technical Test Analyst should contribute to finding and understanding the potential technical risk level for each risk item whereas the Test Analyst contributes to understanding the potential business impact of the problem should it occur.

The impact upon occurrence is often interpreted as the severity of the effect on the users, customers, or other stakeholders. In other words, it arises from business risk. The Test Analyst should contribute to identifying and assessing the potential business domain or user impact for each risk item. Factors influencing business risk include:

- Frequency of use of the affected feature

- Business loss

- Potential financial, ecological or social losses or liability

- Civil or criminal legal sanctions

- Safety concerns

- Fines, loss of license

- Lack of reasonable workarounds

- Visibility of the feature

- Visibility of failure leading to negative publicity and potential image damage

- Loss of customers

Given the available risk information, the Test Analyst needs to establish the levels of business risk per the guidelines established by the Test Manager. These could be classified with terms (e.g., low, medium, high) or numbers. Unless there is a way to objectively measure the risk on a defined scale it cannot be a true quantitative measure. Accurately measuring probability/likelihood and cost/consequence is usually very difficult, so determining risk level is usually done qualitatively. Numbers may be assigned to the qualitative value, but that does not make it a true quantitative measure. For example, the Test Manager may determine that business risk should be categorized with a value from 1 to 10, with 1 being the highest, and therefore riskiest, impact to the business. Once the likelihood (the assessment of the technical risk) and impact (the assessment of the business risk) have been assigned, these values may be multiplied together to determine the overall risk rating for each risk item. That overall rating is then used to prioritize the risk mitigation activities. Some risk-based testing models, such as PRISMA® [vanVeenendaal12], do not combine the risk values, allowing the test approach to address the technical and business risks separately.

2.4.4 Risk Mitigation

During the project, Test Analysts should seek to do the following:

- Reduce product risk by using well-designed test cases that demonstrate unambiguously whether test items pass or fail, and by participating in reviews of software artifacts such as requirements, designs, and user documentation

- Implement appropriate risk mitigation activities identified in the test strategy and test plan

- Re-evaluate known risks based on additional information gathered as the project unfolds, adjusting likelihood, impact, or both, as appropriate

- Recognize new risks identified by information obtained during testing

When one is talking about a product (quality) risk, then testing is a form of mitigation for such risks. By finding defects, testers reduce risk by providing awareness of the defects and opportunities to deal with the defects before release. If the testers find no defects, testing then reduces risk by ensuring that, under certain conditions (i.e., the conditions tested), the system operates correctly. Test Analysts help to determine risk mitigation options by investigating opportunities for gathering accurate test data, creating and testing realistic user scenarios and conducting or overseeing usability studies.

2.4.4.1 Prioritizing the Tests

The level of risk is also used to prioritize tests. A Test Analyst might determine that there is a high risk in the area of transactional accuracy in an accounting system. As a result, to mitigate the risk, the tester may work with other business domain experts to gather a strong set of sample data that can be processed and verified for accuracy. Similarly, a Test Analyst might determine that usability issues are a significant risk for a new product. Rather than wait for a user acceptance test to discover any issues, the Test Analyst might prioritize an early usability test to occur during the integration level to help identify and resolve usability problems early in the testing. This prioritization must be considered as early as possible in the planning stages so that the schedule can accommodate the necessary testing at the necessary time.

In some cases, all of the highest risk tests are run before any lower risk tests, and tests are run in strict risk order (often called “depth-first”); in other cases, a sampling approach is used to select a sample of tests across all the identified risks using risk to weight the selection while at the same time ensuring coverage of every risk at least once (often called “breadth-first”).

Whether risk-based testing proceeds depth-first or breadth-first, it is possible that the time allocated for testing might be consumed without all tests being run. Risk-based testing allows testers to report to management in terms of the remaining level of risk at this point, and allows management to decide whether to extend testing or to transfer the remaining risk onto the users, customers, help desk/technical support, and/or operational staff.

2.4.4.2 Adjusting Testing for Future Test Cycles

Risk assessment is not a one-time activity performed before the start of test implementation; it is a continuous process. Each future planned test cycle should be subjected to new risk analysis to take into account such factors as:

- Any new or significantly changed product risks

- Unstable or defect-prone areas discovered during the testing

- Risks from fixed defects

- Typical defects found during testing

- Areas that have been under-tested (low test coverage)

If additional time for testing is allocated it may be possible to expand the risk coverage into areas of lower risk.

3. Test Techniques - 825 mins.

Keywords

boundary value analysis (BVA), cause-effect graphing, checklist-based testing, classification tree method, combinatorial testing, decision table testing, defect taxonomy, defect-based technique, domain analysis, error guessing, equivalence partitioning, experience-based technique, exploratory testing, orthogonal array, orthogonal array testing, pairwise testing, requirements-based testing, specification-based technique, state transition testing, test charter, use case testing, user story testing

Learning Objectives for Test Techniques

3.2 Specification-Based Techniques

TA-3.2.1

(K2) Explain the use of cause-effects graphs

TA-3.2.2

(K3) Write test cases from a given specification item by applying the equivalence partitioning test design technique to achieve a defined level of coverage

TA-3.2.3

(K3) Write test cases from a given specification item by applying the boundary value analysis test design technique to achieve a defined level of coverage

TA-3.2.4

(K3) Write test cases from a given specification item by applying the decision table test design technique to achieve a defined level of coverage

TA-3.2.5

(K3) Write test cases from a given specification item by applying the state transition test design technique to achieve a defined level of coverage

TA-3.2.6

(K3) Write test cases from a given specification item by applying the pairwise test design technique to achieve a defined level of coverage

TA-3.2.7

(K3) Write test cases from a given specification item by applying the classification tree test design technique to achieve a defined level of coverage

TA-3.2.8

(K3) Write test cases from a given specification item by applying the use case test design technique to achieve a defined level of coverage

TA-3.2.9

(K2) Explain how user stories are used to guide testing in an Agile project

TA-3.2.10

(K3) Write test cases from a given specification item by applying the domain analysis test design technique to achieve a defined level of coverage

TA-3.2.11

(K4) Analyze a system, or its requirement specification, in order to determine likely types of defects to be found and select the appropriate specification-based technique(s)

3.3 Defect-Based Techniques

TA-3.3.1

(K2) Describe the application of defect-based testing techniques and differentiate their use from specification-based techniques

TA-3.3.2

(K4) Analyze a given defect taxonomy for applicability in a given situation using criteria for a good taxonomy

3.4 Experience-Based Techniques

TA-3.4.1

(K2) Explain the principles of experience-based techniques, and the benefits and drawbacks compared to specification-based and defect-based techniques

TA-3.4.2

(K3) For a given scenario, specify exploratory tests and explain how the results can be reported

TA-3.4.3

(K4) For a given project situation, determine which specification-based, defect-based or experience-based techniques should be applied to achieve specific goals

3.1 Introduction

The test design techniques considered in this chapter are divided into the following categories:

- Specification-based (or behavior-based or black box)

- Defect-based

- Experience-based

These techniques are complementary and may be used as appropriate for any given test activity, regardless of which level of testing is being performed.

Note that all three categories of techniques can be used to test both functional or non-functional quality characteristics. Testing non-functional characteristics is discussed in the next chapter.

The test design techniques discussed in these sections may focus primarily on determining optimal test data (e.g., equivalence partitions) or deriving test sequences (e.g., state models). It is common to combine techniques to create complete test cases.

3.2 Specification-Based Techniques

Specification-based techniques are applied to the test conditions to derive test cases based on an analysis of the test basis for a component or system without reference to its internal structure.

Common features of specification-based techniques include:

- Models, e.g., state transition diagrams and decision tables, are created during test design according to the test technique

- Test conditions are derived systematically from these models

Some techniques also provide coverage criteria, which can be used for measuring test design and test execution activities. Completely fulfilling the coverage criteria does not mean that the entire set of tests is complete, but rather that the model no longer suggests any additional tests to increase coverage based on that technique.

Specification-based tests are usually based on the system requirements documents. Since the requirements specification should specify how the system is to behave, particularly in the area of functionality, deriving tests from the requirements is often part of testing the behavior of the system. In some cases there may be no documented requirements but there are implied requirements such as replacing the functionality of a legacy system.

There are a number of specification-based testing techniques. These techniques target different types of software and scenarios. The sections below show the applicability for each technique, some limitations and difficulties that the Test Analyst may experience, the method by which test coverage is measured and the types of defects that are targeted.

3.2.1 Equivalence Partitioning

Equivalence partitioning (EP) is used to reduce the number of test cases that is required to effectively test the handling of inputs, outputs, internal values and time-related values. Partitioning is used to create equivalence classes (often called equivalence partitions) which are created of sets of values that are processed in the same manner. By selecting one representative value from a partition, coverage for all the items in the same partition is assumed.

Applicability

This technique is applicable at any level of testing and is appropriate when all the members of a set of values to be tested are expected to be handled in the same way and where the sets of values used by the application do not interact. The selection of sets of values is applicable to valid and invalid partitions (i.e., partitions containing values that should be considered invalid for the software being tested). This technique is strongest when used in combination with boundary value analysis which expands the test values to include those on the edges of the partitions. This is a commonly used technique for smoke testing a new build or a new release as it quickly determines if basic functionality is working.

Limitations/Difficulties

If the assumption is incorrect and the values in the partition are not handled in exactly the same way, this technique may miss defects. It is also important to select the partitions carefully. For example, an input field that accepts positive and negative numbers would be better tested as two valid partitions, one for the positive numbers and one for the negative numbers, because of the likelihood of different handling. Depending on whether or not zero is allowed, this could become another partition as well. It is important for the Test Analyst to understand the underlying processing in order to determine the best partitioning of the values.

Coverage

Coverage is determined by taking the number of partitions for which a value has been tested and dividing that number by the number of partitions that have been identified. Using multiple values for a single partition does not increase the coverage percentage.

Types of Defects

This technique finds functional defects in the handling of various data values.

3.2.2 Boundary Value Analysis

Boundary value analysis (BVA) is used to test the values that exist on the boundaries of ordered equivalence partitions. There are two ways to approach BVA: two value or three value testing. With two value testing, the boundary value (on the boundary) and the value that is just over the boundary (by the smallest possible increment) are used. For example, if the partition included the values 1 to 10 in increments of 0.5, the two value test values for the upper boundary would be 10 and 10.5. The lower boundary test values would be 1 and 0.5. The boundaries are defined by the maximum and minimum values in the defined equivalence partition.

For three value boundary testing, the values before, on and over the boundary are used. In the previous example, the upper boundary tests would include 9.5, 10 and 10.5. The lower boundary tests would include 1.5, 1 and 0.5. The decision regarding whether to use two or three boundary values should be based on the risk associated with the item being tested, with the three boundary approach being used for the higher risk items.

Applicability

This technique is applicable at any level of testing and is appropriate when ordered equivalence partitions exist. Ordering is required because of the concept of being on and over the boundary. For example, a range of numbers is an ordered partition. A partition that consists of all rectangular objects is not an ordered partition and does not have boundary values. In addition to number ranges, boundary value analysis can be applied to the following:

- Numeric attributes of non-numeric variables (e.g., length)

- Loops, including those in use cases

- Stored data structures

- Physical objects (including memory)

- Time-determined activities

Limitations/Difficulties

Because the accuracy of this technique depends on the accurate identification of the equivalence partitions, it is subject to the same limitations and difficulties. The Test Analyst should also be aware of the increments in the valid and invalid values to be able to accurately determine the values to be tested. Only ordered partitions can be used for boundary value analysis but this is not limited to a range of valid inputs. For example, when testing for the number of cells supported by a spreadsheet, there is a partition that contains the number of cells up to and including the maximum allowed cells (the boundary) and another partition that begins with one cell over the maximum (over the boundary).

Coverage

Coverage is determined by taking the number of boundary conditions that are tested and dividing that by the number of identified boundary conditions (either using the two value or three value method). This will provide the coverage percentage for the boundary testing.

Types of Defects

Boundary value analysis reliably finds displacement or omission of boundaries, and may find cases of extra boundaries. This technique finds defects regarding the handling of the boundary values, particularly errors with less-than and greater-than logic (i.e., displacement). It can also be used to find non-functional defects, for example tolerance of load limits (e.g., system supports 10,000 concurrent users).

3.2.3 Decision Tables

Decision tables are used to test the interaction between combinations of conditions. Decision tables provide a clear method to verify testing of all pertinent combinations of conditions and to verify that all possible combinations are handled by the software under test. The goal of decision table testing is to ensure that every combination of conditions, relationships and constraints is tested. When trying to test every possible combination, decision tables can become very large. A method of intelligently reducing the number of combinations from all possible to those which are “interesting” is called collapsed decision table testing. When this technique is used, the combinations are reduced to those that will produce differing outputs by removing sets of conditions that are not relevant for the outcome. Redundant tests or tests in which the combination of conditions is not possible are removed. The decision whether to use full decision tables or collapsed decision tables is usually risk-based. [Copeland03]

Applicability

This technique is commonly applied for the integration, system and acceptance test levels. Depending on the code, it may also be applicable for component testing when a component is responsible for a set of decision logic. This technique is particularly useful when the requirements are presented in the form of flow charts or tables of business rules. Decision tables are also a requirements definition technique and some requirements specifications may arrive already in this format. Even when the requirements are not presented in a tabular or flow-charted form, condition combinations are usually found in the narrative. When designing decision tables, it is important to consider the defined condition combinations as well as those that are not expressly defined but will exist. In order to design a valid decision table, the tester must be able to derive all expected outcomes for all condition combinations from the specification or test oracle. Only when all interacting conditions are considered can the decision table be used as a good test design tool.

Limitations/Difficulties

Finding all the interacting conditions can be challenging, particularly when requirements are not well-defined or do not exist. It is not unusual to prepare a set of conditions and determine that the expected result is unknown.

Coverage

Minimum test coverage for a decision table is to have one test case for each column. This assumes that there are no compound conditions and that all possible condition combinations have been recorded in a column. When determining tests from a decision table, it is also important to consider any boundary conditions that should be tested. These boundary conditions may result in an increase in the number of test cases needed to adequately test the software. Boundary value analysis and equivalence partitioning are complementary to the decision table technique.

Types of Defects

Typical defects include incorrect processing based on particular combinations of conditions resulting in unexpected results. During the creation of the decision tables, defects may be found in the specification document. The most common types of defects are omissions (there is no information regarding what should actually happen in a certain situation) and contradictions. Testing may also find issues with condition combinations that are not handled or are not handled well.

3.2.4 Cause-Effect Graphing

Cause-effect graphs may be generated from any source which describes the functional logic (i.e., the “rules”) of a program, such as user stories or flow charts. They can be useful to gain a graphical overview of a program’s logical structure and are typically used as the basis for creating decision tables. Capturing decisions as cause-effect graphs and/or decision tables enables systematic test coverage of the program’s logic to be achieved.

Applicability

Cause-effect graphs apply in the same situations as decision tables and also apply to the same testing levels. In particular, a cause-effect graph shows condition combinations that cause results (causality), condition combinations that exclude results (not), multiple conditions that must be true to cause a result (and) and alternative conditions that can be true to cause a particular result (or). These relationships can be easier to see in a cause-effect graph than in a narrative description.

Limitations/Difficulties

Cause-effect graphing requires additional time and effort to learn compared to other test design techniques. It also requires tool support. Cause-effect graphs have a particular notation that must be understood by the creator and reader of the graph.

Coverage

Each possible cause to effect line must be tested, including the combination conditions, to achieve minimum coverage. Cause-effect graphs include a means to define constraints on the data and constraints on the flow of logic.

Types of Defects

These graphs find the same types of combinatorial defects as are found with decision tables. In addition, the creation of the graphs helps define the required level of detail in the test basis, and so helps improve the detail and quality of the test basis and helps the tester identify missing requirements.

3.2.5 State Transition Testing

State transition testing is used to test the ability of the software to enter into and exit from defined states via valid and invalid transitions. Events cause the software to transition from state to state and to perform actions. Events may be qualified by conditions (sometimes called guard conditions or transition guards) which influence the transition path to be taken. For example, a login event with a valid username/password combination will result in a different transition than a login event with an invalid password.

State transitions are tracked in either a state transition diagram that shows all the valid transitions between states in a graphical format or a state table which shows all potential transitions, both valid and invalid.

Applicability

State transition testing is applicable for any software that has defined states and has events that will cause the transitions between those states (e.g., changing screens). State transition testing can be used at any level of testing. Embedded software, web software, and any type of transactional software are good candidates for this type of testing. Control systems, i.e., traffic light controllers, are also good candidates for this type of testing.

Limitations/Difficulties

Determining the states is often the most difficult part of defining the state table or diagram. When the software has a user interface, the various screens that are displayed for the user are often used to define the states. For embedded software, the states may be dependent upon the states that the hardware will experience.

Besides the states themselves, the basic unit of state transition testing is the individual transition, also known as a 0-switch. Simply testing all transitions will find some kinds of state transition defects, but more may be found by testing sequences of transactions. A sequence of two successive transitions is called a 1-switch; a sequence of three successive transitions is a 2-switch, and so forth. (These switches are sometimes alternatively designated as N-1 switches, where N represents the number of transitions that will be traversed. A single transition, for instance (a 0-switch), would be a 1-1 switch. [Bath08]

Coverage

As with other types of test techniques, there is a hierarchy of levels of test coverage. The minimum acceptable degree of coverage is to have visited every state and traversed every transition. 100% transition coverage (also known as 100% 0-switch coverage or 100% logical branch coverage) will guarantee that every state is visited and every transition is traversed, unless the system design or the state transition model (diagram or table) are defective. Depending on the relationships between states and transitions, it may be necessary to traverse some transitions more than once in order to execute other transitions a single time.

The term “n-switch coverage” relates to the number of transitions covered. For example, achieving 100% 1-switch coverage requires that every valid sequence of two successive transitions has been tested at least once. This testing may stimulate some types of failures that 100% 0-switch coverage would miss.

“Round-trip coverage” applies to situations in which sequences of transitions form loops. 100% round-trip coverage is achieved when all loops from any state back to the same state have been tested. This must be tested for all states that are included in loops.

For any of these approaches, a still higher degree of coverage will attempt to include all invalid transitions. Coverage requirements and covering sets for state transition testing must identify whether invalid transitions are included.

Types of Defects

Typical defects include incorrect processing in the current state that is a result of the processing that occurred in a previous state, incorrect or unsupported transitions, states with no exits and the need for states or transitions that do not exist. During the creation of the state machine model, defects may be found in the specification document. The most common types of defects are omissions (there is no information regarding what should actually happen in a certain situation) and contradictions.

3.2.6 Combinatorial Testing Techniques

Combinatorial testing is used when testing software with several parameters, each one with several values, which gives rise to more combinations than are feasible to test in the time allowed. The parameters must be independent and compatible in the sense that any option for any factor can be combined with any option for any other factor. Classification trees allow for some combinations to be excluded, if certain options are incompatible. This does not assume that the combined factors won’t affect each other; they very well might, but should affect each other in acceptable ways.

Combinatorial testing provides a means to identify a suitable subset of these combinations to achieve a predetermined level of coverage. The number of items to include in the combinations can be selected by the Test Analyst, including single items, pairs, triples or more [Copeland03]. There are a number of tools available to aid the Test Analyst in this task (see www.pairwise.org for samples). These tools either require the parameters and their values to be listed (pairwise testing and orthogonal array testing) or to be represented in a graphical form (classification trees) [Grochtmann94]. Pairwise testing is a method applied to testing pairs of variables in combination. Orthogonal arrays are predefined, mathematically accurate tables that allow the Test Analyst to substitute the items to be tested for the variables in the array, producing a set of combinations that will achieve a level of coverage when tested [Koomen06]. Classification tree tools allow the Test Analyst to define the size of combinations to be tested (i.e., combinations of two values, three values, etc.).

Applicability

The problem of having too many combinations of parameter values manifests in at least two different situations related to testing. Some test cases contain several parameters each with a number of possible values, for instance a screen with several input fields. In this case, combinations of parameter values make up the input data for the test cases. Furthermore, some systems may be configurable in a number of dimensions, resulting in a potentially large configuration space. In both these situations, combinatorial testing can be used to identify a subset of combinations, feasible in size.

For parameters with a large number of values, equivalence class partitioning, or some other selection mechanism may first be applied to each parameter individually to reduce the number of values for each parameter, before combinatorial testing is applied to reduce the set of resulting combinations.

These techniques are usually applied to the integration, system and system integration levels of testing.

Limitations/Difficulties

The major limitation with these techniques is the assumption that the results of a few tests are representative of all tests and that those few tests represent expected usage. If there is an unexpected interaction between certain variables, it may go undetected with this type of testing if that particular combination is not tested. These techniques can be difficult to explain to a non-technical audience as they may not understand the logical reduction of tests.

Identifying the parameters and their respective values is sometimes difficult. Finding a minimal set of combinations to satisfy a certain level of coverage is difficult to do manually. Tools usually are used to find the minimum set of combinations. Some tools support the ability to force some (sub-) combinations to be included in or excluded from the final selection of combinations. This capability may be used by the Test Analyst to emphasize or de-emphasize factors based on domain knowledge or product usage information.

Coverage

There are several levels of coverage. The lowest level of coverage is called 1-wise or singleton coverage. It requires each value of every parameter be present in at least one of the chosen combinations. The next level of coverage is called 2-wise or pairwise coverage. It requires every pair of values of any two parameters be included in at least one combination. This idea can be generalized to n-wise coverage, which requires every sub-combination of values of any set of n parameters be included in the set of selected combinations. The higher the n, the more combinations needed to reach 100% coverage. Minimum coverage with these techniques is to have one test case for every combination produced by the tool.

Types of Defects

The most common type of defects found with this type of testing is defects related to the combined values of several parameters.

3.2.7 Use Case Testing

Use case testing provides transactional, scenario-based tests that should emulate usage of the system. Use cases are defined in terms of interactions between the actors and the system that accomplish some goal. Actors can be users or external systems.

Applicability

Use case testing is usually applied at the system and acceptance testing levels. It may be used for integration testing depending on the level of integration and even component testing depending on the behavior of the component. Use cases are also often the basis for performance testing because they portray realistic usage of the system. The scenarios described in the use cases may be assigned to virtual users to create a realistic load on the system.

Limitations/Difficulties

In order to be valid, the use cases must convey realistic user transactions. This information should come from a user or a user representative. The value of a use case is reduced if the use case does not accurately reflect activities of the real user. An accurate definition of the various alternate paths (flows) is important for the testing coverage to be thorough. Use cases should be taken as a guideline, but not a complete definition of what should be tested as they may not provide a clear definition of the entire set of requirements. It may also be beneficial to create other models, such as flow charts, from the use case narrative to improve the accuracy of the testing and to verify the use case itself.

Coverage

Minimum coverage of a use case is to have one test case for the main (positive) path, and one test case for each alternate path or flow. The alternate paths include exception and failure paths. Alternate paths are sometimes shown as extensions of the main path. Coverage percentage is determined by taking the number of paths tested and dividing that by the total number of main and alternate paths.

Types of Defects

Defects include mishandling of defined scenarios, missed alternate path handling, incorrect processing of the conditions presented and awkward or incorrect error reporting.

3.2.8 User Story Testing

In some Agile methodologies, such as Scrum, requirements are prepared in the form of user stories which describe small functional units that can be designed, developed, tested and demonstrated in a single iteration [Cohn04]. These user stories include a description of the functionality to be implemented, any non-functional criteria, and also include acceptance criteria that must be met for the user story to be considered complete.

Applicability

User stories are used primarily in Agile and similar iterative and incremental environments. They are used for both functional testing and non-functional testing. User stories are used for testing at all levels with the expectation that the developer will demonstrate the functionality implemented for the user story prior to handoff of the code to the team members with the next level of testing tasks (e.g., integration, performance testing).

Limitations/Difficulties

Because stories are little increments of functionality, there may be a requirement to produce drivers and stubs in order to actually test the piece of functionality that is delivered. This usually requires an ability to program and to use tools that will help with the testing such as API testing tools. Creation of the drivers and stubs is usually the responsibility of the developer, although a Technical Test Analyst also may be involved in producing this code and utilizing the API testing tools. If a continuous integration model is used, as is the case in most Agile projects, the need for drivers and stubs is minimized.

Coverage

Minimum coverage of a user story is to verify that each of the specified acceptance criteria has been met.

Types of Defects

Defects are usually functional in that the software fails to provide the specified functionality. Defects are also seen with integration issues of the functionality in the new story with the functionality that already exists. Because stories may be developed independently, performance, interface and error handling issues may be seen. It is important for the Test Analyst to perform both testing of the individual functionality supplied as well as integration testing anytime a new story is released for testing.

3.2.9 Domain Analysis

A domain is a defined set of values. The set may be defined as a range of values of a single variable (a one-dimensional domain, e.g., “men aged over 24 and under 66”), or as ranges of values of interacting variables (a multi-dimensional domain, e.g., “men aged over 24 and under 66 AND with weight over 69 kg and under 90 kg”). Each test case for a multi-dimensional domain must include appropriate values for each variable involved.

Domain analysis of a one-dimensional domain typically uses equivalence partitioning and boundary value analysis. Once the partitions are defined, the Test Analyst selects values from each partition that represent a value that is in the partition (IN), outside the partition (OUT), on the boundary of the partition (ON) and just off the boundary of the partition (OFF). By determining these values, each partition is tested along with its boundary conditions. [Black07]

With multi-dimensional domains the number of test cases generated by these methods rises exponentially with the number of variables involved, whereas an approach based on domain theory leads to a linear growth. Also, because the formal approach incorporates a theory of defects (a fault model), which equivalence partitioning and boundary value analysis lack, its smaller test set will find defects in multi-dimensional domains that the larger, heuristic test set would likely miss. When dealing with multi-dimensional domains, the test model may be constructed as a decision table (or “domain matrix”). Identifying test case values for multi-dimensional domains above three dimensions is likely to require computational support.

Applicability

Domain analysis combines the techniques used for decision tables, equivalence partitioning and boundary value analysis to create a smaller set of tests that still cover the important areas and the likely areas of failure. It is often applied in cases where decision tables would be unwieldy because of the large number of potentially interacting variables. Domain analysis can be done at any level of testing but is most frequently applied at the integration and system testing levels.

Limitations/Difficulties

Doing thorough domain analysis requires a strong understanding of the software in order to identify the various domains and potential interaction between the domains. If a domain is left unidentified, the testing can be significantly lacking, but it is likely that the domain will be detected because the OFF and OUT variables may land in the undetected domain. Domain analysis is a strong technique to use when working with a developer to define the testing areas.

Coverage

Minimum coverage for domain analysis is to have a test for each IN, OUT, ON and OFF value in each domain. Where there is an overlap of the values (for example, the OUT value of one domain is an IN value in another domain), there is no need to duplicate the tests. Because of this, the actual tests needed are often less than four per domain.

Types of Defects

Defects include functional problems within the domain, boundary value handling, variable interaction issues and error handling (particularly for the values that are not in a valid domain).

3.2.10 Combining Techniques

Sometimes techniques are combined to create test cases. For example, the conditions identified in a decision table might be subjected to equivalence partitioning to discover multiple ways in which a condition might be satisfied. Test cases would then cover not only every combination of conditions, but also, for those conditions which were partitioned, additional test cases would be generated to cover the equivalence partitions. When selecting the particular technique to be applied, the Test Analyst should consider the applicability of the technique, the limitations and difficulties, and the goals of the testing in terms of coverage and defects to be detected. There may not be a single “best” technique for a situation. Combined techniques will often provide the most complete coverage assuming there is sufficient time and skill to correctly apply the techniques.

3.3 Defect-Based Techniques

3.3.1 Using Defect-Based Techniques

A defect-based test design technique is one in which the type of defect sought is used as the basis for test design, with tests derived systematically from what is known about the type of defect. Unlike specification-based testing which derives its tests from the specification, defect-based testing derives tests from defect taxonomies (i.e., categorized lists) that may be completely independent from the software being tested. The taxonomies can include lists of defect types, root causes, failure symptoms and other defect-related data. Defect-based testing may also use lists of identified risks and risk scenarios as a basis for targeting testing. This test technique allows the tester to target a specific type of defect or to work systematically through a defect taxonomy of known and common defects of a particular type. The Test Analyst uses the taxonomy data to determine the goal of the testing, which is to find a specific type of defect. From this information, the Test Analyst will create the test cases and test conditions that will cause the defect to manifest itself, if it exists.

Applicability

Defect-based testing can be applied at any testing level but is most commonly applied during system testing. There are standard taxonomies that apply to multiple types of software. This non-product specific type of testing helps to leverage industry standard knowledge to derive the particular tests. By adhering to industry-specific taxonomies, metrics regarding defect occurrence can be tracked across projects and even across organizations.

Limitations/Difficulties

Multiple defect taxonomies exist and may be focused on particular types of testing, such as usability. It is important to pick a taxonomy that is applicable to the software being tested, if any are available. For example, there may not be any taxonomies available for innovative software. Some organizations have compiled their own taxonomies of likely or frequently seen defects. Whatever taxonomy is used, it is important to define the expected coverage prior to starting the testing.

Coverage

The technique provides coverage criteria which are used to determine when all the useful test cases have been identified. As a practical matter, the coverage criteria for defect-based techniques tend to be less systematic than for specification-based techniques in that only general rules for coverage are given and the specific decision about what constitutes the limit of useful coverage is discretionary. As with other techniques, the coverage criteria do not mean that the entire set of tests is complete, but rather that defects being considered no longer suggest any useful tests based on that technique.

Types of Defects

The types of defects discovered usually depend on the taxonomy in use. If a user interface taxonomy is used, the majority of the discovered defects would likely be user interface related, but other defects can be discovered as a byproduct of the specific testing.

3.3.2 Defect Taxonomies

Defect taxonomies are categorized lists of defect types. These lists can be very general and used to serve as high-level guidelines or can be very specific. For example, a taxonomy for user interface defects could contain general items such as functionality, error handling, graphics display and performance. A detailed taxonomy could include a list of all possible user interface objects (particularly for a graphical user interface) and could designate the improper handling of these objects, such as:

- Text field

- Valid data is not accepted

- Invalid data is accepted

- Length of input is not verified

- Special characters are not detected

- User error messages are not informative

- User is not able to correct erroneous data

- Rules are not applied

- Date field

- Valid dates are not accepted

- Invalid dates are not rejected

- Date ranges are not verified

- Precision data is not handled correctly (e.g., hh:mm:ss)

- User is not able to correct erroneous data

- Rules are not applied (e.g., ending date must be greater than starting date)

There are many defect taxonomies available, ranging from formal taxonomies that can be purchased to those designed for specific purposes by various organizations. Internally developed defect taxonomies can also be used to target specific defects commonly found within the organization. When creating a new defect taxonomy or customizing one that is available, it is important to first define the goals or objectives of the taxonomy. For example, the goal might be to identify user interface issues that have been discovered in production systems or to identify issues related to the handling of input fields.

To create a taxonomy:

1. Create a goal and define the desired level of detail

2. Select a given taxonomy to use as a basis

3. Define values and common defects experienced in the organization and/or from practice outside

The more detailed the taxonomy, the more time it will take to develop and maintain it, but it will result in a higher level of reproducibility in the test results. Detailed taxonomies can be redundant, but they allow a test team to divide up the testing without a loss of information or coverage.

Once the appropriate taxonomy has been selected, it can be used for creating test conditions and test cases. A risk-based taxonomy can help the testing focus on a specific risk area. Taxonomies can also be used for non-functional areas such as usability, performance, etc. Taxonomy lists are available in various publications, from IEEE, and on the Internet.

3.4 Experience-Based Techniques

Experience-based tests utilize the skill and intuition of the testers, along with their experience with similar applications or technologies. These tests are effective at finding defects but not as appropriate as other techniques to achieve specific test coverage levels or produce reusable test procedures. In cases where system documentation is poor, testing time is severely restricted or the test team has strong expertise in the system to be tested, experience-based testing may be a good alternative to more structured approaches. Experience-based testing may be inappropriate in systems requiring detailed test documentation, high-levels of repeatability or an ability to precisely assess test coverage.

When using dynamic and heuristic approaches, testers normally use experience-based tests, and testing is more reactive to events than pre-planned testing approaches. In addition execution and evaluation are concurrent tasks. Some structured approaches to experience-based tests are not entirely dynamic, i.e., the tests are not created entirely at the same time as the tester executes the test.

Note that although some ideas on coverage are presented for the techniques discussed here, experience-based techniques do not have formal coverage criteria.

3.4.1 Error Guessing

When using the error guessing technique, the Test Analyst uses experience to guess the potential errors that might have been made when the code was being designed and developed. When the expected errors have been identified, the Test Analyst then determines the best methods to use to uncover the resulting defects. For example, if the Test Analyst expects the software will exhibit failures when an invalid password is entered, tests will be designed to enter a variety of different values in the password field to verify if the error was indeed made and has resulted in a defect that can be seen as a failure when the tests are run.

In addition to being used as a testing technique, error guessing is also useful during risk analysis to identify potential failure modes. [Myers79]

Applicability

Error guessing is done primarily during integration and system testing, but can be used at any level of testing. This technique is often used with other techniques and helps to broaden the scope of the existing test cases. Error guessing can also be used effectively when testing a new release of the software to test for common mistakes and errors before starting more rigorous and scripted testing. Checklists and taxonomies may be helpful in guiding the testing.

Limitations/Difficulties

Coverage is difficult to assess and varies widely with the capability and experience of the Test Analyst. It is best used by an experienced tester who is familiar with the types of defects that are commonly introduced in the type of code being tested. Error guessing is commonly used, but is frequently not documented and so may be less reproducible than other forms of testing.

Coverage

When a taxonomy is used, coverage is determined by the appropriate data faults and types of defects. Without a taxonomy, coverage is limited by the experience and knowledge of the tester and the time available. The yield from this technique will vary based on how well the tester can target problematic areas.

Types of Defects

Typical defects are usually those defined in the particular taxonomy or “guessed” by the Test Analyst, that might not have been found in specification-based testing.

3.4.2 Checklist-Based Testing

When applying the checklist-based testing technique, the experienced Test Analyst uses a high-level, generalized list of items to be noted, checked, or remembered, or a set of rules or criteria against which a product has to be verified. These checklists are built based on a set of standards, experience, and other considerations. A user interface standards checklist employed as the basis for testing an application is an example of a checklist-based test.

Applicability

Checklist-based testing is used most effectively in projects with an experienced test team that is familiar with the software under test or familiar with the area covered by the checklist (e.g., to successfully apply a user interface checklist, the Test Analyst may be familiar with user interface testing but not the specific software under test). Because checklists are high-level and tend to lack the detailed steps commonly found in test cases and test procedures, the knowledge of the tester is used to fill in the gaps. By removing the detailed steps, checklists are low maintenance and can be applied to multiple similar releases. Checklists can be used for any level of testing. Checklists are also used for regression testing and smoke testing.

Limitations/Difficulties

The high-level nature of the checklists can affect the reproducibility of test results. It is possible that several testers will interpret the checklists differently and will follow different approaches to fulfil the checklist items. This may cause different results, even though the same checklist is used. This can result in wider coverage but reproducibility is sometimes sacrificed. Checklists may also result in over-confidence regarding the level of coverage that is achieved since the actual testing depends on the tester’s judgment. Checklists can be derived from more detailed test cases or lists and tend to grow over time. Maintenance is required to ensure that the checklists are covering the important aspects of the software being tested.

Coverage

The coverage is as good as the checklist but, because of the high-level nature of the checklist, the results will vary based on the Test Analyst who executes the checklist.

Types of Defects

Typical defects found with this technique include failures resulting from varying the data, the sequence of steps or the general workflow during testing. Using checklists can help keep the testing fresh as new combinations of data and processes are allowed during testing.

3.4.3 Exploratory Testing

Exploratory testing is characterized by the tester simultaneously learning about the product and its defects, planning the testing work to be done, designing and executing the tests, and reporting the results. The tester dynamically adjusts test goals during execution and prepares only lightweight documentation. [Whittaker09]

Applicability

Good exploratory testing is planned, interactive, and creative. It requires little documentation about the system to be tested and is often used in situations where the documentation is not available or is not adequate for other testing techniques. Exploratory testing is often used to augment other testing and to serve as a basis for the development of additional test cases.

Limitations/Difficulties

Exploratory testing can be difficult to manage and schedule. Coverage can be sporadic and reproducibility is difficult. Using charters to designate the areas to be covered in a testing session and time-boxing to determine the time allowed for the testing is one method used to manage exploratory testing. At the end of a testing session or set of sessions, the test manager may hold a debriefing session to gather the results of the tests and determine the charters for the next sessions. Debriefing sessions are difficult to scale for large testing teams or large projects.

Another difficulty with exploratory sessions is to accurately track them in a test management system. This is sometimes done by creating test cases that are actually exploratory sessions. This allows the time allocated for the exploratory testing and the planned coverage to be tracked with the other testing efforts.

Since reproducibility may be difficult with exploratory testing, this can also cause problems when needing to recall the steps to reproduce a failure. Some organizations use the capture/playback capability of a test automation tool to record the steps taken by an exploratory tester. This provides a complete record of all activities during the exploratory session (or any experience-based testing session). Digging through the details to find the actual cause of the failure can be tedious, but at least there is a record of all the steps that were involved.

Coverage

Charters may be created to specify tasks, objectives, and deliverables. Exploratory sessions are then planned to achieve those objectives. The charter may also identify where to focus the testing effort, what is in and out of scope of the testing session, and what resources should be committed to complete the planned tests. A session may be used to focus on particular defect types and other potentially problematic areas that can be addressed without the formality of scripted testing.

Types of Defects

Typical defects found with exploratory testing are scenario-based issues that were missed during scripted functional testing, issues that fall between functional boundaries, and workflow related issues. Performance and security issues are also sometimes uncovered during exploratory testing.

3.4.4 Applying the Best Technique

Defect- and experience-based techniques require the application of knowledge about defects and other testing experiences to target testing in order to increase defect detection. They range from “quick tests” in which the tester has no formally pre-planned activities to perform, through pre-planned sessions to scripted sessions. They are almost always useful but have particular value in the following circumstances:

- No specifications are available

- There is poor documentation of the system under test

- Insufficient time is allowed to design and create detailed tests

- Testers are experienced in the domain and/or the technology

- Diversity from scripted testing is a goal to maximize test coverage

- Operational failures are to be analyzed

Defect- and experience-based techniques are also useful when used in conjunction with specification-based techniques, as they fill the gaps in test coverage that result from systematic weaknesses in these techniques. As with the specification-based techniques, there is not one perfect technique for all situations. It is important for the Test Analyst to understand the advantages and disadvantages of each technique and to be able to select the best technique or set of techniques for the situation, considering the project type, schedule, access to information, skills of the tester and other factors that can influence the selection.

4. Testing Software Quality Characteristics - 120 mins.

Keywords

accessibility testing, accuracy testing, attractiveness, heuristic evaluation, interoperability testing, learnability, operability, suitability testing, SUMI, understandability, usability testing, WAMMI

Learning Objectives for Testing Software Quality Characteristics

4.2 Quality Characteristics for Business Domain Testing

TA-4.2.1

(K2) Explain by example what testing techniques are appropriate to test accuracy, suitability, interoperability and compliance characteristics.

TA-4.2.2

(K2) For the accuracy, suitability and interoperability characteristics, define the typical defects to be targeted

TA-4.2.3

(K2) For the accuracy, suitability and interoperability characteristics, define when the characteristic should be tested in the lifecycle

TA-4.2.4

(K4) For a given project context, outline the approaches that would be suitable to verify and validate both the implementation of the usability requirements and the fulfillment of the user’s expectations

4.1 Introduction

While the previous chapter described specific techniques available to the tester, this chapter considers the application of those techniques in evaluating the principal characteristics used to describe the quality of software applications or systems.

This syllabus discusses the quality characteristics which may be evaluated by a Test Analyst. The attributes to be evaluated by the Technical Test Analyst are considered in the Advanced Technical Test Analyst syllabus. The description of product quality characteristics provided in ISO 9126 is used as a guide to describing the characteristics. Other standards, such as the ISO 25000 [ISO25000] series (which has superseded ISO 9126) may also be of use. The ISO quality characteristics are divided into product quality characteristics (attributes), each of which may have sub-characteristics (sub-attributes). These are shown in the table below, together with an indication of which characteristics/sub-characteristics are covered by the Test Analyst and Technical Test Analyst syllabi:

The Test Analyst should concentrate on the software quality characteristics of functionality and usability. Accessibility testing should also be conducted by the Test Analyst. Although it is not listed as a sub-characteristic, accessibility is often considered to be part of usability testing. Testing for the other quality characteristics is usually considered to be the responsibility of the Technical Test Analyst. While this allocation of work may vary in different organizations, it is the one that is followed in these ISTQB syllabi.

The sub-characteristic of compliance is shown for each of the quality characteristics. In the case of certain safety-critical or regulated environments, each quality characteristic may have to comply with specific standards and regulations (e.g., functionality compliance may indicate that the functionality comply with a specific standard such as using a particular communication protocol in order to be able to send/receive data from a chip). Because those standards can vary widely depending on the industry, they will not be discussed in depth here. If the Test Analyst is working in an environment that is affected by compliance requirements, it is important to understand those requirements and to ensure that both the testing and the test documentation will fulfill the compliance requirements.

For all of the quality characteristics and sub-characteristics discussed in this section, the typical risks must be recognized so that an appropriate testing strategy can be formed and documented. Quality characteristic testing requires particular attention to lifecycle timing, required tools, software and documentation availability, and technical expertise. Without planning a strategy to deal with each characteristic and its unique testing needs, the tester may not have adequate planning, ramp up and test execution time built into the schedule. Some of this testing, e.g., usability testing, can require allocation of special human resources, extensive planning, dedicated labs, specific tools, specialized testing skills and, in most cases, a significant amount of time. In some cases, usability testing may be performed by a separate group of usability, or user experience, experts.

Quality characteristic and sub-characteristic testing must be integrated into the overall testing schedule, with adequate resources allocated to the effort. Each of these areas has specific needs, targets specific issues and may occur at different times during the software development lifecycle, as discussed in the sections below.

While the Test Analyst may not be responsible for the quality characteristics that require a more technical approach, it is important that the Test Analyst be aware of the other characteristics and understand the overlap areas for testing. For example, a product that fails performance testing will also likely fail in usability testing if it is too slow for the user to use effectively. Similarly, a product with interoperability issues with some components is probably not ready for portability testing as that will tend to obscure the more basic problems when the environment is changed.

4.2 Quality Characteristics for Business Domain Testing

Functional testing is a primary focus for the Test Analyst. Functional testing is focused on “what” the product does. The test basis for functional testing is generally a requirements or specification document, specific domain expertise or implied need. Functional tests vary according to the test level in which they are conducted and can also be influenced by the software development lifecycle. For example, a functional test conducted during integration testing will test the functionality of interfacing modules which implement a single defined function. At the system test level, functional tests include testing the functionality of the application as a whole. For systems of systems, functional testing will focus primarily on end to end testing across the integrated systems. In an Agile environment, functional testing is usually limited to the functionality made available in the particular iteration or sprint although regression testing for an iteration may cover all released functionality.

A wide variety of test techniques are employed during functional test (see Chapter 3). Functional testing may be performed by a dedicated tester, a domain expert, or a developer (usually at the component level).

In addition to the functional testing covered in this section, there are also two quality characteristics that are a part of the Test Analyst’s area of responsibility that are considered to be non-functional (focused on “how” the product delivers the functionality) testing areas. These two non-functional attributes are usability and accessibility.

The following quality characteristics are considered in this section:

- Functional quality sub-characteristics

- Accuracy

- Suitability

- Interoperability

- Non-functional quality characteristics

- Usability

- Accessibility

4.2.1 Accuracy Testing

Functional accuracy involves testing the application’s adherence to the specified or implied requirements and may also include computational accuracy. Accuracy testing employs many of the test techniques explained in Chapter 3 and often uses the specification or a legacy system as the test oracle. Accuracy testing can be conducted at any stage in the lifecycle and is targeted on incorrect handling of data or situations.

4.2.2 Suitability Testing

Suitability testing involves evaluating and validating the appropriateness of a set of functions for its intended specified tasks. This testing can be based on use cases. Suitability testing is usually conducted during system testing, but may also be conducted during the later stages of integration testing. Defects discovered in this testing are indications that the system will not be able to meet the needs of the user in a way that will be considered acceptable.

4.2.3 Interoperability Testing

Interoperability testing tests the degree to which two or more systems or components can exchange information and subsequently use the information that has been exchanged. Testing must cover all the intended target environments (including variations in the hardware, software, middleware, operating system, etc.) to ensure the data exchange will work properly. In reality, this may only be feasible for a relatively small number of environments. In that case interoperability testing may be limited to a selected representative group of environments. Specifying tests for interoperability requires that combinations of the intended target environments are identified, configured and available to the test team. These environments are then tested using a selection of functional test cases which exercise the various data exchange points present in the environment.

Interoperability relates to how different software systems interact with each other. Software with good interoperability characteristics can be integrated with a number of other systems without requiring major changes. The number of changes and the effort required to perform those changes may be used as a measure of interoperability.

Testing for software interoperability may, for example, focus on the following design features:

- Use of industry-wide communications standards, such as XML

- Ability to automatically detect the communications needs of the systems it interacts with and adjust accordingly

Interoperability testing may be particularly significant for organizations developing Commercial Off The Shelf (COTS) software and tools and organizations developing systems of systems.

This type of testing is performed during component integration and system testing focusing on the interaction of the system with its environment. At the system integration level, this type of testing is conducted to determine how well the fully developed system interacts with other systems. Because systems may interoperate on multiple levels, the Test Analyst must understand these interactions and be able to create the conditions that will exercise the various interactions. For example, if two systems will exchange data, the Test Analyst must be able to create the necessary data and the transactions required to perform the data exchange. It is important to remember that all interactions may not be clearly specified in the requirements documents. Instead, many of these interactions will be defined only in the system architecture and design documents. The Test Analyst must be able and prepared to examine those documents to determine the points of information exchange between systems and between the system and its environment to ensure all are tested. Techniques such as decision tables, state transition diagrams, use cases and combinatorial testing are all applicable to interoperability testing. Typical defects found include incorrect data exchange between interacting components.

4.2.4 Usability Testing

It is important to understand why users might have difficulty using the system. To gain this understanding it is first necessary to appreciate that the term “user” may apply to a wide range of different types of persons, ranging from IT experts to children to people with disabilities.

Some national institutions (e.g., the British Royal National Institute for the Blind), recommend that web pages be accessible for disabled, blind, partially sighted, mobility impaired, deaf and cognitively-disabled users. Checking that applications and web sites are usable for the above users may also improve the usability for everyone else. Accessibility is discussed more below.

Usability testing tests the ease by which users can use or learn to use the system to reach a specified goal in a specific context. Usability testing is directed at measuring the following:

- Effectiveness - capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use

- Efficiency - capability of the product to enable users to expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use

- Satisfaction - capability of the software product to satisfy users in a specified context of use

Attributes that may be measured include:

- Understandability - attributes of the software that affect the effort required by the user to recognize the logical concept and its applicability

- Learnability - attributes of software that affect the effort required by the user to learn the application

- Operability - attributes of the software that affect the effort required by the user to conduct tasks effectively and efficiently

- Attractiveness - the capability of the software to be liked by the user

Usability testing is usually conducted in two steps:

- Formative Usability Testing - testing that is conducted iteratively during the design and prototyping stages to help guide (or “form”) the design by identifying usability design defects

- Summative Usability Testing - testing that is conducted after implementation to measure the usability and identify problems with a completed component or system

Usability tester skills should include expertise or knowledge in the following areas:

- Sociology

- Psychology

- Conformance to national standards (including accessibility standards)

- Ergonomics

4.2.4.1 Conducting Usability Tests

Validation of the actual implementation should be done under conditions as close as possible to those under which the system will be used. This may involve setting up a usability lab with video cameras, mock up offices, review panels, users, etc., so that development staff can observe the effect of the actual system on real people. Formal usability testing often requires some amount of preparing the “users” (these could be real users or user representatives) either by providing set scripts or instructions for them to follow. Other free form tests allow the user to experiment with the software so the observers can determine how easy or difficult it is for the user to figure out how to accomplish their tasks.

Many usability tests may be executed by the Test Analyst as part of other tests, for example during functional system test. To achieve a consistent approach to the detection and reporting of usability defects in all stages of the lifecycle, usability guidelines may be helpful. Without usability guidelines, it may be difficult to determine what is “unacceptable” usability. For example, is it unreasonable for a user to have to make 10 mouse clicks to log into an application? Without specific guidelines, the Test Analyst can be in the difficult position of defending defect reports that the developer wants to close because the software works “as designed”. It is very important to have the verifiable usability specifications defined in the requirements as well as to have a set of usability guidelines that are applied to all similar projects. The guidelines should include such items as accessibility of instructions, clarity of prompts, number of clicks to complete an activity, error messaging, processing indicators (some type of indicator for the user that the system is processing and cannot accept further inputs at the time), screen layout guidelines, use of colors and sounds and other factors that affect the user’s experience.

4.2.4.2 Usability Test Specification

Principal techniques for usability testing are:

- Inspecting, evaluating or reviewing

- Dynamically interacting with prototypes

- Verifying and validating the actual implementation

- Conducting surveys and questionnaires

Inspecting, evaluating or reviewing

Inspection or review of the requirements specification and designs from a usability perspective that increase the user’s level of involvement can be cost effective by finding problems early. Heuristic evaluation (systematic inspection of a user interface design for usability) can be used to find the usability problems in the design so that they can be attended to as part of an iterative design process. This involves having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the “heuristics”). Reviews are more effective when the user interface is more visible. For example, sample screen shots are usually easier to understand and interpret than a narrative description of the functionality provided by a particular screen. Visualization is important for an adequate usability review of the documentation.

Dynamically interacting with prototypes

When prototypes are developed, the Test Analyst should work with the prototypes and help the developers evolve the prototype by incorporating user feedback into the design. In this way, prototypes can be refined and the user can get a more realistic view of how the finished product will look and feel.

Verifying and validating the actual implementation

Where the requirements specify usability characteristics for the software (e.g., the number of mouse clicks to accomplish a specific goal), test cases should be created to verify that the software implementation has included these characteristics.

For performing validation of the actual implementation, tests specified for functional system test may be developed as usability test scenarios. These test scenarios measure specific usability characteristics, such as learnability or operability, rather than functional outcomes.

Test scenarios for usability may be developed to specifically test syntax and semantics. Syntax is the structure or grammar of the interface (e.g., what can be entered in an input field) whereas semantics describes the meaning and purpose (e.g., reasonable and meaningful system messages and output provided to the user) of the interface.

Black box techniques (for example those described in Section 3.2), particularly use cases which can be defined in plain text or with UML (Unified Modeling Language), are sometimes employed in usability testing.

Test scenarios for usability testing also need to include user instructions, allocation of time for pre- and post-test interviews for giving instructions and receiving feedback and an agreed protocol for conducting the sessions. This protocol includes a description of how the test will be carried out, timings, note taking and session logging, and the interview and survey methods to be used.

Conducting surveys and questionnaires

Survey and questionnaire techniques may be applied to gather observations and feedback regarding user behavior with the system. Standardized and publicly available surveys such as SUMI (Software Usability Measurement Inventory) and WAMMI (Website Analysis and MeasureMent Inventory) permit benchmarking against a database of previous usability measurements. In addition, since SUMI provides concrete measurements of usability, this can provide a set of completion / acceptance criteria.

4.2.5 Accessibility Testing

It is important to consider the accessibility to software for those with particular needs or restrictions for its use. This includes those with disabilities. Accessibility testing should consider the relevant standards, such as the Web Content Accessibility Guidelines, and legislation, such as Disability Discrimination Acts (UK, Australia) and Section 508 (US). Accessibility, similar to usability, must be considered during the design phases. Testing often occurs during the integration levels and continues through system testing and into the acceptance testing levels. Defects are usually determined when the software fails to meet the designated regulations or standards defined for the software.

5. Reviews - 165 mins.

Keywords

none

Learning Objectives for Reviews

5.1 Introduction

TA-5.1.1

(K2) Explain why review preparation is important for the Test Analyst

5.2 Using Checklists in Reviews

TA-5.2.1

(K4) Analyze a use case or user interface and identify problems according to checklist information provided in the syllabus

TA-5.2.2

(K4) Analyze a requirements specification or user story and identify problems according to checklist information provided in the syllabus

5.1 Introduction

A successful review process requires planning, participation and follow-up. Test Analysts must be active participants in the review process, providing their unique views. They should have formal review training to better understand their respective roles in any review process. All review participants must be committed to the benefits of a well-conducted review. When done properly, reviews can be the single biggest, and most cost-effective, contributor to overall delivered quality.

Regardless of the type of review being conducted, the Test Analyst must allow adequate time to prepare. This includes time to review the work product, time to check cross-referenced documents to verify consistency, and time to determine what might be missing from the work product. Without adequate preparation time, the Test Analyst could be restricted only to editing what is already in the document rather than participating in an efficient review that maximizes the use of the review team’s time and provides the best feedback possible. A good review includes understanding what is written, determining what is missing, and verifying that the described product is consistent with other products that are either already developed or are in development. For example, when reviewing an integration level test plan, the Test Analyst must also consider the items that are being integrated. What are the conditions needed for them to be ready for integration? Are there dependencies that must be documented? Is there data available to test the integration points? A review is not isolated to the work product being reviewed; it must also consider the interaction of that item with the others in the system.

It is easy for the author of a product being reviewed to feel criticized. The Test Analyst should be sure to approach any review comments from the view point of working together with the author to create the best product possible. By using this approach, comments will be worded constructively and will be oriented toward the work product and not the author. For example, if a statement is ambiguous, it is better to say “I do not understand what I should be testing to verify that this requirement has been implemented correctly. Can you help me understand it?” rather than “This requirement is ambiguous and no one will be able to figure it out.” The Test Analyst’s job in a review is to ensure that the information provided in the work product will be sufficient to support the testing effort. If the information is not there, is not clear, or does not provide the necessary level of detail, then this is likely to be a defect that needs to be corrected by the author. By maintaining a positive approach rather than a critical approach, comments will be better received and the meeting will be more productive.

5.2 Using Checklists in Reviews

Checklists are used during reviews to remind the participants to check specific points during the review. Checklists can also help to de-personalize the review, e.g., “This is the same checklist we use for every review, we are not targeting only your work product.” Checklists can be generic and used for all reviews or can focus on specific quality characteristics, areas or types of documents. For example, a generic checklist might verify the general document properties such as having a unique identifier, no TBD references, proper formatting and similar conformance items. A specific checklist for a requirements document might contain checks for the proper use of the terms “shall” and “should”, checks for the testability of each stated requirement and so forth. The format of the requirements may also indicate the type of checklist to be used. A requirements document that is in narrative text format will have different review criteria than one that is based on diagrams.

Checklists may also be oriented toward a programmer/architect skill set or a tester skill set. In the case of the Test Analyst, the tester skill set checklist would be the most appropriate. These checklists might include such items as shown below.

Checklists used for the requirements, use cases and user stories generally have a different focus than those used for the code or architecture. These requirements-oriented checklists could include the following items:

- Testability of each requirement

- Acceptance criteria for each requirement

- Availability of a use case calling structure, if applicable

- Unique identification of each requirement/use case/user story

- Versioning of each requirement/use case/user story

- Traceability for each requirement from business/marketing requirements

- Traceability between requirements and use cases

The above is meant only to serve as an example. It is important to remember that if a requirement is not testable, meaning that it is defined in such a way that the Test Analyst cannot determine how to test it, then there is a defect in that requirement. For example, a requirement that states “The software should be very user friendly” is untestable. How can the Test Analyst determine if the software is user friendly, or even very user friendly? If, instead, the requirement said “The software must conform to the usability standards stated in the usability standards document”, and if the usability standards document really exists, then this is a testable requirement. It is also an overarching requirement because this one requirement applies to every item in the interface. In this case, this one requirement could easily spawn many individual test cases in a non-trivial application. Traceability from this requirement, or perhaps from the usability standards document, to the test cases is also critical because if the referenced usability specification should change, all the test cases will need to be reviewed and updated as needed.

A requirement is also untestable if the tester is unable to determine whether the test passed or failed, or is unable to construct a test that can pass or fail. For example, “System shall be available 100% of the time, 24 hours per day, 7 days per week, 365 (or 366) days a year” is untestable.

A simple checklist for use case reviews may include the following questions:

- Is the main path (scenario) clearly defined?

- Are all alternative paths (scenarios) identified, complete with error handling?

- Are the user interface messages defined?

- Is there only one main path (there should be, otherwise there are multiple use cases)?

- Is each path testable?

A simple checklist for usability for a user interface of an application may include:

- Is each field and its function defined?

- Are all error messages defined?

- Are all user prompts defined and consistent?

- Is the tab order of the fields defined?

- Are there keyboard alternatives to mouse actions?

- Are there “shortcut” key combinations defined for the user (e.g., cut and paste)?

- Are there dependencies between fields (such as a certain date has to be later than another date)?

- Is there a screen layout?

- Does the screen layout match the specified requirements?

- Is there an indicator for the user that appears when the system is processing?

- Does the screen meet the minimum mouse click requirement (if defined)?

- Does the navigation flow logically for the user based on use case information?

- Does the screen meet any requirements for learnability?

- Is there help text available for the user?

- Is there hover text available for the user?

- Will the user consider this to be “attractive” (subjective assessment)?

- Is the use of colors consistent with other applications and with organization standards?

- Are the sound effects used appropriately and are they configurable?

- Does the screen meet localization requirements?

- Can the user determine what to do (understandability) (subjective assessment)?

- Will the user be able to remember what to do (learnability) (subjective assessment)?

In an Agile project, requirements usually take the form of user stories. These stories represent small units of demonstrable functionality. Whereas a use case is a user transaction that traverses multiple areas of functionality, a user story is more isolated and is generally scoped by the time it takes to develop it. A checklist for a user story may include:

- Is the story appropriate for the target iteration/sprint?

- Are the acceptance criteria defined and testable?

- Is the functionality clearly defined?

- Are there any dependencies between this story and others?

- Is the story prioritized?

- Does the story contain one item of functionality?

Of course if the story defines a new interface, then using a generic story checklist (such as the one above) and a detailed user interface checklist would be appropriate.

A checklist can be tailored based on the following:

- Organization (e.g., considering company policies, standards, conventions)

- Project / development effort (e.g., focus, technical standards, risks)

- Object being reviewed (e.g., code reviews might be tailored to specific programming languages)

Good checklists will find problems and will also help to start discussions regarding other items that might not have been specifically referenced in the checklist. Using a combination of checklists is a strong way to ensure a review achieves the highest quality work product. Using standard checklists such as those referenced in the Foundation Level syllabus and developing organizationally specific checklists such as the ones shown above will help the Test Analyst be effective in reviews.

For more information on reviews and inspections see [Gilb93] and [Wiegers03].

6. Defect Management – 120 mins.

Keywords

Defect taxonomy, phase containment, root cause analysis

Learning Objectives for Defect Management

6.2 When Can a Defect be Detected?

TA-6.2.1

(K2) Explain how phase containment can reduce costs

6.3 Defect Report Fields

TA-6.3.1

(K2) Explain the information that may be needed when documenting a non-functional defect

6.4 Defect Classification

TA-6.4.1

(K4) Identify, gather and record classification information for a given defect

6.5 Root Cause Analysis

TA-6.5.1

(K2) Explain the purpose of root cause analysis

6.1 Introduction

Test Analysts evaluate the behavior of the system in terms of business and user needs, e.g., would the user know what to do when faced with this message or behavior. By comparing the actual with the expect result, the Test Analyst determines if the system is behaving correctly. An anomaly (also called an incident) is an unexpected occurrence that requires further investigation. An anomaly may be a failure caused by a defect. An anomaly may or may not result in the generation of a defect report. A defect is an actual problem that should be resolved.

6.2 When Can a Defect be Detected?

A defect can be detected through static testing and the symptoms of the defect, the failure, can be detected through dynamic testing. Each phase of the software development lifecycle should provide methods for detecting and eliminating potential failures. For example, during the development phase, code and design reviews should be used to detect defects. During dynamic testing, test cases are used to detect failures.

The earlier a defect is detected and corrected, the lower the cost of quality for the system as a whole. For example, static testing can find defects before dynamic testing is possible. This is one of the reasons why static testing is a cost effective approach to producing high quality software.

The defect tracking system should allow the Test Analyst to record the phase in the lifecycle in which the defect was introduced and the phase in which it was found. If the two phases are the same, then perfect phase containment has been achieved. This means the defect was introduced and found in the same phase and didn’t “escape” to a later phase. An example of this would be an incorrect requirement that is identified during the requirements review and is corrected there. Not only is this an efficient use of the requirements review, but it also prevents that defect from incurring additional work which would make it more expensive for the organization. If an incorrect requirement “escapes” from the requirements review and is subsequently implemented by the developer, tested by the Test Analyst, and caught by the user during user acceptance testing, all the work done on that requirement was wasted time (not to mention that the user may now have lost confidence in the system).

Phase containment is an effective way to reduce the costs of defects.

6.3 Defect Report Fields

The fields (parameters) supplied for a defect report are intended to provide enough information so the defect report is actionable. An actionable defect report is:

- Complete - all the necessary information is in the report

- Concise - no extraneous information is in the report

- Accurate - the information in the report is correct and clearly states the expected and actual results as well as the proper steps to reproduce

- Objective - the report is a professionally written statement of fact

The information recorded in a defect report should be divided into fields of data. The more well-defined the fields, the easier it is to report individual defects as well as to produce trend reports and other summary reports. When a defined number of options are available for a field, having drop down lists of the available values can decrease the time needed when recording a defect. Drop down lists are only effective when the number of options is limited and the user will not need to scroll through a long list to find the correct option. Different types of defect reports require different information and the defect management tool should be flexible enough to prompt for the correct fields depending on the defect type. Data should be recorded in distinct fields, ideally supported by data validation in order to avoid data entry failures, and to ensure effective reporting.

Defect reports are written for failures discovered during functional and non-functional testing. The information in a defect report should always be oriented toward clearly identifying the scenario in which the problem was detected, including steps and data required to reproduce that scenario, as well as the expected and actual results. Non-functional defect reports may require more details regarding the environment, other performance parameters (e.g., size of the load), sequence of steps and expected results. When documenting a usability failure, it is important to state what the user expected the software to do. For example, if the usability standard is that an operation should be completed in less than four mouse clicks, the defect report should state how many clicks were required versus the stated standard. In cases where a standard is not available and the requirements did not cover the non-functional quality aspects of the software, the tester may use the “reasonable person” test to determine that the usability is unacceptable. In that case, the expectations of that “reasonable person” must be clearly stated in the defect report. Because non-functional requirements are sometimes missing in the requirements documentation, documenting non-functional failures presents more challenges for the tester in documenting the “expected” versus the “actual” behavior.

While the usual goal in writing a defect report is to obtain a fix for the problem, the defect information must also be supplied to support accurate classification, risk analysis, and process improvement.

6.4 Defect Classification

There are multiple levels of classification that a defect report may receive throughout its lifecycle. Proper defect classification is an integral part of proper defect reporting. Classifications are used to group defects, to evaluate the effectiveness of testing, to evaluate the effectiveness of the development lifecycle and to determine interesting trends.

Common classification information for newly identified defects includes:

- Project activity that resulted in the defect being detected – e.g., review, audit, inspection, coding, testing

- Project phase in which the defect was introduced (if known) – e.g., requirements, design, detailed design, code

- Project phase in which the defect was detected – e.g., requirements, design, detailed design, code, code review, unit test, integration test, system test, acceptance test

- Suspected cause of defect – e.g., requirements, design, interface, code, data

- Repeatability – e.g., once, intermittent, reproducible

- Symptom – e.g., crash, hang, user interface error, system error, performance

Once the defect has been investigated, further classification may be possible:

- Root cause - the mistake that was made that resulted in the defect, e.g., process, coding error, user error, test error, configuration issue, data issue, third party software, external software problem, documentation issue

- Source – the work product in which the mistake made, e.g., requirements, design, detailed design, architecture, database design, user documentation, test documentation

- Type – e.g., logic problem, computational problem, timing issue, data handling, enhancement

When the defect is fixed (or has been deferred or has failed confirmation), even more classification information may be available, such as:

- Resolution – e.g., code change, documentation change, deferred, not a problem, duplicate

- Corrective action – e.g., requirements review, code review, unit test, configuration documentation, data preparation, no change made

In addition to these classification categories, defects are also frequently classified based on severity and priority. In addition, depending on the project, it may make sense to classify based on mission safety impact, project schedule impact, project costs, project risk and project quality impact. These classifications may be considered in agreements regarding how quickly a fix will be delivered.

The final area of classification is the final resolution. Defects are often grouped together based on their resolution, e.g., fixed/verified, closed/not a problem, deferred, open/unresolved. This classification usually is used throughout a project as the defects are tracked through their lifecycle.

The classification values used by an organization are often customized. The above are only examples of some of the common values used in industry. It is important that the classification values be used consistently in order to be useful. Too many classification fields will make opening and processing a defect somewhat time consuming, so it is important to weigh the value of the data being gathered against the incremental cost for every defect processed. The ability to customize the classification values gathered by a tool is often an important factor in tool selection.

6.5 Root Cause Analysis

The purpose of root cause analysis is to determine what caused the defect to occur and to provide data to support process changes that will remove root causes that are responsible for a significant portion of the defects. Root cause analysis is usually conducted by the person who investigates and either fixes the problem or determines the problem should not or cannot be fixed. This is usually the developer. Setting a preliminary root cause value is commonly done by the Test Analyst who will make an educated guess regarding what probably caused the problem. When confirming the fix, the Test Analyst will verify the root cause setting entered by the developer. At the point the root cause is determined, it is also common to determine or confirm the phase in which the defect was introduced.

Typical root causes include:

- Unclear requirement

- Missing requirement

- Wrong requirement

- Incorrect design implementation

- Incorrect interface implementation

- Code logic error

- Calculation error

- Hardware error

- Interface error

- Invalid data

This root cause information is aggregated to determine common issues that are resulting in the creation of defects. For example, if a large number of defects is caused by unclear requirements, it would make sense to apply more effort to conducting effective requirements reviews. Similarly if interface implementation is an issue across development groups, joint design sessions might be needed.

Using root cause information for process improvement helps an organization to monitor the benefits of effective process changes and to quantify the costs of the defects that can be attributed to a particular root cause. This can help provide funding for process changes that might require purchasing additional tools and equipment as well as changing schedule timing. The ISTQB Expert Level syllabus “Improving the Test Process” [ISTQB_EL_ITP] considers root cause analysis in more detail.

7. Test Tools - 45 mins.

Keywords

keyword-driven testing, test data preparation tool, test design tool, test execution tool

Learning Objectives for Test Tools

7.2 Test Tools and Automation

TA-7.2.1

(K2) Explain the benefits of using test data preparation tools, test design tools and test execution tools

TA-7.2.2

(K2) Explain the Test Analyst’s role in keyword-driven automation

TA-7.2.3

(K2) Explain the steps for troubleshooting an automated test execution failure

7.1 Introduction

Test tools can greatly improve the efficiency and accuracy of the test effort, but only if the proper tools are implemented in the proper way. Test tools have to be managed as another aspect of a well-run test organization. The sophistication and applicability of test tools vary widely and the tool market is constantly changing. Tools are usually available from commercial tool vendors as well as from various freeware or shareware tool sites.

7.2 Test Tools and Automation

Much of a Test Analyst’s job requires the effective use of tools. Knowing which tools to use, and when, can increase the Test Analyst’s efficiency and can help to provide better testing coverage in the time allowed.

7.2.1 Test Design Tools

Test design tools are used to help create test cases and test data to be applied for testing. These tools may work from specific requirements document formats, models (e.g., UML), or inputs provided by the Test Analyst. Test design tools are often designed and built to work with particular formats and particular products such as specific requirements management tools.

Test design tools can provide information for the Test Analyst to use when determining the types of tests that are needed to obtain the targeted level of test coverage, confidence in the system, or product risk mitigation actions. For example, classification tree tools generate (and display) the set of combinations that is needed to reach full coverage based on a selected coverage criterion. This information then can be used by the Test Analyst to determine the test cases that must be executed.

7.2.2 Test Data Preparation Tools

Test data preparation tools provide several benefits. Some test data preparation tools are able to analyze a document such as a requirements document or even the source code to determine the data required during testing to achieve a level of coverage. Other test data preparation tools can take a data set from a production system and “scrub” or “anonymize” it to remove any personal information while still maintaining the internal integrity of that data. The scrubbed data can then be used for testing without the risk of a security leak or misuse of personal information. This is particularly important where large volumes of realistic data are required. Other data generation tools can be used to generate test data from given sets of input parameters (i.e., for use in random testing). Some of these will analyze the database structure to determine what inputs will be required from the Test Analyst.

7.2.3 Automated Test Execution Tools

Test execution tools are mostly used by Test Analysts at all levels of testing to run tests and check the outcome of the tests. The objective of using a test execution tool is typically one or more of the following:

- To reduce costs (in terms of effort and/or time)

- To run more tests

- To run the same test in many environments

- To make test execution more repeatable

- To run tests that would be impossible to run manually (i.e., large data validation tests)

These objectives often overlap into the main objectives of increasing coverage while reducing costs.

7.2.3.1 Applicability

The return on investment for test execution tools is usually highest when automating regression tests because of the low-level of maintenance expected and the repeated execution of the tests. Automating smoke tests can also be an effective use of automation due to the frequent use of the tests, the need for a quick result and, although the maintenance cost may be higher, the ability to have an automated way to evaluate a new build in a continuous integration environment.

Test execution tools are commonly used during the system and integration testing levels. Some tools, particularly API testing tools, may be used at the component testing level as well. Leveraging the tools where they are most applicable will help to improve the return in investment.

7.2.3.2 Test Automation Tool Basics

Test execution tools work by executing a set of instructions written in a programming language, often called a scripting language. The instructions to the tool are at a very detailed level that specifies inputs, order of the input, specific values used for the inputs and the expected outputs. This can make the detailed scripts susceptible to changes in the software under test (SUT), particularly when the tool is interacting with the graphical user interface (GUI).

Most test execution tools include a comparator which provides the ability to compare an actual result to a stored expected result.

7.2.3.3 Test Automation Implementation

The tendency in test execution automation (as in programming) is to move from detailed low-level instructions to more high-level languages, utilizing libraries, macros and sub-programs. Design techniques such as keyword-driven and action word-driven capture a series of instructions and reference those with a particular “keyword” or “action word”. This allows the Test Analyst to write test cases in human language while ignoring the underlying programming language and lower level functions. Using this modular writing technique allows easier maintainability during changes to the functionality and interface of the software under test. [Bath08] The use of keywords in automated scripts is discussed more below.

Models can be used to guide the creation of the keywords or action words. By looking at the business process models which are often included in the requirements documents, the Test Analyst can determine the key business processes that must be tested. The steps for these processes can then be determined, including the decision points that may occur during the processes. The decision points can become action words that the test automation can obtain and use from the keyword or action word spreadsheets. Business process modeling is a method of documenting the business processes and can be used to identify these key processes and decision points. The modeling can be done manually or by using tools that will act off inputs based on business rules and process descriptions.

7.2.3.4 Improving the Success of the Automation Effort

When determining which tests to automate, each candidate test case or candidate test suite must be assessed to see if it merits automation. Many unsuccessful automation projects are based on automating the readily available manual test cases without checking the actual benefit from the automation. It may be optimal for a given set of test cases (a suite) to contain manual, semi-automated and fully automated tests.

The following aspects should be considered when implementing a test execution automation project:

Possible benefits:

- Automated test execution time will become more predictable

- Regression testing and defect validation using automated tests will be faster and more reliable late in the project

- The status and technical growth of the tester or test team may be enhanced by the use of automated tools

- Automation can be particularly helpful with iterative and incremental development lifecycles to provide better regression testing for each build or iteration

- Coverage of certain test types may only be possible with automated tools (e.g., large data validation efforts)

- Test execution automation can be more cost-effective than manual testing for large data input, conversion and comparison testing efforts by providing fast and consistent input and verification.

Possible risks:

- Incomplete, ineffective or incorrect manual testing may be automated “as is”

- The testware may be difficult to maintain, requiring multiple changes when the software under test is changed

- Direct tester involvement in test execution may be reduced, resulting in less defect detection

- The test team may have insufficient skills to use the automated tools effectively

- Irrelevant tests that do not contribute to the overall test coverage may be automated because they exist and are stable

- Tests may become unproductive as the software stabilizes (pesticide paradox)

During deployment of a test execution automation tool, it is not always wise to automate manual test cases as is, but to redefine the test cases for better automation use. This includes formatting the test cases, considering re-use patterns, expanding input by using variables instead of using hard-coded values and utilizing the full benefits of the test tool. Test execution tools usually have the ability to traverse multiple tests, group tests, repeat tests and change order of execution, all while providing analysis and reporting facilities.

For many test execution automation tools, programming skills are necessary to create efficient and effective tests (scripts) and test suites. It is common that large automated test suites are very difficult to update and manage if not designed with care. Appropriate training in test tools, programming and design techniques is valuable to make sure the full benefits of the tools are leveraged.

During test planning, it is important to allow time to periodically execute the automated test cases manually in order to retain the knowledge of how the test works and to verify correct operation as well as to review input data validity and coverage.

7.2.3.5 Keyword-Driven Automation

Keywords (sometimes referred to as action words) are mostly, but not exclusively, used to represent high-level business interactions with a system (e.g., “cancel order”). Each keyword is typically used to represent a number of detailed interactions between an actor and the system under test. Sequences of keywords (including relevant test data) are used to specify test cases.[Buwalda01]

In test automation a keyword is implemented as one or more executable test scripts. Tools read test cases written as a sequence of keywords that call the appropriate test scripts which implement the keyword functionality. The scripts are implemented in a highly modular manner to enable easy mapping to specific keywords. Programming skills are needed to implement these modular scripts.

The primary advantages of keyword-driven test automation are:

- Keywords that relate to a particular application or business domain can be defined by domain experts. This can make the task of test case specification more efficient.

- A person with primarily domain expertise can benefit from automatic test case execution (once the keywords have been implemented as scripts) without having to understand the underlying automation code.

- Test cases written using keywords are easier to maintain because they are less likely to need modification if details in the software under test change.

- Test case specifications are independent of their implementation. The keywords can be implemented using a variety of scripting languages and tools.

The automation scripts (the actual automation code) that use the keyword/action word information are usually written by developers or Technical Test Analysts while the Test Analysts usually create and maintain the keyword/action word data. While keyword-driven automation is usually run during the system testing phase, code development may start as early as the integration phases. In an iterative environment, the test automation development is a continuous process.

Once the input keywords and data are created, the Test Analysts usually assume responsibility to execute the keyword-driven test cases and to analyze any failures that may occur. When an anomaly is detected, the Test Analyst must investigate the cause of failure to determine if the problem is with the keywords, the input data, the automation script itself or with the application being tested. Usually the first step in troubleshooting is to execute the same test with the same data manually to see if the failure is in the application itself. If this does not show a failure, the Test Analyst should review the sequence of tests that led up to the failure to determine if the problem occurred in a previous step (perhaps by producing incorrect data), but the problem did not surface until later in the processing. If the Test Analyst is unable to determine the cause of failure, the trouble shooting information should be turned over to the Technical Test Analyst or developer for further analysis.

7.2.3.6 Causes for Failures of the Automation Effort

Test execution automation projects often fail to achieve their goals. These failures may be due to insufficient flexibility in the usage of the testing tool, insufficient programming skills in the testing team or an unrealistic expectation of the problems that can be solved with test execution automation. It is important to note that any test execution automation takes management, effort, skills and attention, just as any software development project. Time has to be devoted to creating a sustainable architecture, following proper design practices, providing configuration management and following good coding practices. The automated test scripts have to be tested because they are likely to contain defects. The scripts may need to be tuned for performance. Tool usability must be considered, not just for the developer but also for the people who will be using the tool to execute scripts. It may be necessary to design an interface between the tool and the user that will provide access to the test cases in a way that is organized logically for the tester but still provides the accessibility needed by the tool.

8. References

8.1 Standards

- [ISO25000] ISO/IEC 25000:2005, Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE)
Chapters 1 and 4

- [ISO9126] ISO/IEC 9126-1:2001, Software Engineering - Software Product Quality,
Chapters 1 and 4

- [RTCA DO-178B/ED-12B]: Software Considerations in Airborne Systems and Equipment Certification, RTCA/EUROCAE ED12B.1992.
Chapter 1

8.2 ISTQB Documents

- [ISTQB_AL_OVIEW] ISTQB Advanced Level Overview, Version 1.0

- [ISTQB_ALTM_SYL] ISTQB Advanced Level Test Manager Syllabus, Version 1.0

- [ISTQB_ALTTA_SYL] ISTQB Advanced Level Technical Test Analyst Syllabus, Version 1.0

- [ISTQB_FL_SYL] ISTQB Foundation Level Syllabus, Version 2011

- [ISTQB_GLOSSARY] Standard glossary of terms used in Software Testing, Version 2.2, 2012

8.3 Books

[Bath08] Graham Bath, Judy McKay, “The Software Test Engineer’s Handbook”, Rocky Nook, 2008, ISBN 978-1-933952-24-6

[Beizer95] Boris Beizer, “Black-box Testing”, John Wiley & Sons, 1995, ISBN 0-471-12094-4

[Black02]: Rex Black, “Managing the Testing Process (2nd edition)”, John Wiley & Sons: New York, 2002, ISBN 0-471-22398-0

[Black07]: Rex Black, “Pragmatic Software Testing”, John Wiley and Sons, 2007, ISBN 978-0-470-12790-2

[Buwalda01]: Hans Buwalda, “Integrated Test Design and Automation”, Addison-Wesley Longman, 2001, ISBN 0-201-73725-6

[Cohn04]: Mike Cohn, “User Stories Applied: For Agile Software Development”, Addison-Wesley Professional, 2004, ISBN 0-321-20568-5

[Copeland03]: Lee Copeland, “A Practitioner’s Guide to Software Test Design”, Artech House, 2003, ISBN 1-58053-791-X

[Craig02]: Rick David Craig, Stefan P. Jaskiel, “Systematic Software Testing”, Artech House, 2002, ISBN 1-580-53508-9

[Gerrard02]: Paul Gerrard, Neil Thompson, “Risk-based e-business Testing”, Artech House, 2002, ISBN 1-580-53314-0

[Gilb93]: Tom Gilb, Graham Dorothy, “Software Inspection”, Addison-Wesley, 1993, ISBN 0-201-63181-4

[Graham07]: Dorothy Graham, Erik van Veenendaal, Isabel Evans, Rex Black “Foundations of Software Testing”, Thomson Learning, 2007, ISBN 978-1-84480-355-2

[Grochmann94]: M. Grochmann (1994), Test case design using Classification Trees, in: conference proceedings STAR 1994

[Koomen06]: Tim Koomen, Leo van der Aalst, Bart Broekman, Michiel Vroon “TMap NEXT, for result driven testing”, UTN Publishers, 2006, ISBN 90-72194-80-2

[Myers79]: Glenford J. Myers, “The Art of Software Testing”, John Wiley & Sons, 1979, ISBN 0-471-46912-2

[Splaine01]: Steven Splaine, Stefan P. Jaskiel, “The Web-Testing Handbook”, STQE Publishing, 2001, ISBN 0-970-43630-0

[vanVeenendaal12]: Erik van Veenendaal, “Practical risk-based testing – The PRISMA approach”, UTN Publishers, The Netherlands, ISBN 9789490986070

[Wiegers03]: Karl Wiegers, “Software Requirements 2”, Microsoft Press, 2003, ISBN 0-735-61879-8

[Whittaker03]: James Whittaker, “How to Break Software”, Addison-Wesley, 2003, ISBN 0-201-79619-8

[Whittaker09]: James Whittaker, “Exploratory Software Testing”, Addison-Wesley, 2009, ISBN 0-321-63641-4

8.4 Other References

The following references point to information available on the Internet and elsewhere. Even though these references were checked at the time of publication of this Advanced Level syllabus, the ISTQB cannot be held responsible if the references are not available anymore.

- Chapter 3
- Czerwonka, Jacek: www.pairwise.org
- Bug Taxonomy: www.testingeducation.org/a/bsct2.pdf
- Sample Bug Taxonomy based on Boris Beizer’s work: inet.uni2.dk/~vinter/bugtaxst.doc
- Good overview of various taxonomies: testingeducation.org/a/bugtax.pdf
- Heuristic Risk-Based Testing By James Bach
- From “Exploratory & Risk-Based Testing (2004) www.testingeducation.org”
- Exploring Exploratory Testing, Cem Kaner and Andy Tikam, 2003
- Pettichord, Bret, “An Exploratory Testing Workshop Report”,
www.testingcraft.com/exploratorypettichord

- Chapter 4
- www.testingstandards.co.uk

9. Index

0-switch, 31

accessibility, 41

accessibility testing, 47

accuracy, 41

accuracy testing, 43

action word-driven, 58

action words, 59

activities, 10

Agile, 10, 14, 15, 22, 33, 34, 43, 51, 61

anonymize, 57

applying the best technique, 39

attractiveness, 41

automation benefits, 59

automation risks, 59

boundary value analysis, 26, 28

breadth-first, 24

business process modeling, 58

BVA, 26

cause-effect graphing, 26, 30

centralized testing, 22

checklist-based testing, 26, 38

checklists in reviews, 49

classification tree, 26, 32

combinatorial testing, 26, 32, 44

combinatorial testing techniques, 31

combining techniques, 35

compliance, 42

concrete test cases, 8, 13

decision table, 26, 29

defect

detection, 53

fields, 53

defect classification, 54

defect taxonomy, 26, 35, 36, 52

defect tracking, 21

defect-based, 35

defect-based technique, 26, 35

depth-first, 24

distributed testing, 22

distributed, outsourced & insourced testing, 22

domain analysis, 26, 34

embedded iterative, 10

equivalence partitioning, 26, 27

error guessing, 26, 37

evaluating exit criteria and reporting, 18

evaluation, 46

exit criteria, 8

experience-based technique, 26

experience-based techniques, 16, 26, 37, 39, 40

exploratory testing, 26, 38

false-negative result, 17

false-positive result, 17

functional quality characteristics, 43

functional testing, 43

heuristic, 41, 46

high-level test case, 8

incident, 17

insourced testing, 22

inspection, 46

interoperability, 41

interoperability testing, 44

ISO 25000, 15, 42

ISO 9126, 15, 42

keyword-driven, 56, 58

keyword-driven automation, 59

learnability, 41

logical test case, 8

logical test cases, 13

low-level test case, 8

metrics, 12

N-1 switches, 31

non-functional quality characteristics, 43

n-switch coverage, 31

operability, 41

orthogonal array, 26, 32

orthogonal array testing, 26, 32

outsourced testing, 22

pairwise, 32

pairwise testing, 26

pesticide paradox, 16

phase containment, 52, 53

product risk, 20

product risks, 12

prototypes, 46

quality characteristics, 42

quality sub-characteristics, 42

questionnaires, 46

regression test set, 19

requirements-based testing, 26

requirements-oriented checklists, 50

retrospective meetings, 19

review, 46, 49

risk analysis, 20

risk assessment, 23

risk identification, 20, 23

risk level, 20

risk management, 20

risk mitigation, 20, 21, 24

risk-based testing, 20, 22

risk-based testing strategy, 15

root cause, 12, 54, 55

root cause analysis, 52, 55

SDLC

Agile methods, 10

iterative, 10

software lifecycle, 9

specification-based technique, 26

specification-based techniques, 27

standards

DO-178B, 16

ED-12B, 16

UML, 46

state transition testing, 26, 30

suitability, 41

suitability testing, 44

SUMI, 41, 46

surveys, 46

test analysis, 12

test basis, 14

test case, 13

test charter, 26

test closure activities, 18

test condition, 12

test control, 8

test design, 8, 12

test environment, 16

test estimates, 11

test execution, 8, 16

test implementation, 8, 15

test logging, 17

test monitoring, 20

test monitoring and control, 11

test oracle, 14

test planning, 8, 11

test plans, 11

test progress monitoring & control, 21

test strategy, 12, 13, 20

test suites, 15

test techniques, 26

testing software quality characteristics, 41

tools

test design tool, 29, 56, 57

test data preparation tool, 56, 57

test execution tool, 56, 57

traceability, 12

understandability, 41

unscripted testing, 16

untestable, 50

usability, 41

usability test specification, 46

usability testing, 44

use case testing, 26, 33

user stories, 14, 15, 30, 33, 50, 51

user story testing, 26, 33

validation, 46

WAMMI, 41, 46

Standard glossary of terms used in Software Testing

Version 2.2 (dd. October 19th, 2012)

Produced by the ‘Glossary Working Party’

International Software Testing Qualifications Board

Editor : Erik van Veenendaal (The Netherlands)

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © 2012, International Software Testing Qualifications Board (hereinafter called ISTQB®).

Contributors

Rex Black (USA)

Josie Crawford (Australia)

Enst Düring (Norway)

Sigrid Eldh (Sweden)

Isabel Evans (UK)

Simon Frankish (UK)

David Fuller (Australia)

Annu George (India)

Dorothy Graham (UK)

Mats Grindal (Sweden)

Matthias Hamburg (Germany)

Julian Harty (UK)

David Hayman (UK)

Bernard Homes (France)

Ian Howles (UK)

Juha Itkonen (Finland)

Paul Jorgensen (USA)

Vipul Kocher (India)

Gerard Kruijff (The Netherlands)

Fernando Lamas de Oliveira (Portugal)

Tilo Linz (Germany)

Gustavo Marquez Sosa (Spain)

Judy McKay (US)

Don Mills (UK)

Peter Morgan (UK)

Thomas Müller (Switzerland)

Avi Ofer (Israel)

Ana Paiva (Portugal)

Dale Perry (USA)

Ander Pettersson (Sweden)

Horst Pohlmann (Germany)

Juha Pomppu (Finland)

Meile Posthuma (The Netherlands)

Erkki Pöyhönen (Finland)

Maaret Pyhäjärvi (Finland)

Andy Redwood (UK)

Stuart Reid (UK)

Piet de Roo (The Netherlands)

Steve Sampson (UK)

Shane Saunders (UK)

Hans Schaefer (Norway)

Jurriën Seubers (The Netherlands)

Dave Sherratt (UK)

Mike Smith (UK)

Andreas Spillner (Germany)

Lucjan Stapp (Poland)

Richard Taylor (UK)

Geoff Thompson (UK)

Stephanie Ulrich (Germany)

Matti Vuori (Finland)

Gearrel Welvaart (The Netherlands)

Paul Weymouth (UK)

Pete Williams (UK)

Change History

Version 2.2 d.d. MM-DD-2012
This new version has been developed to support the Advanced level syllabi (Version 2012) and the Expert level syllabus Test Management (Version 2011). In addition a new of change request have been implemented in version 2.2 of the ISTQB Glossary.

New terms added:

Terms changed;

- actor

- action word driven testing

- analytical testing

- accuracy

- anti-pattern

- agile testing

- API (Application Programming Interface) testing

- big-bang testing

- atomic condition

- Capability Maturity Model (CMM)

- attack-based testing

- Capability Maturity Model Integration (CMMI)

- combinatorial testing

- classification tree method

- CMMI

- Commercial Off-The-Shelft software

- confidence interval

- condition

- consultative testing

- condition determination coverage

- control chart

- condition determination testing

- control flow testing

- critical success factor

- convergence metric

- Critical Testing Processes

- custom tool

- cyclomatic complexity

- data quality

- daily build

- defect category

- data-driven testing

- defect management committee

- data flow testing

- defect triage committee

- dd-path

- defect type

- defect-based technique

- domain analysis

- defect-based test design technique

- effectiveness

- definition-use pair

- embedded iterative development model

- efficiency

- experience-based testing

- elementary comparison testing

- factory acceptance testing

- extreme programming

- failover testing

- fault seeding

- fault injection

- heuristic evaluation

- feature-driven development

- keyword-driven testing

- hardware-software integration testing

- lead assessor

- insourced testing

- load testing tool

- man in the middle attack

- maturity

- methodical testing

- mind map

- model-based testing

- modified condition decision coverage

- Myers-Briggs Type Indicator (MBTI)

- modified condition decision testing

- neighborhood integration testing

- modified multiple condition coverage

- open source tool

- multiple condition coverage

- operational profiling

- performance profiling

- outsourced testing

- resumption criteria

- pairwise integration testing

- root cause

- phase containment

- software quality

- planning poker

- SPI

- predicate

- static analysis

- process-compliant testing

- static testing

- quality control

- stress testing

- quality risk

- technical review

- RACI matrix

- test case specification

- reactive testing

- test design specification

- regression-averse testing

- test execution schedule

- resumption requirements

- Test Process Improvement (TPI)

- review plan

- Test Maturity Model (TMM)

- risk assessment

- Test Maturity Model integration

- risk impact

- test procedure specification

- risk likelihood

- testable requirement

- Shewhart chart

- thread testing

- short-circuiting

- Total Quality Management

- S.M.A.R.T. goal methodology

- software integrity level

- standard-compliant testing

- structure-based technique

- SUMI

- test architect

- test data management

- test director

- test mission

- three point estimation

- TMMi

- TPI Next

- user story

- user story testing

- WAMMI

- Website Analysis and MeasureMent Inventory (WAMMI)

Table of Contents

Foreword

1. Introduction

2. Scope

3. Glossary structure

4. Trademarks

5. Definitions

A

B

C

D

E

F

G

H

I

K

L

M

N

O

P

Q

R

S

T

U

V

W

Annex A References

Foreword

In compiling this glossary the working party has sought the views and comments of as broad a spectrum of opinion as possible in industry, commerce and government bodies and organizations, with the aim of producing an international testing standard which would gain acceptance in as wide a field as possible. Total agreement will rarely, if ever, be achieved in compiling a document of this nature. Contributions to this glossary have been received from the testing communities in Australia, Belgium, Finland, France, Germany, India, Israel, The Netherlands, Norway, Portugal, Spain, Sweden, Switzerland, United Kingdom, and USA.

Many software testers have used BS 7925-1, the British Standard Glossary of Software Testing Terms, since its original publication in 1998. The standard was initially developed with a bias towards component testing, but, since its publication, many comments and proposals for new definitions have been submitted to both improve and expand the standard to cover a wider range of software testing. The ISTQB testing glossary has incorporated many of these suggested updates. It is used as a reference document for the International Software Testing Qualification Board (ISTQB) software testing qualification scheme.

The ISTQB Glossary has two main objectives:

- Support the understanding of ISTQB syllabi by defining the terms used in the various syllabi

- Support communication within the international testing community and with its stakeholders by providing a standard testing vocabulary.

ISTQB National or Regional Boards can use the ISTQB Glossary, to translate into their local language. These boards may adapt the ISTQB Glossary to their particular language needs.

1. Introduction

Much time and effort is wasted both within and between industry, commerce, government and professional and academic institutions when ambiguities arise as a result of the inability to differentiate adequately between such terms as ‘statement coverage’ and ‘decision coverage’; ‘test suite’, ‘test specification’ and ‘test plan’ and similar terms which form an interface between various sectors of society. Moreover, the professional or technical use of these terms is often at variance, with different meanings attributed to them.

2. Scope

This document presents concepts, terms and definitions designed to aid communication in (software) testing and related disciplines.

3. Glossary structure

Arrangement

The glossary has been arranged in a single section of definitions ordered alphabetically. Some terms are preferred to other synonymous ones, in which case, the definition of the preferred term appears, with the synonymous ones referring to that. For example structural testing refers to white box testing. For synonyms, the “See” indicator is used

“See also” cross-references are also used. They assist the user to quickly navigate to the right index term. “See also” cross-references are constructed for relationships such as broader term to a narrower term, and overlapping meaning between two terms.

Keywords

The ISTQB Glossary contains many terms for different reasons. Some are provided to “just” support the reader of an ISTQB syllabus in understanding the text. Some are there because the term was used in a previous version of a syllabus and the principle of backwards compatibility is being applied. However, probably the most important terms are the (examinable) keywords that are explicitly identified by the various ISTQB syllabi. An important user group of these keywords are the (test)professional who are preparing for an ISTQB exam. To support them, the keywords that they need to understand for a particular exam are indicated in this glossary. Note that the principle of inheritance is applicable, e.g., at an ISTQB Advanced exam one still needs to understand all ISTQB Foundation keywords. The keywords are indicated in the following manner:

F:

Keyword ISTQB Foundation syllabus

ATM:

Keyword ISTQB Advanced – Test Management syllabus

ATA:

Keyword ISTQB Advanced – Test Analyst syllabus

ATT:

Keyword ISTQB Advanced – Technical Test Analyst syllabus

EITP:

Keyword ISTQB Expert – Improving the Testing Process syllabus

ETM:

Keyword ISTQB Expert – Test Management syllabus.

Note that if a keyword is identified in a syllabus, but is not the preferred term according the glossary both the keyword and the term it refers to (using the “See” indicator) are labeled with the appropriate syllabus indicator.

References

In this glossary references, are used in two ways.

- Square brackets without the addition of “after”, e.g., [ISO 9126], mean that the exact text of the reference is used.

- In case a definition from a reference has been adapted to the context of the ISTQB Glossary by minor changes, the addition “after” is used, e.g., [After ISO 9126].

4. Trademarks

In this document the following trademarks are used:

- CMMI and IDEAL are registered trademarks of Carnegie Mellon University

- EFQM is a registered trademark of the EFQM Foundation

- Rational Unified Process is a registered trademark of Rational Software Corporation

- STEP is a registered trademark of Software Quality Engineering

- TMap, TPA and TPI Next are registered trademarks of Sogeti Nederland BV

- TMMi is a registered trademark of the TMMi Foundation

5. Definitions

A

abstract test case: See high level test case.

acceptance: See acceptance testing.

acceptance criteria: The exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. [IEEE 610]

acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610]

ATA

accessibility testing: Testing to determine the ease by which users with disabilities can use a component or system. [Gerrard]

accuracy: The capability of the software product to provide the right or agreed results or effects with the needed degree of precision. [ISO 9126] See also functionality.

ATA

accuracy testing: The process of testing to determine the accuracy of a software product

EITP

acting (IDEAL): The phase within the IDEAL model where the improvements are developed, put into practice, and deployed across the organization. The acting phase consists of the activities: create solution, pilot/test solution, refine solution and implement solution. See also IDEAL.

action word driven testing: See keyword-driven testing

actor: User or any other person or system that interacts with the system under test in a specific way.

actual outcome: See actual result.

actual result: The behavior produced/observed when a component or system is tested.

ad hoc review: See informal review.

ad hoc testing: Testing carried out informally; no formal test preparation takes place, no recognized test design technique is used, there are no expectations for results and arbitrariness guides the test execution activity.

ATT

adaptability: The capability of the software product to be adapted for different specified environments without applying actions or means other than those provided for this purpose for the software considered. [ISO 9126] See also portability.

agile manifesto: A statement on the values that underpin agile software development. The values are:

- individuals and interactions over processes and tools

- working software over comprehensive documentation

- customer collaboration over contract negotiation

- responding to change over following a plan.

EITP

agile software development: A group of software development methodologies based on iterative incremental development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams.

EITP

agile testing: Testing practice for a project using agile software development methodologies, incorporating techniques and methods, such as extreme programming (XP), treating development as the customer of testing and emphasizing the test-first design paradigm. See also test driven development.

algorithm test: [TMap] See branch testing.

F

alpha testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers’ site, but outside the development organization. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing.

ETM

analytical testing: Testing based on a systematic analysis of e.g., product risks or requirements.

ATT

analyzability: The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified. [ISO 9126] See also maintainability.

analyzer: See static analyzer.

ATM

anomaly: Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc. or from someone’s perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation. [IEEE 1044] See also bug, defect, deviation, error, fault, failure, incident, problem.

ATT

anti-pattern: Repeated action, process, structure or reusable solution that initially appears to be beneficial and is commonly used but is ineffective and/or counterproductive in practice.

API (Application Programming Interface) testing: Testing the code which enables communication between different processes, programs and/or systems. API testing often involves negative testing, e.g., to validate the robustness of error handling. See also interface testing.

arc testing: See branch testing.

EITP

assessment report: A document summarizing the assessment results, e.g. conclusions, recommendations and findings. See also process assessment.

EITP

assessor: A person who conducts an assessment; any member of an assessment team.

ATT

atomic condition: A condition that cannot be decomposed, i.e., a condition that does not contain two or more single conditions joined by a logical operator (AND, OR, XOR).

F

attack: Directed and focused attempt to evaluate the quality, especially reliability, of a test object by attempting to force specific failures to occur. See also negative testing.

attack-based testing: An experience-based testing technique that uses software attacks to induce failures, particularly security related failures. See also attack.

ATA

attractiveness: The capability of the software product to be attractive to the user. [ISO 9126] See also usability.

ATM

audit: An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify:
(1) the form or content of the products to be produced
(2) the process by which the products shall be produced
(3) how compliance to standards or guidelines shall be measured. [IEEE 1028]

audit trail: A path by which the original input to a process (e.g. data) can be traced back through the process, taking the process output as a starting point. This facilitates defect analysis and allows a process audit to be carried out. [After TMap]

automated testware: Testware used in automated testing, such as tool scripts.

availability: The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage. [IEEE 610]

B

back-to-back testing: Testing in which two or more variants of a component or system are executed with the same inputs, the outputs compared, and analyzed in cases of discrepancies. [IEEE 610]

EITP

balanced scorecard: A strategic tool for measuring whether the operational activities of a company are aligned with its objectives in terms of business vision and strategy. See also corporate dashboard, scorecard.

baseline: A specification or software product that has been formally reviewed or agreed upon, that thereafter serves as the basis for further development, and that can be changed only through a formal change control process. [After IEEE 610]

basic block: A sequence of one or more consecutive executable statements containing no branches. Note: A node in a control flow graph represents a basic block.

basis test set: A set of test cases derived from the internal structure of a component or specification to ensure that 100% of a specified coverage criterion will be achieved.

bebugging: [Abbott] See fault seeding.

behavior: The response of a component or system to a set of input values and preconditions.

benchmark test: (1) A standard against which measurements or comparisons can be made. (2) A test that is be used to compare components or systems to each other or to a standard as in (1). [After IEEE 610]

bespoke software: Software developed specifically for a set of users or customers. The opposite is off-the-shelf software.

best practice: A superior method or innovative practice that contributes to the improved performance of an organization under given context, usually recognized as ‘best’ by other peer organizations.

F

beta testing: Operational testing by potential and/or existing users/customers at an external site not otherwise involved with the developers, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes. Beta testing is often employed as a form of external acceptance testing for off-the-shelf software in order to acquire feedback from the market.

big-bang testing: An integration testing approach in which software elements, hardware elements, or both are combined all at once into a component or an overall system, rather than in stages. [After IEEE 610] See also integration testing.

black box technique: See black box test design technique.

F

ATA

black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

F

black box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system.

blocked test case: A test case that cannot be executed because the preconditions for its execution are not fulfilled.

bottom-up testing: An incremental approach to integration testing where the lowest level components are tested first, and then used to facilitate the testing of higher level components. This process is repeated until the component at the top of the hierarchy is tested. See also integration testing.

boundary value: An input value or output value which is on the edge of an equivalence partition or at the smallest incremental distance on either side of an edge, for example the minimum or maximum value of a range.

F

ATA

boundary value analysis: A black box test design technique in which test cases are designed based on boundary values. See also boundary value.

boundary value coverage: The percentage of boundary values that have been exercised by a test suite.

boundary value testing: See boundary value analysis.

branch: A basic block that can be selected for execution based on a program construct in which one of two or more alternative program paths is available, e.g. case, jump, go to, if-then-else.

branch condition: See condition.

branch condition combination coverage: See multiple condition coverage.

branch condition combination testing: See multiple condition testing.

branch condition coverage: See condition coverage.

branch coverage: The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage.

branch testing: A white box test design technique in which test cases are designed to execute branches.

buffer: A device or storage area used to store data temporarily for differences in rates of data flow, time or occurrence of events, or amounts of data that can be handled by the devices or processes involved in the transfer or use of the data. [IEEE 610]

buffer overflow: A memory access failure due to the attempt by a process to store data beyond the boundaries of a fixed length buffer, resulting in overwriting of adjacent memory areas or the raising of an overflow exception. See also buffer.

F

bug: See defect.

bug report: See defect report.

bug taxonomy: See defect taxonomy.

bug tracking tool: See defect management tool.

business process-based testing: An approach to testing in which test cases are designed based on descriptions and/or knowledge of business processes.

C

call graph: An abstract representation of calling relationships between subroutines in a program.

ATM

EITP

Capability Maturity Model Integration: A framework that describes the key elements of an effective product development and maintenance process. The Capability Maturity Model Integration covers best-practices for planning, engineering and managing product development and maintenance. [CMMI]

ATT

capture/playback tool: A type of test execution tool where inputs are recorded during manual testing in order to generate automated test scripts that can be executed later (i.e. replayed). These tools are often used to support automated regression testing.

capture/replay tool: See capture/playback tool.

CASE: Acronym for Computer Aided Software Engineering.

CAST: Acronym for Computer Aided Software Testing. See also test automation.

EITP

causal analysis: The analysis of defects to determine their root cause. [CMMI]

cause-effect analysis: See cause-effect graphing.

cause-effect decision table: See decision table.

EITP

cause-effect diagram: A graphical representation used to organize and display the interrelationships of various possible root causes of a problem. Possible causes of a real or potential defect or failure are organized in categories and subcategories in a horizontal tree-structure, with the (potential) defect or failure as the root node. [After Juran]

EITP

cause-effect graph: A graphical representation of inputs and/or stimuli (causes) with their associated outputs (effects), which can be used to design test cases.

ATA

cause-effect graphing: A black box test design technique in which test cases are designed from cause-effect graphs. [BS 7925/2]

certification: The process of confirming that a component, system or person complies with its specified requirements, e.g. by passing an exam.

change control: See configuration control.

change control board: See configuration control board.

EITP

change management: (1) A structured approach to transitioning individuals, and organizations from a current state to a desired future state. (2) Controlled way to effect a change, or a proposed change, to a product or service. See also configuration management.

ATT

changeability: The capability of the software product to enable specified modifications to be implemented. [ISO 9126] See also maintainability.

charter: See test charter.

checker: See reviewer.

ATA

checklist-based testing: An experience-based test design technique whereby the experienced tester uses a high-level list of items to be noted, checked, or remembered, or a set of rules or criteria against which a product has to be verified.

Chow’s coverage metrics: See N-switch coverage. [Chow]

classification tree: A tree showing equivalence partitions hierarchically ordered, which is used to design test cases in the classification tree method. See also classification tree method.

ATA

classification tree method: A black box test design technique in which test cases, described by means of a classification tree, are designed to execute combinations of representatives of input and/or output domains. [Grochtmann]

clear-box testing: See white-box testing.

ATM

EITP

CMMI: See Capability Maturity Model Integration.

code: Computer instructions and data definitions expressed in a programming language or in a form output by an assembler, compiler or other translator. [IEEE 610]

code analyzer: See static code analyzer.

F

code coverage: An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage.

code-based testing: See white box testing.

EITP

codependent behavior: Excessive emotional or psychological dependence on another person, specifically in trying to change that person’s current (undesirable) behavior while supporting them in continuing that behavior. For example, in software testing, complaining about late delivery to test and yet enjoying the necessary “heroism” working additional hours to make up time when delivery is running late, therefore reinforcing the lateness.

ATT

co-existence: The capability of the software product to co-exist with other independent software in a common environment sharing common resources. [ISO 9126] See also portability.

ATA

combinatorial testing: A means to identify a suitable subset of test combinations to achieve a predetermined level of coverage when testing an object with multiple parameters and where those parameters themselves each have several values, which gives rise to more combinations than are feasible to test in the time allowed. See also classification tree method, pairwise testing, orthogonal array testing.

F

Commercial Off-The-Shelf software: See off-the-shelf software.

comparator: See test comparator.

compatibility testing: See interoperability testing.

F

compiler: A software tool that translates programs expressed in a high order language into their machine language equivalents. [IEEE 610]

complete testing: See exhaustive testing.

completion criteria: See exit criteria.

F

complexity: The degree to which a component or system has a design and/or internal structure that is difficult to understand, maintain and verify. See also cyclomatic complexity.

compliance: The capability of the software product to adhere to standards, conventions or regulations in laws and similar prescriptions. [ISO 9126]

compliance testing: The process of testing to determine the compliance of the component or system.

component: A minimal software item that can be tested in isolation.

component integration testing: Testing performed to expose defects in the interfaces and interaction between integrated components.

component specification: A description of a component’s function in terms of its output values for specified input values under specified conditions, and required non-functional behavior (e.g. resource-utilization).

F

component testing: The testing of individual software components. [After IEEE 610]

compound condition: Two or more single conditions joined by means of a logical operator (AND, OR or XOR), e.g. ‘A>B AND C>1000’.

ATA

concrete test case: See low level test case.

concurrency testing: Testing to determine how the occurrence of two or more activities within the same interval of time, achieved either by interleaving the activities or by simultaneous execution, is handled by the component or system. [After IEEE 610]

condition: A logical expression that can be evaluated as True or False, e.g. A>B. See also condition testing.

condition combination coverage: See multiple condition coverage.

condition combination testing: See multiple condition testing.

condition coverage: The percentage of condition outcomes that have been exercised by a test suite. 100% condition coverage requires each single condition in every decision statement to be tested as True and False.

condition determination coverage: See modified condition decision coverage.

condition determination testing: See modified condition decision testing.

condition outcome: The evaluation of a condition to True or False.

ATT

condition testing: A white box test design technique in which test cases are designed to execute condition outcomes.

ETM

confidence interval: In managing project risks, the period of time within which a contingency action must be implemented in order to be effective in reducing the impact of the risk.

confidence test: See smoke test.

configuration: The composition of a component or system as defined by the number, nature, and interconnections of its constituent parts.

configuration auditing: The function to check on the contents of libraries of configuration items, e.g. for standards compliance. [IEEE 610]

configuration control: An element of configuration management, consisting of the evaluation, co-ordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. [IEEE 610]

configuration control board (CCB): A group of people responsible for evaluating and approving or disapproving proposed changes to configuration items, and for ensuring implementation of approved changes. [IEEE 610]

configuration identification: An element of configuration management, consisting of selecting the configuration items for a system and recording their functional and physical characteristics in technical documentation. [IEEE 610]

configuration item: An aggregation of hardware, software or both, that is designated for configuration management and treated as a single entity in the configuration management process. [IEEE 610]

F

configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements. [IEEE 610]

F

configuration management tool: A tool that provides support for the identification and control of configuration items, their status over changes and versions, and the release of baselines consisting of configuration items.

configuration testing: See portability testing.

F

confirmation testing: See re-testing.

conformance testing: See compliance testing.

consistency: The degree of uniformity, standardization, and freedom from contradiction among the documents or parts of a component or system. [IEEE 610]

ETM

consultative testing: Testing driven by the advice and guidance of appropriate experts from outside the test team (e.g., technology experts and/or business domain experts).

EITP

content-based model: A process model providing a detailed description of good engineering practices, e.g. test practices.

EITP

continuous representation: A capability maturity model structure wherein capability levels provide a recommended order for approaching process improvement within specified process areas. [CMMI]

ETM

control chart: A statistical process control tool used to monitor a process and determine whether it is statistically controlled. It graphically depicts the average value and the upper and lower control limits (the highest and lowest values) of a process.

F

control flow: A sequence of events (paths) in the execution through a component or system.

ATT

control flow analysis: A form of static analysis based on a representation of unique paths (sequences of events) in the execution through a component or system. Control flow analysis evaluates the integrity of control flow structures, looking for possible control flow anomalies such as closed loops or logically unreachable process steps.

control flow graph: An abstract representation of all possible sequences of events (paths) in the execution through a component or system.

control flow path: See path.

ATT

control flow testing: An approach to structure-based testing in which test cases are designed to execute specific sequences of events. Various techniques exist for control flow testing, e.g., decision testing, condition testing, and path testing, that each have their specific approach and level of control flow coverage. See also decision testing, condition testing, path testing.

ETM

convergence metric: A metric that shows progress toward a defined criterion, e.g., convergence of the total number of test executed to the total number of tests planned for execution.

conversion testing: Testing of software used to convert data from existing systems for use in replacement systems.

EITP

corporate dashboard: A dashboard-style representation of the status of corporate performance data. See also balanced scorecard, dashboard.

cost of quality: The total costs incurred on quality activities and issues and often split into prevention costs, appraisal costs, internal failure costs and external failure costs.

F

COTS: Acronym for Commercial Off-The-Shelf software. See off-the-shelf software.

coverage: The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite.

coverage analysis: Measurement of achieved coverage to a specified coverage item during test execution referring to predetermined criteria to determine whether additional testing is required and if so, which test cases are needed.

coverage item: An entity or property used as a basis for test coverage, e.g. equivalence partitions or code statements.

coverage measurement tool: See coverage tool.

F

coverage tool: A tool that provides objective measures of what structural elements, e.g. statements, branches have been exercised by a test suite.

EITP

critical success factor: An element necessary for an organization or project to achieve its mission. Critical success factors are the critical factors or activities required for ensuring the success.

ATM

EITP

Critical Testing Processes: A content-based model for test process improvement built around twelve critical processes. These include highly visible processes, by which peers and management judge competence and mission-critical processes in which performance affects the company’s profits and reputation. See also content-based model.

ATM

EITP

CTP: See Critical Testing Processes.

custom software: See bespoke software.

ATM

custom tool: A software tool developed specifically for a set of users or customers.

ATT

cyclomatic complexity: The maximum number of linear, independent paths through a program. Cyclomatic complexity may be computed as: L – N + 2P, where

- L = the number of edges/links in a graph

- N = the number of nodes in a graph

- P = the number of disconnected parts of the graph (e.g. a called graph or subroutine) [After McCabe]

cyclomatic number: See cyclomatic complexity.

D

daily build: A development activity whereby a complete system is compiled and linked every day (often overnight), so that a consistent system is available at any time including all latest changes.

ETM

dashboard: A representation of dynamic measurements of operational performance for some organization or activity, using metrics represented via metaphores such as visual ‘dials’, ‘counters’, and other devices resembling those on the dashboard of an automobile, so that the effects of events or activities can be easily understood and related to operational goals. See also corporate dashboard, scorecard.

data definition: An executable statement where a variable is assigned a value.

F

ATT

data-driven testing: A scripting technique that stores test input and expected results in a table or spreadsheet, so that a single control script can execute all of the tests in the table. Data-driven testing is often used to support the application of test execution tools such as capture/playback tools. [Fewster and Graham] See also keyword-driven testing.

F

data flow: An abstract representation of the sequence and possible changes of the state of data objects, where the state of an object is any of: creation, usage, or destruction. [Beizer]

ATT

data flow analysis: A form of static analysis based on the definition and usage of variables.

data flow coverage: The percentage of definition-use pairs that have been exercised by a test suite.

data flow testing: A white box test design technique in which test cases are designed to execute definition-use pairs of variables.

data integrity testing: See database integrity testing.

data quality; An attribute of data that indicates correctness with respect to some pre-defined criteria, e.g., business expectations, requirements on data integrity, data consistency.

database integrity testing: Testing the methods and processes used to access and manage the data(base), to ensure access methods, processes and data rules function as expected and that during access to the database, data is not corrupted or unexpectedly deleted, updated or created.

dd-path: A path between two decisions of an algorithm, or two decision nodes of a corresponding graph, that includes no other decisions. See also path.

dead code: See unreachable code.

debugger: See debugging tool.

F

debugging: The process of finding, analyzing and removing the causes of failures in software.

F

ATT

debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs and find the corresponding defect. Debuggers enable programmers to execute programs step by step, to halt a program at any program statement and to set and examine program variables.

decision: A program point at which the control flow has two or more alternative routes. A node with two or more links to separate branches.

decision condition coverage: The percentage of all condition outcomes and decision outcomes that have been exercised by a test suite. 100% decision condition coverage implies both 100% condition coverage and 100% decision coverage.

ATT

decision condition testing: A white box test design technique in which test cases are designed to execute condition outcomes and decision outcomes.

F

decision coverage: The percentage of decision outcomes that have been exercised by a test suite. 100% decision coverage implies both 100% branch coverage and 100% statement coverage.

decision outcome: The result of a decision (which therefore determines the branches to be taken).

decision table: A table showing combinations of inputs and/or stimuli (causes) with their associated outputs and/or actions (effects), which can be used to design test cases.

F

ATA

decision table testing: A black box test design technique in which test cases are designed to execute the combinations of inputs and/or stimuli (causes) shown in a decision table. [Veenendaal04] See also decision table.

decision testing: A white box test design technique in which test cases are designed to execute decision outcomes.

F

ATM

defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

ATA

defect-based technique: See defect-based test design technique.

ATA

defect-based test design technique: A procedure to derive and/or select test cases targeted at one or more defect categories, with tests being developed from what is known about the specific defect category. See also defect taxonomy.

defect category: See defect type.

F

defect density: The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, e.g. lines-of-code, number of classes or function points).

EITP

Defect Detection Percentage (DDP): The number of defects found by a test phase, divided by the number found by that test phase and any other means afterwards.

defect management: The process of recognizing, investigating, taking action and disposing of defects. It involves recording defects, classifying them and identifying the impact. [After IEEE 1044]

ATM

defect management committee: A cross-functional team of stakeholders who manage reported defects from initial detection to ultimate resolution (defect removal, defect deferral, or report cancellation). In some cases, the same team as the configuration control board. See also configuration control board.

defect management tool: A tool that facilitates the recording and status tracking of defects and changes. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of defects and provide reporting facilities. See also incident management tool.

defect masking: An occurrence in which one defect prevents the detection of another. [After IEEE 610]

defect report: A document reporting on any flaw in a component or system that can cause the component or system to fail to perform its required function. [After IEEE 829]

ATA

defect taxonomy: A system of (hierarchical) categories designed to be a useful aid for reproducibly classifying defects.

defect tracking tool: See defect management tool.

ATM

defect triage committee: See defect management committee.

defect type: An element in a taxonomy of defects. Defect taxonomies can be identified with respect to a variety of considerations, including, but not limited to:

- Phase or development activity in which the defect is created, e.g., a specification error or a coding error

- Characterization of defects, e.g., an “off-by-one” defect

- Incorrectness, e.g., an incorrect relational operator, a programming language syntax error, or an invalid assumption

- Performance issues, e.g., excessive execution time, insufficient availability.

ATT

definition-use pair: The association of a definition of a variable with the subsequent use of that variable. Variable uses include computational (e.g. multiplication) or to direct the execution of a path (“predicate” use).

deliverable: Any (work) product that must be delivered to someone other than the (work) product’s author.

EITP

Deming cycle: An iterative four-step problem-solving process, (plan-do-check-act), typically used in process improvement. [After Deming]

design-based testing: An approach to testing in which test cases are designed based on the architecture and/or detailed design of a component or system (e.g. tests of interfaces between components or systems).

desk checking: Testing of software or a specification by manual simulation of its execution. See also static testing.

development testing: Formal or informal testing conducted during the implementation of a component or system, usually in the development environment by developers. [After IEEE 610]

deviation: See incident.

deviation report: See incident report.

EITP

diagnosing (IDEAL): The phase within the IDEAL model where it is determined where one is, relative to where one wants to be. The diagnosing phase consists of the activities: characterize current and desired states and develop recommendations. See also IDEAL.

dirty testing: See negative testing.

documentation testing: Testing the quality of the documentation, e.g. user guide or installation guide.

domain: The set from which valid input and/or output values can be selected.

ATA

domain analysis: A black box test design technique that is used to identify efficient and effective test cases when multiple variables can or should be tested together. It builds on and generalizes equivalence partitioning and boundary values analysis. See also boundary value analysis, equivalence partitioning.

F

driver: A software component or test tool that replaces a component that takes care of the control and/or the calling of a component or system. [After TMap]

ATT

dynamic analysis: The process of evaluating behavior, e.g. memory performance, CPU usage, of a system or component during execution. [After IEEE 610]

F

dynamic analysis tool: A tool that provides run-time information on the state of the software code. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic and to monitor the allocation, use and de-allocation of memory and to flag memory leaks.

dynamic comparison: Comparison of actual and expected results, performed while the software is being executed, for example by a test execution tool.

F

dynamic testing: Testing that involves the execution of the software of a component or system.

E

ATM

effectiveness: The capability of producing an intended result. See also efficiency.

ATM

ATT

efficiency: (1) The capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions. [ISO 9126]
(2) The capability of a process to produce the intended outcome, relative to the amount of resources used

efficiency testing: The process of testing to determine the efficiency of a software product.

EITP

EFQM (European Foundation for Quality Management) excellence model: A non-prescriptive framework for an organisation’s quality management system, defined and owned by the European Foundation for Quality Management, based on five ‘Enabling’ criteria (covering what an organisation does), and four ‘Results’ criteria (covering what an organisation achieves).

elementary comparison testing: A black box test design technique in which test cases are designed to execute combinations of inputs using the concept of modified condition decision coverage. [TMap]

embedded iterative development model: A development lifecycle sub-model that applies an iterative approach to detailed design, coding and testing within an overall sequential model. In this case, the high level design documents are prepared and approved for the entire project but the actual detailed design, code development and testing are conducted in iterations.

EITP

emotional intelligence: The ability, capacity, and skill to identify, assess, and manage the emotions of one’s self, of others, and of groups.

emulator: A device, computer program, or system that accepts the same inputs and produces the same outputs as a given system. [IEEE 610] See also simulator.

F

entry criteria: The set of generic and specific conditions for permitting a process to go forward with a defined task, e.g. test phase. The purpose of entry criteria is to prevent a task from starting which would entail more (wasted) effort compared to the effort needed to remove the failed entry criteria. [Gilb and Graham]

entry point: An executable statement or process step which defines a point at which a given process is intended to begin.

equivalence class: See equivalence partition.

equivalence partition: A portion of an input or output domain for which the behavior of a component or system is assumed to be the same, based on the specification.

equivalence partition coverage: The percentage of equivalence partitions that have been exercised by a test suite.

F

ATA

equivalence partitioning: A black box test design technique in which test cases are designed to execute representatives from equivalence partitions. In principle test cases are designed to cover each partition at least once.

F

error: A human action that produces an incorrect result. [After IEEE 610]

F

ATA

error guessing: A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them.

error seeding: See fault seeding.

error seeding tool: See fault seeding tool.

error tolerance: The ability of a system or component to continue normal operation despite the presence of erroneous inputs. [After IEEE 610].

EITP

establishing (IDEAL): The phase within the IDEAL model where the specifics of how an organization will reach its destination are planned. The establishing phase consists of the activities: set priorities, develop approach and plan actions. See also IDEAL.

evaluation: See testing.

exception handling: Behavior of a component or system in response to erroneous input, from either a human user or from another component or system, or to an internal failure.

executable statement: A statement which, when compiled, is translated into object code, and which will be executed procedurally when the program is running and may perform an action on data.

exercised: A program element is said to be exercised by a test case when the input value causes the execution of that element, such as a statement, decision, or other structural element.

F

exhaustive testing: A test approach in which the test suite comprises all combinations of input values and preconditions.

F

ATM

ATA

exit criteria: The set of generic and specific conditions, agreed upon with the stakeholders for permitting a process to be officially completed. The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham]

exit point: An executable statement or process step which defines a point at which a given process is intended to cease..

expected outcome: See expected result.

expected result: The behavior predicted by the specification, or another source, of the component or system under specified conditions.

ATA

experience-based technique: See experience-based test design technique.

F

ATA

experience-based test design technique: Procedure to derive and/or select test cases based on the tester’s experience, knowledge and intuition.

experience-based testing: Testing based on the tester’s experience, knowledge and intuition.

F

ATA

exploratory testing: An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. [After Bach]

EITP

extreme programming (XP): A software engineering methodology used within agile software development whereby core practices are programming in pairs, doing extensive code review, unit testing of all code, and simplicity and clarity in code. See also agile software development.

F

factory acceptance testing: Acceptance testing conducted at the site at which the product is developed and performed by employees of the supplier organization, to determine whether or not a component or system satisfies the requirements, normally including hardware as well as software. See also alfa testing.

fail: A test is deemed to fail if its actual result does not match its expected result.

failover testing: Testing by simulating failure modes or actually causing failures in a controlled environment. Following a failure, the failover mechanism is tested to ensure that data is not lost or corrupted and that any agreed service levels are maintained (e.g., function availability or response times). See also recoverability testing.

F

ATM

failure: Deviation of the component or system from its expected delivery, service or result. [After Fenton]

failure mode: The physical or functional manifestation of a failure. For example, a system in failure mode may be characterized by slow operation, incorrect outputs, or complete termination of execution. [IEEE 610]

EITP

Failure Mode and Effect Analysis (FMEA): A systematic approach to risk identification and analysis of identifying possible modes of failure and attempting to prevent their occurrence. See also Failure Mode, Effect and Criticality Analysis (FMECA).

Failure Mode, Effects, and Criticality Analysis (FMECA): An extension of FMEA, as in addition to the basic FMEA, it includes a criticality analysis, which is used to chart the probability of failure modes against the severity of their consequences. The result highlights failure modes with relatively high probability and severity of consequences, allowing remedial effort to be directed where it will produce the greatest value. See also Failure Mode and Effect Analysis (FMEA).

F

failure rate: The ratio of the number of failures of a given category to a given unit of measure, e.g. failures per unit of time, failures per number of transactions, failures per number of computer runs. [IEEE 610]

ATM

false-fail result: A test result in which a defect is reported although no such defect actually exists in the test object.

ATM

false-pass result: A test result which fails to identify the presence of a defect that is actually present in the test object.

ATM

false-positive result: See false-fail result.

ATM

false-negative result: See false-pass result.

F

fault: See defect.

F

fault attack: See attack.

fault density: See defect density.

Fault Detection Percentage (FDP): See Defect Detection Percentage (DDP).

fault injection: The process of intentionally adding defects to a system for the purpose of finding out whether the system can detect, and possibly recover from, a defect. Fault injection intended to mimic failures that might occur in the field. See also fault tolerance.

fault masking: See defect masking.

fault seeding: The process of intentionally adding defects to those already in the component or system for the purpose of monitoring the rate of detection and removal, and estimating the number of remaining defects. Fault seeding is typically part of development (prerelease) testing and can be performed at any test level (component, integration, or system). [After IEEE 610]

ATT

fault seeding tool: A tool for seeding (i.e. intentionally inserting) faults in a component or system.

fault tolerance: The capability of the software product to maintain a specified level of performance in cases of software faults (defects) or of infringement of its specified interface. [ISO 9126] See also reliability, robustness.

EITP

Fault Tree Analysis (FTA): A technique used to analyze the causes of faults (defects). The technique visually models how logical relationships between failures, human errors, and external events can combine to cause specific faults to disclose.

feasible path: A path for which a set of input values and preconditions exists which causes it to be executed.

feature: An attribute of a component or system specified or implied by requirements documentation (for example reliability, usability or design constraints). [After IEEE 1008]

ETM

feature-driven development: An iterative and incremental software development process driven from a client-valued functionality (feature) perspective. Feature-driven development is mostly used in agile software development. See also agile software development.

F

field testing: See beta testing.

finite state machine: A computational model consisting of a finite number of states and transitions between those states, possibly with accompanying actions. [IEEE 610]

finite state testing: See state transition testing.

fishbone diagram: See cause-effect diagram.

F

formal review: A review characterized by documented procedures and requirements, e.g. inspection.

frozen test basis: A test basis document that can only be amended by a formal change control process. See also baseline.

Function Point Analysis (FPA): Method aiming to measure the size of the functionality of an information system. The measurement is independent of the technology. This measurement may be used as a basis for the measurement of productivity, the estimation of the needed resources, and project control.

functional integration: An integration approach that combines the components or systems for the purpose of getting a basic functionality working early. See also integration testing.

F

functional requirement: A requirement that specifies a function that a component or system must perform. [IEEE 610]

functional test design technique: Procedure to derive and/or select test cases based on an analysis of the specification of the functionality of a component or system without reference to its internal structure. See also black box test design technique.

F

functional testing: Testing based on an analysis of the specification of the functionality of a component or system. See also black box testing.

functionality: The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. [ISO 9126]

functionality testing: The process of testing to determine the functionality of a software product.

G

glass box testing: See white box testing.

EITP

Goal Question Metric: An approach to software measurement using a three-level model: conceptual level (goal), operational level (question) and quantitative level (metric).

EITP

GQM: See Goal Question Metric.

H

hardware-software integration testing: Testing performed to expose defects in the interfaces and interaction between hardware and software components. See also integration testing.

hazard analysis: A technique used to characterize the elements of risk. The result of a hazard analysis will drive the methods used for development and testing of a system. See also risk analysis.

ATA

heuristic evaluation: A usability review technique that targets usability problems in the user interface or user interface design. With this technique, the reviewers examine the interface and judge its compliance with recognized usability principles (the “heuristics”).

ATA

high level test case: A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available. See also low level test case.

horizontal traceability: The tracing of requirements for a test level through the layers of test documentation (e.g. test plan, test design specification, test case specification and test procedure specification or test script).

hyperlink: A pointer within a web page that leads to other web pages.

ATT

hyperlink test tool: A tool used to check that no broken hyperlinks are present on a web site.

I

EITP

IDEAL: An organizational improvement model that serves as a roadmap for initiating, planning, and implementing improvement actions. The IDEAL model is named for the five phases it describes: initiating, diagnosing, establishing, acting, and learning.

F

impact analysis: The assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements.

F

incident: Any event occurring that requires investigation. [After IEEE 1008]

F

incident logging: Recording the details of any incident that occurred, e.g. during testing.

F

incident management: The process of recognizing, investigating, taking action and disposing of incidents. It involves logging incidents, classifying them and identifying the impact. [After IEEE 1044]

F

incident management tool: A tool that facilitates the recording and status tracking of incidents. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of incidents and provide reporting facilities. See also defect management tool.

F

incident report: A document reporting on any event that occurred, e.g. during the testing, which requires investigation. [After IEEE 829]

F

incremental development model: A development lifecycle where a project is broken into a series of increments, each of which delivers a portion of the functionality in the overall project requirements. The requirements are prioritized and delivered in priority order in the appropriate increment. In some (but not all) versions of this lifecycle model, each subproject follows a ‘mini V-model’ with its own design, coding and testing phases.

incremental testing: Testing where components or systems are integrated and tested one or some at a time, until all the components or systems are integrated and tested.

F

ATM

independence of testing: Separation of responsibilities, which encourages the accomplishment of objective testing. [After DO-178b]

EITP

indicator: A measure that can be used to estimate or predict another measure. [ISO 14598]

infeasible path: A path that cannot be exercised by any set of possible input values.

F

ATM

informal review: A review not based on a formal (documented) procedure.

EITP

initiating (IDEAL): The phase within the IDEAL model where the groundwork is laid for a successful improvement effort. The initiating phase consists of the activities: set context, build sponsorship and charter infrastructure. See also IDEAL.

input: A variable (whether stored within a component or outside) that is read by a component.

input domain: The set from which valid input values can be selected. See also domain.

input value: An instance of an input. See also input.

insourced testing: Testing performed by people who are co-located with the project team but are not fellow employees.

F

ATM

EITP

inspection: A type of peer review that relies on visual examination of documents to detect defects, e.g. violations of development standards and non-conformance to higher level documentation. The most formal review technique and therefore always based on a documented procedure. [After IEEE 610, IEEE 1028] See also peer review.

inspection leader: See moderator.

inspector: See reviewer.

ATT

installability: The capability of the software product to be installed in a specified environment [ISO 9126]. See also portability.

installability testing: The process of testing the installability of a software product. See also portability testing.

installation guide: Supplied instructions on any suitable media, which guides the installer through the installation process. This may be a manual guide, step-by-step procedure, installation wizard, or any other similar process description.

installation wizard: Supplied software on any suitable media, which leads the installer through the installation process. It normally runs the installation process, provides feedback on installation results, and prompts for options.

instrumentation: The insertion of additional code into the program in order to collect information about program behavior during execution, e.g. for measuring code coverage.

instrumenter: A software tool used to carry out instrumentation.

intake test: A special instance of a smoke test to decide if the component or system is ready for detailed and further testing. An intake test is typically carried out at the start of the test execution phase. See also smoke test.

F

integration: The process of combining components or systems into larger assemblies.

F

integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing.

integration testing in the large: See system integration testing.

integration testing in the small: See component integration testing.

interface testing: An integration test type that is concerned with testing the interfaces between components or systems.

interoperability: The capability of the software product to interact with one or more specified components or systems. [After ISO 9126] See also functionality.

F

ATA

interoperability testing: The process of testing to determine the interoperability of a software product. See also functionality testing.

invalid testing: Testing using input values that should be rejected by the component or system. See also error tolerance, negative testing.

ETM

Ishikawa diagram: See cause-effect diagram.

isolation testing: Testing of individual components in isolation from surrounding components, with surrounding components being simulated by stubs and drivers, if needed.

item transmittal report: See release note.

F

iterative development model: A development lifecycle where a project is broken into a usually large number of iterations. An iteration is a complete development loop resulting in a release (internal or external) of an executable product, a subset of the final product under development, which grows from iteration to iteration to become the final product.

K

key performance indicator: See performance indicator.

F

ATA

ATT

keyword-driven testing: A scripting technique that uses data files to contain not only test data and expected results, but also keywords related to the application being tested. The keywords are interpreted by special supporting scripts that are called by the control script for the test. See also data-driven testing.

L

LCSAJ: A Linear Code Sequence And Jump, consists of the following three items (conventionally identified by line numbers in a source code listing): the start of the linear sequence of executable statements, the end of the linear sequence, and the target line to which control flow is transferred at the end of the linear sequence.

LCSAJ coverage: The percentage of LCSAJs of a component that have been exercised by a test suite. 100% LCSAJ coverage implies 100% decision coverage.

LCSAJ testing: A white box test design technique in which test cases are designed to execute LCSAJs.

EITP

lead assessor: The person who leads an assessment. In some cases, for instance CMMi and TMMi when formal assessments are conducted, the lead assessor must be accredited and formally trained.

ATA

learnability: The capability of the software product to enable the user to learn its application. [ISO 9126] See also usability.

EITP

learning (IDEAL): The phase within the IDEAL model where one learns from experiences and improves one’s ability to adopt new processes and technologies in the future. The learning phase consists of the activities: analyze and validate, and propose future actions. See also IDEAL.

ATM

level test plan: A test plan that typically addresses one test level. See also test plan.

EITP

lifecycle model: A partitioning of the life of a product or project into phases. [CMMI] See also software lifecycle.

link testing: See component integration testing.

load profile: A specification of the activity which a component or system being tested may experience in production. A load profile consists of a designated number of virtual users who process a defined set of transactions in a specified time period and according to a predefined operational profile. See also operational profile.

F

load testing: A type of performance testing conducted to evaluate the behavior of a component or system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to determine what load can be handled by the component or system. See also performance testing, stress testing.

F

load testing tool: A tool to support load testing whereby it can simulate increasing load, e.g., numbers of concurrent users and/or transactions within a specified time-period. See also performance testing tool.

logic-coverage testing: See white box testing. [Myers]

logic-driven testing: See white box testing.

ATA

logical test case: See high level test case.

ATA

low level test case: A test case with concrete (implementation level) values for input data and expected results. Logical operators from high level test cases are replaced by actual values that correspond to the objectives of the logical operators. See also high level test case.

M

man in the middle attack: The interception, mimicking and/or altering and subsequent relaying of communications (e.g., credit card transactions) by a third party such that a user remains unaware of that third party’s presence.

maintainability: The ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment. [ISO 9126]

F

ATT

maintainability testing: The process of testing to determine the maintainability of a software product.

maintenance: Modification of a software product after delivery to correct defects, to improve performance or other attributes, or to adapt the product to a modified environment. [IEEE 1219]

F

maintenance testing: Testing the changes to an operational system or the impact of a changed environment to an operational system.

ATM

management review: A systematic evaluation of software acquisition, supply, development, operation, or maintenance process, performed by or on behalf of management that monitors progress, determines the status of plans and schedules, confirms requirements and their system allocation, or evaluates the effectiveness of management approaches to achieve fitness for purpose. [After IEEE 610, IEEE 1028]

EITP

manufacturing-based quality: A view of quality, whereby quality is measured by the degree to which a product or service conforms to its intended design and requirements. Quality arises from the process(es) used. [After Garvin] See also product-based quality, transcendent-based quality, user-based quality, value-based quality.

ATM

master test plan: A test plan that typically addresses multiple test levels. See also test plan.

ATT

maturity: (1) The capability of an organization with respect to the effectiveness and efficiency of its processes and work practices. See also Capability Maturity Model Integration, Test Maturity Model integration.
(2) The capability of the software product to avoid failure as a result of defects in the software. [ISO 9126] See also reliability.

EITP

maturity level: Degree of process improvement across a predefined set of process areas in which all goals in the set are attained. [TMMi]

maturity model: A structured collection of elements that describe certain aspects of maturity in an organization, and aid in the definition and understanding of an organization’s processes. A maturity model often provides a common language, shared vision and framework for prioritizing improvement actions.

Mean Time Between Failures: The arithmetic mean (average) time between failures of a system. The MTBF is typically part of a reliability growth model that assumes the failed system is immediately repaired, as a part of a defect fixing process. See also reliability growth model.

Mean Time To Repair: The arithmetic mean (average) time a system will take to recover from any failure. This typically includes testing to insure that the defect has been resolved.

EITP

measure: The number or category assigned to an attribute of an entity by making a measurement. [ISO 14598]

measurement: The process of assigning a number or category to an entity to describe an attribute of that entity. [ISO 14598]

measurement scale: A scale that constrains the type of data analysis that can be performed on it. [ISO 14598]

ATT

memory leak: A memory access failure due to a defect in a program’s dynamic store allocation logic that causes it to fail to release memory after it has finished using it, eventually causing the program and/or other concurrent processes to fail due to lack of memory.

ETM

methodical testing: Testing based on a standard set of tests, e.g., a checklist, a quality standard, or a set of generalized test cases.

F

EITP

metric: A measurement scale and the method used for measurement. [ISO 14598]

migration testing: See conversion testing.

milestone: A point in time in a project at which defined (intermediate) deliverables and results should be ready.

EITP

mind map: A diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central keyword or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.

F

mistake: See error.

ETM

model-based testing: Testing based on a model of the component or system under test, e.g., reliability growth models, usage models such as operational profiles or behavioural models such as decision table or state transition diagram.

F

modeling tool: A tool that supports the creation, amendment and verification of models of the software or system [Graham].

F

ATM

moderator: The leader and main person responsible for an inspection or other review process.

modified condition decision coverage: The percentage of all single condition outcomes that independently affect a decision outcome that have been exercised by a test case suite. 100% modified condition decision coverage implies 100% decision condition coverage.

modified condition decision testing: A white box test design technique in which test cases are designed to execute single condition outcomes that independently affect a decision outcome.

modified multiple condition coverage: See modified condition decision coverage.

modified multiple condition testing: See modified condition decision testing.

module: See component.

module testing: See component testing.

monitor: A software tool or hardware device that runs concurrently with the component or system under test and supervises, records and/or analyses the behavior of the component or system. [After IEEE 610]

F

monitoring tool: See monitor.

monkey testing: Testing by means of a random selection from a large range of inputs and by randomly pushing buttons, ignorant of how the product is being used.

MTBF: See Mean Time Between Failures.

MTTR: See Mean Time To Repair.

multiple condition: See compound condition.

multiple condition coverage: The percentage of combinations of all single condition outcomes within one statement that have been exercised by a test suite. 100% multiple condition coverage implies 100% modified condition decision coverage.

ATT

multiple condition testing: A white box test design technique in which test cases are designed to execute combinations of single condition outcomes (within one statement).

mutation analysis: A method to determine test suite thoroughness by measuring the extent to which a test suite can discriminate the program from slight variants (mutants) of the program.

mutation testing: See back-to-back testing.

ETM

Myers-Briggs Type Indicator (MBTI): An indicator of psychological preference representing the different personalities and communication styles of people.

N

N-switch coverage: The percentage of sequences of N+1 transitions that have been exercised by a test suite. [Chow]

N-switch testing: A form of state transition testing in which test cases are designed to execute all valid sequences of N+1 transitions. [Chow] See also state transition testing.

negative testing: Tests aimed at showing that a component or system does not work. Negative testing is related to the testers’ attitude rather than a specific test approach or test design technique, e.g. testing with invalid input values or exceptions. [After Beizer].

ATT

neighborhood integration testing: A form of integration testing where all of the nodes that connect to a given node are the basis for the integration testing.

non-conformity: Non fulfillment of a specified requirement. [ISO 9000]

F

non-functional requirement: A requirement that does not relate to functionality, but to attributes such as reliability, efficiency, usability, maintainability and portability.

non-functional test design technique: Procedure to derive and/or select test cases for non-functional testing based on an analysis of the specification of a component or system without reference to its internal structure. See also black box test design technique.

non-functional testing: Testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability.

O

off-the-shelf software: A software product that is developed for the general market, i.e. for a large number of customers, and that is delivered to many customers in identical format.

ATM

open source tool: A software tool that is available to all potential users in source code form, usually via the internet; its users are permitted, usually under licence, to study, change, improve and, at times, to distribute the software.

ATA

operability: The capability of the software product to enable the user to operate and control it. [ISO 9126] See also usability.

ATT

operational acceptance testing: Operational testing in the acceptance test phase, typically performed in a (simulated) operational environment by operations and/or systems administration staff focusing on operational aspects, e.g. recoverability, resource-behavior, installability and technical compliance. See also operational testing.

operational environment: Hardware and software products installed at users’ or customers’ sites where the component or system under test will be used. The software may include operating systems, database management systems, and other applications.

ETM

ATT

operational profile: The representation of a distinct set of tasks performed by the component or system, possibly based on user behavior when interacting with the component or system, and their probabilities of occurrence. A task is logical rather that physical and can be executed over several machines or be executed in non-contiguous time segments.

operational profile testing: Statistical testing using a model of system operations (short duration tasks) and their probability of typical use. [Musa]

ETM

operational profiling: The process of developing and implementing an operational profile. See also operational profile.

operational testing: Testing conducted to evaluate a component or system in its operational environment. [IEEE 610]

oracle: See test oracle.

ATA

orthogonal array: A 2-dimensional array constructed with special mathematical properties, such that choosing any two columns in the array provides every pair combination of each number in the array.

orthogonal array testing: A systematic way of testing all-pair combinations of variables using orthogonal arrays. It significantly reduces the number of all combinations of variables to test all pair combinations. See also pairwise testing.

outcome: See result.

output: A variable (whether stored within a component or outside) that is written by a component.

output domain: The set from which valid output values can be selected. See also domain.

output value: An instance of an output. See also output.

outsourced testing: Testing performed by people who are not co-located with the project team and are not fellow employees.

P

pair programming: A software development approach whereby lines of code (production and/or test) of a component are written by two programmers sitting at a single computer. This implicitly means ongoing real-time code reviews are performed.

pair testing: Two persons, e.g. two testers, a developer and a tester, or an end-user and a tester, working together to find defects. Typically, they share one computer and trade control of it while testing.

ATT

pairwise integration testing: A form of integration testing that targets pairs of components that work together, as shown in a call graph.

ATA

pairwise testing: A black box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters. See also orthogonal array testing.

EITP

Pareto analysis: A statistical technique in decision making that is used for selection of a limited number of factors that produce significant overall effect. In terms of quality improvement, a large majority of problems (80%) are produced by a few key causes (20%).

partition testing: See equivalence partitioning. [Beizer]

pass: A test is deemed to pass if its actual result matches its expected result.

pass/fail criteria: Decision rules used to determine whether a test item (function) or feature has passed or failed a test. [IEEE 829]

path: A sequence of events, e.g. executable statements, of a component or system from an entry point to an exit point.

path coverage: The percentage of paths that have been exercised by a test suite. 100% path coverage implies 100% LCSAJ coverage.

path sensitizing: Choosing a set of input values to force the execution of a given path.

ATT

path testing: A white box test design technique in which test cases are designed to execute paths.

F

peer review: A review of a software work product by colleagues of the producer of the product for the purpose of identifying defects and improvements. Examples are inspection, technical review and walkthrough.

performance: The degree to which a system or component accomplishes its designated functions within given constraints regarding processing time and throughput rate. [After IEEE 610] See also efficiency.

performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive development, e.g. lead-time slip for software development. [CMMI]

performance profiling: The task of analyzing, e.g., identifying performance bottlenecks based on generated metrics, and tuning the performance of a software component or system using tools.

F

ATT

performance testing: The process of testing to determine the performance of a software product. See also efficiency testing.

F

ATT

performance testing tool: A tool to support performance testing that usually has two main facilities: load generation and test transaction measurement. Load generation can simulate either multiple users or high volumes of input data. During execution, response time measurements are taken from selected transactions and these are logged. Performance testing tools normally provide reports based on test logs and graphs of load against response times.

ATA

ATM

phase containment: The percentage of defects that are removed in the same phase of the software lifecycle in which they were introduced.

phase test plan: A test plan that typically addresses one test phase. See also test plan.

ETM

planning poker: A consensus-based estimation technique, mostly used to estimate effort or relative size of user stories in agile software development. It is a variation of the Wide Band Delphi method using a deck of cards with values representing the units in which the team estimates. See also agile software development, Wide Band Delphi.

pointer: A data item that specifies the location of another data item; for example, a data item that specifies the address of the next employee record to be processed. [IEEE 610]

portability: The ease with which the software product can be transferred from one hardware or software environment to another. [ISO 9126]

F

ATT

portability testing: The process of testing to determine the portability of a software product.

postcondition: Environmental and state conditions that must be fulfilled after the execution of a test or test procedure.

post-execution comparison: Comparison of actual and expected results, performed after the software has finished running.

post-project meeting: See retrospective meeting.

precondition: Environmental and state conditions that must be fulfilled before the component or system can be executed with a particular test or test procedure.

predicate: A statement that can evaluate to true or false and may be used to determine the control flow of subsequent decision logic. See also decision.

predicted outcome: See expected result.

pretest: See intake test.

ATM

priority: The level of (business) importance assigned to an item, e.g. defect.

F

probe effect: The effect on the component or system by the measurement instrument when the component or system is being measured, e.g. by a performance testing tool or monitor. For example performance may be slightly worse when performance testing tools are being used.

problem: See defect.

problem management: See defect management.

problem report: See defect report.

procedure testing: Testing aimed at ensuring that the component or system can operate in conjunction with new or existing users’ business procedures or operational procedures.

process: A set of interrelated activities, which transform inputs into outputs. [ISO 12207]

EITP

process assessment: A disciplined evaluation of an organization’s software processes against a reference model. [after ISO 15504]

ETM

process-compliant testing: Testing that follows a set of defined processes, e.g., defined by an external party such as a standards committee. See also standard-compliant testing.

process cycle test: A black box test design technique in which test cases are designed to execute business procedures and processes. [TMap] See also procedure testing.

process improvement: A program of activities designed to improve the performance and maturity of the organization’s processes, and the result of such a program. [CMMI]

EITP

process model: A framework wherein processes of the same nature are classified into a overall model, e.g. a test improvement model.

EITP

product-based quality: A view of quality, wherein quality is based on a well-defined set of quality attributes. These attributes must be measured in an objective and quantitative way. Differences in the quality of products of the same type can be traced back to the way the specific quality attributes have been implemented. [After Garvin] See also manufacturing-based quality, quality attribute, transcendent-based quality, user-based quality, value-based quality.

F

ATM

ATA

ATT

product risk: A risk directly related to the test object. See also risk.

production acceptance testing: See operational acceptance testing.

program instrumenter: See instrumenter.

program testing: See component testing.

project: A project is a unique set of coordinated and controlled activities with start and finish dates undertaken to achieve an objective conforming to specific requirements, including the constraints of time, cost and resources. [ISO 9000]

EITP

project retrospective: A structured way to capture lessons learned and to create specific action plans for improving on the next project or next project phase.

F

ATM

project risk: A risk related to management and control of the (test) project, e.g. lack of staffing, strict deadlines, changing requirements, etc. See also risk.

project test plan: See master test plan.

pseudo-random: A series which appears to be random but is in fact generated according to some prearranged sequence.

Q

qualification: The process of demonstrating the ability to fulfill specified requirements. Note the term ‘qualified’ is used to designate the corresponding status. [ISO 9000]

F

quality: The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations. [After IEEE 610]

quality assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled. [ISO 9000]

quality attribute: A feature or characteristic that affects an item’s quality. [IEEE 610]

quality characteristic: See quality attribute.

quality control: The operational techniques and activities, part of quality management, that are focused on fulfilling quality requirements. [after ISO 8402]

quality gate: A special milestone in a project. Quality gates are located between those phases of a project strongly depending on the outcome of a previous phase. A quality gate includes a formal check of the documents of the previous phase.

quality management: Coordinated activities to direct and control an organization with regard to quality. Direction and control with regard to quality generally includes the establishment of the quality policy and quality objectives, quality planning, quality control, quality assurance and quality improvement. [ISO 9000]

ATM

quality risk: A risk related to a quality attribute. See also quality attribute, product risk.

R

ETM

RACI matrix: A matrix describing the participation by various roles in completing tasks or deliverables for a project or process. It is especially useful in clarifying roles and responsibilities. RACI is an acronym derived from the four key responsibilities most typically used: Responsible, Accountable, Consulted, and Informed.

random testing: A black box test design technique where test cases are selected, possibly using a pseudo-random generation algorithm, to match an operational profile. This technique can be used for testing non-functional attributes such as reliability and performance.

EITP

Rational Unified Process: A proprietary adaptable iterative software development process framework consisting of four project lifecycle phases: inception, elaboration, construction and transition.

ETM

reactive testing: Testing that dynamically responds to the actual system under test and test results being obtained. Typically reactive testing has a reduced planning cycle and the design and implementation test phases are not carried out until the test object is received.

recorder: See scribe.

ATT

record/playback tool: See capture/playback tool.

recoverability: The capability of the software product to re-establish a specified level of performance and recover the data directly affected in case of failure. [ISO 9126] See also reliability.

ATT

recoverability testing: The process of testing to determine the recoverability of a software product. See also reliability testing.

recovery testing: See recoverability testing.

ETM

regression-averse testing: Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels.

F

regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed.

regulation testing: See compliance testing.

release note: A document identifying test items, their configuration, current status and other delivery information delivered by development to testing, and possibly other stakeholders, at the start of a test execution phase. [After IEEE 829]

reliability: The ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations. [ISO 9126]

ATT

reliability growth model: A model that shows the growth in reliability over time during continuous testing of a component or system as a result of the removal of defects that result in reliability failures.

F

ATT

reliability testing: The process of testing to determine the reliability of a software product.

ATT

replaceability: The capability of the software product to be used in place of another specified software product for the same purpose in the same environment. [ISO 9126] See also portability.

F

requirement: A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. [After IEEE 610]

ATA

requirements-based testing: An approach to testing in which test cases are designed based on test objectives and test conditions derived from requirements, e.g. tests that exercise specific functions or probe non-functional attributes such as reliability or usability.

F

requirements management tool: A tool that supports the recording of requirements, requirements attributes (e.g. priority, knowledge responsible) and annotation, and facilitates traceability through layers of requirements and requirements change management. Some requirements management tools also provide facilities for static analysis, such as consistency checking and violations to pre-defined requirements rules.

requirements phase: The period of time in the software lifecycle during which the requirements for a software product are defined and documented. [IEEE 610]

resource utilization: The capability of the software product to use appropriate amounts and types of resources, for example the amounts of main and secondary memory used by the program and the sizes of required temporary or overflow files, when the software performs its function under stated conditions. [After ISO 9126] See also efficiency.

ATT

resource utilization testing: The process of testing to determine the resource-utilization of a software product. See also efficiency testing.

result: The consequence/outcome of the execution of a test. It includes outputs to screens, changes to data, reports, and communication messages sent out. See also actual result, expected result.

resumption criteria: The criteria used to restart all or a portion of the testing activities that were suspended previously.

resumption requirements: The defined set of testing activities that must be repeated when testing is re-started after a suspension. [After IEEE 829]

F

re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions.

EITP

retrospective meeting: A meeting at the end of a project during which the project team members evaluate the project and learn lessons that can be applied to the next project.

F

ATM

review: An evaluation of a product or project status to ascertain discrepancies from planned results and to recommend improvements. Examples include management review, informal review, technical review, inspection, and walkthrough. [After IEEE 1028]

ATM

review plan: A document describing the approach, resources and schedule of intended review activities. It identifies, amongst others: documents and code to be reviewed, review types to be used, participants, as well as entry and exit criteria to be applied in case of formal reviews, and the rationale for their choice. It is a record of the review planning process.

F

review tool: A tool that provides support to the review process. Typical features include review planning and tracking support, communication support, collaborative reviews and a repository for collecting and reporting of metrics.

F

ATM

reviewer: The person involved in the review that identifies and describes anomalies in the product or project under review. Reviewers can be chosen to represent different viewpoints and roles in the review process.

F

ATM

risk: A factor that could result in future negative consequences; usually expressed as impact and likelihood.

ATM

ATA

ATT

risk analysis: The process of assessing identified risks to estimate their impact and probability of occurrence (likelihood).

ATM

ATT

risk assessment: The process of assessing a given project or product risk to determine its level of risk, typically by assigning likelihood and impact ratings and then aggregating those ratings into a single risk priority rating. See also product risk, project risk, risk, risk impact, risk level, risk likelihood.

F

ATM

ATA

ATT

risk-based testing: An approach to testing to reduce the level of product risks and inform stakeholders of their status, starting in the initial stages of a project. It involves the identification of product risks and the use of risk levels to guide the test process.

risk category: See risk type.

ATM

ATA

ATT

risk control: The process through which decisions are reached and protective measures are implemented for reducing risks to, or maintaining risks within, specified levels.

ATM

ATA

risk identification: The process of identifying risks using techniques such as brainstorming, checklists and failure history. ATT

risk impact: The damage that will be caused if the risk become an actual outcome or event.

ATM

ATA

ATT

risk level: The importance of a risk as defined by its characteristics impact and likelihood. The level of risk can be used to determine the intensity of testing to be performed. A risk level can be expressed either qualitatively (e.g. high, medium, low) or quantitatively.

risk likelihood: The estimated probability that a risk will become an actual outcome or event.

ATM

ATA

risk management: Systematic application of procedures and practices to the tasks of identifying, analyzing, prioritizing, and controlling risk.

ATM

ATA

ATT

risk mitigation: See risk control.

risk type: A set of risks grouped by one or more common factors such as a quality attribute, cause, location, or potential effect of risk;. A specific set of product risk types is related to the type of testing that can mitigate (control) that risk type. For example the risk of user-interactions being misunderstood can be mitigated by usability testing.

ATT

robustness: The degree to which a component or system can function correctly in the presence of invalid inputs or stressful environmental conditions. [IEEE 610] See also error-tolerance, fault-tolerance.

F

robustness testing: Testing to determine the robustness of the software product.

ATM

root cause: A source of a defect such that if it is removed, the occurrence of the defect type is decreased or removed. [CMMI]

ATA

root cause analysis: An analysis technique aimed at identifying the root causes of defects. By directing corrective measures at root causes, it is hoped that the likelihood of defect recurrence will be minimized.

EITP

RUP: See Rational Unified Process.

S

safety: The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use. [ISO 9126]

safety critical system: A system whose failure or malfunction may result in death or serious injury to people, or loss or severe damage to equipment, or environmental harm.

safety testing: Testing to determine the safety of a software product.

sanity test: See smoke test.

scalability: The capability of the software product to be upgraded to accommodate increased loads. [After Gerrard]

scalability testing: Testing to determine the scalability of the software product.

scenario testing: See use case testing.

scorecard: A representation of summarized performance measurements representing progress towards the implementation of long-term goals. A scorecard provides static measurements of performance over or at the end of a defined interval. See also balanced scorecard, dashboard.

F

scribe: The person who records each defect mentioned and any suggestions for process improvement during a review meeting, on a logging form. The scribe should ensure that the logging form is readable and understandable.

scripted testing: Test execution carried out by following a previously documented sequence of tests.

F

scripting language: A programming language in which executable test scripts are written, used by a test execution tool (e.g. a capture/playback tool).

EITP

SCRUM: An iterative incremental framework for managing projects commonly used with agile software development. See also agile software development.

security: Attributes of software products that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs and data. [ISO 9126] See also functionality.

F

ATT

security testing: Testing to determine the security of the software product. See also functionality testing.

security testing tool: A tool that provides support for testing security characteristics and vulnerabilities.

F

security tool: A tool that supports operational security.

serviceability testing: See maintainability testing.

session-based test management: A method for measuring and managing session-based testing, e.g. exploratory testing.

session-based testing: An approach to testing in which test activities are planned as uninterrupted sessions of test design and execution, often used in conjunction with exploratory testing.

ATM

severity: The degree of impact that a defect has on the development or operation of a component or system. [After IEEE 610]

ETM

Shewhart chart: See control chart.

ATT

short-circuiting: A programming language/interpreter technique for evaluating compound conditions in which a condition on one side of a logical operator may not be evaluated if the condition on the other side is sufficient to determine the final outcome.

simulation: The representation of selected behavioral characteristics of one physical or abstract system by another system. [ISO 2382/1]

simulator: A device, computer program or system used during testing, which behaves or operates like a given system when provided with a set of controlled inputs. [After IEEE 610, DO178b] See also emulator.

site acceptance testing: Acceptance testing by users/customers at their site, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes, normally including hardware as well as software.

ETM

S.M.A.R.T. goal methodology: A methodology whereby objectives are defined very specifically rather than generically. SMART is an acronym derived from the attributes of the objective to be defined: Specific, Measurable, Attainable, Relevant and Timely.

smoke test: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertaining that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. See also intake test.

software: Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system. [IEEE 610]

software attack: See attack.

Software Failure Mode and Effect Analysis (SFMEA): See Failure Mode and Effect Analysis (FMEA).

Software Failure Mode, Effects, and Criticality Analysis (SFMECA): See Failure Mode, Effects, and Criticality Analysis (FMECA).

Software Fault Tree Analysis (SFTA): See Fault Tree Analysis (FTA).

software feature: See feature.

software integrity level: The degree to which software complies or must comply with a set of stakeholder-selected software and/or software-based system characteristics (e.g., software complexity, risk assessment, safety level, security level, desired performance, reliability, or cost) which are defined to reflect the importance of the software to its stakeholders.

EITP

software lifecycle: The period of time that begins when a software product is conceived and ends when the software is no longer available for use. The software lifecycle typically includes a concept phase, requirements phase, design phase, implementation phase, test phase, installation and checkout phase, operation and maintenance phase, and sometimes, retirement phase. Note these phases may overlap or be performed iteratively.

EITP

Software Process Improvement: A program of activities designed to improve the performance and maturity of the organization’s software processes and the results of such a program. [After CMMI]

software product characteristic: See quality attribute.

software quality: The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs. [After ISO 9126] See also quality.

software quality characteristic: See quality attribute.

software test incident: See incident.

software test incident report: See incident report.

ATA

Software Usability Measurement Inventory (SUMI): A questionnaire-based usability test technique for measuring software quality from the end user’s point of view. [Veenendaal04]

source statement: See statement.

specification: A document that specifies, ideally in a complete, precise and verifiable manner, the requirements, design, behavior, or other characteristics of a component or system, and, often, the procedures for determining whether these provisions have been satisfied. [After IEEE 610]

specification-based testing: See black box testing.

ATA

specification-based technique: See black box test design technique.

specification-based test design technique: See black box test design technique.

specified input: An input for which the specification predicts a result.

EITP

SPI: See Software Process Improvement.

ATT

stability: The capability of the software product to avoid unexpected effects from modifications in the software. [ISO 9126] See also maintainability.

EITP

staged representation: A model structure wherein attaining the goals of a set of process areas establishes a maturity level; each level builds a foundation for subsequent levels. [CMMI]

EITP

standard: Formal, possibly mandatory, set of requirements developed and used to prescribe consistent approaches to the way of working or to provide guidelines (e.g., ISO/IEC standards, IEEE standards, and organizational standards). [After CMMI]

ETM

standard-compliant testing: Testing that complies to a set of requirements defined by a standard, e.g., an industry testing standard or a standard for testing safety-critical systems. See also process-compliant testing.

standard software: See off-the-shelf software.

standards testing: See compliance testing.

state diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610]

state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions.

state transition: A transition between two states of a component or system.

F

ATA

state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. See also N-switch testing.

statement: An entity in a programming language, which is typically the smallest indivisible unit of execution.

F

statement coverage: The percentage of executable statements that have been exercised by a test suite.

ATT

statement testing: A white box test design technique in which test cases are designed to execute statements.

F

ATT

static analysis: Analysis of software development artifacts, e.g. requirements or code, carried out without execution of these software development artifacts. Static analysis is usually carried out by means of a supporting tool.

F

static analysis tool: See static analyzer.

ATT

static analyzer: A tool that carries out static analysis.

static code analysis: Analysis of source code carried out without execution of that software.

static code analyzer: A tool that carries out static code analysis. The tool checks source code, for certain properties such as conformance to coding standards, quality metrics or data flow anomalies.

F

static testing: Testing of a software development artifact, e.g., requirements, design or code, without execution of these artifacts, e.g., reviews or static analysis.

statistical testing: A test design technique in which a model of the statistical distribution of the input is used to construct representative test cases. See also operational profile testing.

status accounting: An element of configuration management, consisting of the recording and reporting of information needed to manage a configuration effectively. This information includes a listing of the approved configuration identification, the status of proposed changes to the configuration, and the implementation status of the approved changes. [IEEE 610]

ATM

EITP

STEP: See Systematic Test and Evaluation Process.

storage: See resource utilization.

storage testing: See resource utilization testing.

F

stress testing: A type of performance testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers. [After IEEE 610] See also performance testing, load testing.

F

stress testing tool: A tool that supports stress testing.

structural coverage: Coverage measures based on the internal structure of a component or system.

structural test design technique: See white-box test design technique.

F

structural testing: See white-box testing.

structure-based test design technique: See white-box test design technique.

ATT

structure-based technique: See white-box test design technique.

F

structure-based testing: See white-box testing.

structured walkthrough: See walkthrough.

F

stub: A skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component. [After IEEE 610]

subpath: A sequence of executable statements within a component.

suitability: The capability of the software product to provide an appropriate set of functions for specified tasks and user objectives. [ISO 9126] See also functionality.

ATA

suitability testing: The process of testing to determine the suitability of a software product

ATA

SUMI: See Software Usability Measurement Inventory.

suspension criteria: The criteria used to (temporarily) stop all or a portion of the testing activities on the test items. [After IEEE 829]

syntax testing: A black box test design technique in which test cases are designed based upon the definition of the input domain and/or output domain.

system: A collection of components organized to accomplish a specific function or set of functions. [IEEE 610]

system integration testing: Testing the integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet).

system of systems: Multiple heterogeneous, distributed systems that are embedded in networks at multiple levels and in multiple interconnected domains, addressing large-scale inter-disciplinary common problems and purposes, usually without a common management structure.

F

system testing: The process of testing an integrated system to verify that it meets specified requirements. [Hetzel]

ATM

EITP

Systematic Test and Evaluation Process: A structured testing methodology, also used as a content-based model for improving the testing process. Systematic Test and Evaluation Process (STEP) does not require that improvements occur in a specific order. See also content-based model.

T

F

ATM

technical review: A peer group discussion activity that focuses on achieving consensus on the technical approach to be taken. [Gilb and Graham], [IEEE 1028] See also peer review.

test: A set of one or more test cases. [IEEE 829]

F

ATM

test approach: The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) project’s goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed

ETM

test architect : (1) A person who provides guidance and strategic direction for a test organization and for its relationship with other disciplines.
(2) A person who defines the way testing is structured for a given system, including topics such as test tools and test data management.

test automation: The use of software to perform or support test activities, e.g. test management, test design, test execution and results checking.

F

test basis: All documents from which the requirements of a component or system can be inferred. The documentation on which the test cases are based. If a document can be amended only by way of formal amendment procedure, then the test basis is called a frozen test basis. [After TMap]

test bed: See test environment.

F

ATM

test case: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

test case design technique: See test design technique.

F

test case specification: A document specifying a set of test cases (objective, inputs, test actions, expected results, and execution preconditions) for a test item. [After IEEE 829] See also test specification.

test case suite: See test suite.

ATA

test charter: A statement of test objectives, and possibly test ideas about how to test. Test charters are used in exploratory testing. See also exploratory testing.

ATM

test closure: During the test closure phase of a test process data is collected from completed activities to consolidate experience, testware, facts and numbers. The test closure phase consists of finalizing and archiving the testware and evaluating the test process, including preparation of a test evaluation report. See also test process.

F

test comparator: A test tool to perform automated test comparison of actual results with expected results.

test comparison: The process of identifying differences between the actual results produced by the component or system under test and the expected results for a test. Test comparison can be performed during test execution (dynamic comparison) or after test execution.

test completion criteria: See exit criteria.

F

ATM

test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.

F

ATM

ATA

test control: A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned. See also test management.

F

test coverage: See coverage.

test cycle: Execution of the test process against a single identifiable release of the test object.

F

test data: Data that exists (for example, in a database) before a test is executed, and that affects or is affected by the component or system under test.

F

ATA

test data preparation tool: A type of test tool that enables data to be selected from existing databases or created, generated, manipulated and edited for use in testing.

test data management: The process of analyzing test data requirements, designing test data structures, creating and maintaining test data.

test deliverable: Any test (work) product that must be delivered to someone other than the test (work) product’s author. See also deliverable.

F

ATM

ATA

test design: (1) See test design specification.
(2) The process of transforming general testing objectives into tangible test conditions and test cases.

test design specification: A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases. [After IEEE 829] See also test specification.

F

test design technique: Procedure used to derive and/or select test cases.

F

ATA

test design tool: A tool that supports the test design activity by generating test inputs from a specification that may be held in a CASE tool repository, e.g. requirements management tool, from specified test conditions held in the tool itself, or from code.

ATM

test director: A senior manager who manages test managers. See also test manager.

F

ETM

test driven development: A way of developing software where the test cases are developed, and often automated, before the software is developed to run those test cases.

test driver: See driver.

F

test environment: An environment containing hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test. [After IEEE 610]

ATM

test estimation: The calculated approximation of a result related to various aspects of testing (e.g. effort spent, completion date, costs involved, number of test cases, etc.) which is usable even if input data may be incomplete, uncertain, or noisy.

test evaluation report: A document produced at the end of the test process summarizing all testing activities and results. It also contains an evaluation of the test process and lessons learned.

F

ATM

ATA

test execution: The process of running a test on the component or system under test, producing actual result(s).

test execution automation: The use of software, e.g. capture/playback tools, to control the execution of tests, the comparison of actual results to expected results, the setting up of test preconditions, and other test control and reporting functions.

test execution phase: The period of time in a software development lifecycle during which the components of a software product are executed, and the software product is evaluated to determine whether or not requirements have been satisfied. [IEEE 610]

F

test execution schedule: A scheme for the execution of test procedures. Note: The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed.

test execution technique: The method used to perform the actual test execution, either manual or automated.

F

ATA

ATT

test execution tool: A type of test tool that is able to execute other software using an automated test script, e.g. capture/playback. [Fewster and Graham]

test fail: See fail.

test generator: See test data preparation tool.

F

test harness: A test environment comprised of stubs and drivers needed to execute a test.

ATM

ATA

test implementation: The process of developing and prioritizing test procedures, creating test data and, optionally, preparing test harnesses and writing automated test scripts.

EITP

test improvement plan: A plan for achieving organizational test process improvement objectives based on a thorough understanding of the current strengths and weaknesses of the organization’s test processes and test process assets. [After CMMI]

test incident: See incident.

test incident report: See incident report.

test infrastructure: The organizational artifacts needed to perform testing, consisting of test environments, test tools, office environment and procedures.

test input: The data received from an external source by the test object during test execution. The external source can be hardware, software or human.

test item: The individual element to be tested. There usually is one test object and many test items. See also test object.

test item transmittal report: See release note.

F

test leader: See test manager.

F

ATM

test level: A group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels are component test, integration test, system test and acceptance test. [After TMap]

F

ATM

test log: A chronological record of relevant details about the execution of tests. [IEEE 829]

test logging: The process of recording information about tests executed into a test log.

ATM

test management: The planning, estimating, monitoring and control of test activities, typically carried out by a test manager.

F

ATT

test management tool: A tool that provides support to the test management and control part of a test process. It often has several capabilities, such as testware management, scheduling of tests, the logging of results, progress tracking, incident management and test reporting.

F

test manager: The person responsible for project management of testing activities and resources, and evaluation of a test object. The individual who directs, controls, administers, plans and regulates the evaluation of a test object.

ATM

EITP

Test Maturity Model integration: A five level staged framework for test process improvement, related to the Capability Maturity Model Integration (CMMI), that describes the key elements of an effective test process.

ETM

test mission: The purpose of testing for an organization, often documented as part of the test policy. See also test policy.

F

ATM

ATA

test monitoring: A test management task that deals with the activities related to periodically checking the status of a test project. Reports are prepared that compare the actuals to that which was planned. See also test management.

test object: The component or system to be tested. See also test item.

F

test objective: A reason or purpose for designing and executing a test.

test oracle: A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code. [After Adrion]

test outcome: See result.

test pass: See pass.

test performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive test development, e.g. Defect Detection Percentage (DDP).

test phase: A distinct set of test activities collected into a manageable phase of a project, e.g. the execution activities of a test level. [After Gerrard]

F

ATM

test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process. [After IEEE 829]

ATM

test planning: The activity of establishing or updating a test plan.

ATA

Test Point Analysis (TPA): A formula based test estimation method based on function point analysis. [TMap]

F

ATM

EITP

ETM

test policy: A high level document describing the principles, approach and major objectives of the organization regarding testing.

F

ATM

test procedure: See test procedure specification.

F

ATM

test procedure specification: A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829] See also test specification.

test process: The fundamental test process comprises test planning and control, test analysis and design, test implementation and execution, evaluating exit criteria and reporting, and test closure activities.

EITP

Test Process Group: A collection of (test) specialists who facilitate the definition, maintenance, and improvement of the test processes used by an organization. [After CMMI]

EITP

test process improvement manifesto: A statement that echoes the agile manifesto, and defines values for improving the testing process. The values are:

- flexibility over detailed processes

- best practices over templates

- deployment orientation over process orientation

- peer reviews over quality assurance (departments)

- business driven over model driven. [Veenendaal08]

EITP

test process improver: A person implementing improvements in the test process based on a test improvement plan.

test progress report: A document summarizing testing activities and results, produced at regular intervals, to report progress of testing activities against a baseline (such as the original test plan) and to communicate risks and alternatives requiring a decision to management.

test record: See test log.

test recording: See test logging.

test report: See test summary report and test progress report.

test reproducibility: An attribute of a test indicating whether the same results are produced each time the test is executed.

test requirement: See test condition.

test result: See result.

test rig: See test environment.

test run: Execution of a test on a specific version of the test object.

test run log: See test log.

test scenario: See test procedure specification.

test schedule: A list of activities, tasks or events of the test process, identifying their intended start and finish dates and/or times, and interdependencies.

F

ATM

test script: Commonly used to refer to a test procedure specification, especially an automated one.

test session: An uninterrupted period of time spent in executing tests. In exploratory testing, each test session is focused on a charter, but testers can also explore new opportunities or issues during a session. The tester creates and executes test cases on the fly and records their progress. See also exploratory testing.

test set: See test suite.

test situation: See test condition.

test specification: A document that consists of a test design specification, test case specification and/or test procedure specification.

test specification technique: See test design technique.

test stage: See test level.

F

ATM

ATA

ETM

test strategy: A high-level description of the test levels to be performed and the testing within those levels for an organization or programme (one or more projects).

F

test suite: A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one.

F

ATM

test summary report: A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items against exit criteria. [After IEEE 829]

test target: A set of exit criteria.

test technique: See test design technique.

EITP

test tool: A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis. [TMap] See also CAST.

test type: A group of test activities aimed at testing a component or system focused on a specific test objective, i.e. functional test, usability test, regression test etc. A test type may take place on one or more test levels or test phases. [After TMap]

ATT

testability: The capability of the software product to enable modified software to be tested. [ISO 9126] See also maintainability.

testability review: A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. [After TMap]

testable requirement: A requirements that is stated in terms that permit establishment of test designs (and subsequently test cases) and execution of tests to determine whether the requirement has been met. [After IEEE 610]

F

tester: A skilled professional who is involved in the testing of a component or system.

F

testing: The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.

F

testware: Artifacts produced during the test process required to plan, design, and execute tests, such as documentation, scripts, inputs, expected results, set-up and clear-up procedures, files, databases, environment, and any additional software or utilities used in testing. [After Fewster and Graham]

thread testing: An approach to component integration testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by levels of a hierarchy.

three point estimation: A test estimation method using estimated values for the “best case”, “worst case”, and “most likely case” of the matter being estimated, to define the degree of certainty associated with the resultant estimate.

time behavior: See performance.

ATM

EITP

TMMi: See Test Maturity Model integration.

top-down testing: An incremental approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested. See also integration testing.

EITP

Total Quality Management: An organization-wide management approach centered on quality, based on the participation of all members of the organization and aiming at long-term success through customer satisfaction, and benefits to all members of the organization and to society. Total Quality Management consists of planning, organizing, directing, control, and assurance. [After ISO 8402]

ATM

EITP

TPI Next: A continuous business-driven framework for test process improvement that describes the key elements of an effective and efficient test process.

EITP

TPG: See Test Process Group.

EITP

TQM: See Total Quality Management.

F

traceability: The ability to identify related items in documentation and software, such as requirements with associated tests. See also horizontal traceability, vertical traceability.

EITP

transactional analysis: The analysis of transactions between people and within people’s minds; a transaction is defined as a stimulus plus a response. Transactions take place between people and between the ego states (personality segments) within one person’s mind.

EITP

transcendent-based quality: A view of quality, wherein quality cannot be precisely defined, but we know it when we see it, or are aware of its absence when it is missing. Quality depends on the perception and affective feelings of an individual or group of individuals towards a product. [After Garvin] See also manufacturing-based quality, product-based quality, user-based quality, value-based quality.

U

ATA

understandability: The capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use. [ISO 9126] See also usability.

unit: See component.

F

unit test framework: A tool that provides an environment for unit or component testing in which a component can be tested in isolation or with suitable stubs and drivers. It also provides other support for the developer, such as debugging capabilities. [Graham]

unit testing: See component testing.

unreachable code: Code that cannot be reached and therefore is impossible to execute.

usability: The capability of the software to be understood, learned, used and attractive to the user when used under specified conditions. [ISO 9126]

F

ATA

usability testing: Testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions. [After ISO 9126]

use case: A sequence of transactions in a dialogue between an actor and a component or system with a tangible result, where an actor can be a user or anything that can exchange information with the system.

F

ATA

use case testing: A black box test design technique in which test cases are designed to execute scenarios of use cases.

F

user acceptance testing: See acceptance testing.

EITP

user-based quality: A view of quality, wherein quality is the capacity to satisfy needs, wants and desires of the user(s). A product or service that does not fulfill user needs is unlikely to find any users. This is a context dependent, contingent approach to quality since different business characteristics require different qualities of a product. [after Garvin] See also manufacturing-based quality, product-based quality, transcendent-based quality, value-based quality.

user scenario testing: See use case testing

user story: A high-level user or business requirement commonly used in agile software development, typically consisting of one or more sentences in the everyday or business language capturing what functionality a user needs, any non-functional criteria, and also includes acceptance criteria. See also agile software development, requirement.

ATA

user story testing: A black box test design technique in which test cases are designed based on user stories to verify their correct implementation. See also user story.

user test: A test whereby real-life users are involved to evaluate the usability of a component or system.

V

F

V-model: A framework to describe the software development lifecycle activities from requirements specification to maintenance. The V-model illustrates how testing activities can be integrated into each phase of the software development lifecycle.

F

validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]

EITP

value-based quality: A view of quality, wherein quality is defined by price. A quality product or service is one that provides desired performance at an acceptable cost. Quality is determined by means of a decision process with stakeholders on trade-offs between time, effort and cost aspects. [After Garvin] See also manufacturing-based quality, product-based quality, transcendent-based quality, user-based quality.

variable: An element of storage in a computer that is accessible by a software program by referring to it by a name.

F

verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. [ISO 9000]

F

version control: See configuration control.

vertical traceability: The tracing of requirements through the layers of development documentation to components.

volume testing: Testing where the system is subjected to large volumes of data. See also resource-utilization testing.

W

F

ATM

walkthrough: A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content. [Freedman and Weinberg, IEEE 1028] See also peer review.

ATA

WAMMI: See Website Analysis and MeasureMent Inventory.

WBS: See Work Breakdown Structure.

ATA

Website Analysis and MeasureMent Inventory (WAMMI): A questionnaire-based usability test technique for measuring web site software quality from the end user’s point of view.

white-box technique: See white-box test design technique.

F

ATT

white-box test design technique: Procedure to derive and/or select test cases based on an analysis of the internal structure of a component or system.

F

white-box testing: Testing based on an analysis of the internal structure of the component or system.

ATM

Wide Band Delphi: An expert based test estimation technique that aims at making an accurate estimation using the collective wisdom of the team members.

ATT

wild pointer: A pointer that references a location that is out of scope for that pointer or that does not exist. See also pointer.

Work Breakdown Structure: An arrangement of work elements and their relationship to each other and to the end product. [CMMI]

Annex A References

Standards

[DO-178b] DO-178B:1992. Software Considerations in Airborne Systems and Equipment Certification, Requirements and Technical Concepts for Aviation (RTCA SC167).

[IEEE 610] IEEE 610.12:1990. Standard Glossary of Software Engineering Terminology.

[IEEE 829] IEEE 829:1998. Standard for Software Test Documentation.

[IEEE 1008] IEEE 1008:1993. Standard for Software Unit Testing.

[IEEE 1028] IEEE 1028:1997. Standard for Software Reviews and Audits.

[IEEE 1044] IEEE 1044:1993. Standard Classification for Software Anomalies.

[IEEE 1219] IEEE 1219:1998. Software Maintenance.

[ISO 2382/1] ISO/IEC 2382-1:1993. Data processing - Vocabulary - Part 1: Fundamental terms.

[ISO 9000] ISO 9000:2005. Quality Management Systems - Fundamentals and Vocabulary.

[ISO 9126] ISO/IEC 9126-1:2001. Software Engineering – Software Product Quality – Part 1: Quality characteristics and sub-characteristics.

[ISO 12207] ISO/IEC 12207:1995. Information Technology – Software Lifecycle Processes.

[ISO 14598] ISO/IEC 14598-1:1999. Information Technology – Software Product Evaluation - Part 1: General Overview.

[ISO 15504] ISO/IEC 15504-9: 1998. Information Technology – Software Process Assessment – Part 9: Vocabulary

Books and papers

[Abbott] J. Abbot (1986), Software Testing Techniques, NCC Publications.

[Adrion] W. Adrion, M. Branstad and J. Cherniabsky (1982), Validation, Verification and Testing of Computer Software, in: Computing Surveys, Vol. 14, No 2, June 1982.

[Bach] J. Bach (2004), Exploratory Testing, in: E. van Veenendaal, The Testing Practitioner – 2nd edition, UTN Publishing, ISBN 90-72194-65-9.

[Beizer] B. Beizer (1990), Software Testing Techniques, van Nostrand Reinhold, ISBN 0-442-20672-0

[Chow] T. Chow (1978), Testing Software Design Modelled by Finite-Sate Machines, in: IEEE Transactions on Software Engineering, Vol. 4, No 3, May 1978.

[CMM] M. Paulk, C. Weber, B. Curtis and M.B. Chrissis (1995), The Capability Maturity Model, Guidelines for Improving the Software Process, Addison-Wesley, ISBN 0-201-54664-7

[CMMI] M.B. Chrissis, M. Konrad and S. Shrum (2004), CMMI, Guidelines for Process Integration and Product Improvement, Addison Wesley, ISBN 0-321-15496-7

[Deming] D. W. Edwards (1986), Out of the Crisis, MIT Center for Advanced Engineering Study, ISBN 0-911379-01-0

[Fenton] N. Fenton (1991), Software Metrics: a Rigorous Approach, Chapman & Hall, ISBN 0-53249-425-1

[Fewster and Graham] M. Fewster and D. Graham (1999), Software Test Automation, Effective use of test execution tools, Addison-Wesley, ISBN 0-201-33140-3.

[Freedman and Weinberg] D. Freedman and G. Weinberg (1990), Walkthroughs, Inspections, and Technical Reviews, Dorset House Publishing, ISBN 0-932633-19-6.

[Garvin] D.A. Garvin (1984), What does product quality really mean?, in: Sloan Management Review, Vol. 26, nr. 1 1984

[Gerrard] P. Gerrard and N. Thompson (2002), Risk-Based E-Business Testing, Artech House Publishers, ISBN 1-58053-314-0.

[Gilb and Graham] T. Gilb and D. Graham (1993), Software Inspection, Addison-Wesley, ISBN 0-201-63181-4.

[Graham] D. Graham, E. van Veenendaal, I. Evans and R. Black (2007), Foundations of Software Testing, Thomson Learning, ISBN 978-1-84480-355-2

[Grochtmann] M. Grochtmann (1994), Test Case Design Using Classification Trees, in: Conference Proceedings STAR 1994.

[Hetzel] W. Hetzel (1988), The complete guide to software testing – 2nd edition, QED Information Sciences, ISBN 0-89435-242-3.

[Juran] J.M. Juran (1979), Quality Control Handbook, McGraw-Hill

[McCabe] T. McCabe (1976), A complexity measure, in: IEEE Transactions on Software Engineering, Vol. 2, pp. 308-320.

[Musa] J. Musa (1998), Software Reliability Engineering Testing, McGraw-Hill Education, ISBN 0-07913-271-5.

[Myers] G. Myers (1979), The Art of Software Testing, Wiley, ISBN 0-471-04328-1.

[TMap] M. Pol, R. Teunissen, E. van Veenendaal (2002), Software Testing, A guide to the TMap Approach, Addison Wesley, ISBN 0-201-745712.

[TMMi] E. van Veenendaal and J. Cannegieter (2011), The Little TMMi, UTN Publishing, ISBN 97-89490986-03-2

[Veenendaal04] E. van Veenendaal (2004), The Testing Practitioner – 2nd edition, UTN Publishing, ISBN 90-72194-65-9.

[Veenendaal08] E. van Veenendaal (2008), Test Improvement Manifesto, in: Testing Experience, Issue 04/08, December 2008