Certified Tester

Advanced Level Overview

Version 2012

International Software Testing Qualifications Board

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®).

Advanced Level Working Group: Mike Smith (Chair), Bernard Homès (Vice Chair), Syllabus Coordinators Graham Bath, Rex Black, Judy McKay 2012

Digital book(s) (epub and mobi) produced by Booknook.biz under the direction of Rex Black and Lois Kostroski of the ASTQB.

Revision History

Version

Date

Remarks

0.1

October 4th 2011

Initial version:

0.2

October 5th 2011

TA edits from Judy McKay

0.3

October 19th 2011

WG review comments

0.4

14 DEC 2011

Incorporation of NB review comments

0.5

16 DEC 2011

Incorporation of review comments in San Francisco

Alpha

17 FEB 2012

Updates for Alpha release

Beta

6 APR 2012

Incorporation of comments from NBs received on time from Alpha release.

Beta

8 June 2012

Beta release for NB review after copy edit

Release Candidate

28 JULY 2012

Addition of explanation regarding standards and ISTQB web site additions

GA

19 OCT 2012

General availability release

Table of Contents

Revision History

Acknowlegements

E-book Search Instructions

1.    Introduction to the Advanced Level

1.1    Career Paths for Testers

1.2    Intended Audience

1.3    Learning Objectives

1.4    Entry Requirements

1.5    Structure and Course Duration

1.6    Handling of Standards

1.7    Keeping It Current

2.    Overview of Advanced Level Syllabi

2.1    Advanced Level: Test Manager (TM)

2.1.1     Business Outcomes

2.1.2     Content

2.2    Advanced Level: Test Analyst (TA)

2.2.1     Business Outcomes

2.2.2     Content

2.3    Advanced Level: Technical Test Analyst (TTA)

2.3.1     Business Outcomes

2.3.2     Content

3.    Appendix: Main Changes in the 2012 Syllabi

4.    Abbreviations

5.    References

5.1    Trademarks

5.2    Documents and Web Sites

Acknowledgements

This document was produced by a core team from the International Software Testing Qualifications Board Advanced Level Working Group - Graham Bath, Mike Smith, Rex Black, Judy McKay; 2010-2012.

The core team thanks the review team and all National Boards for their suggestions and input.

At the time the Advanced Level Syllabus was completed the Advanced Level Working Group had the following membership (alphabetical order):

Graham Bath, Rex Black, Maria Clara Choucair, Debra Friedenberg, Bernard Homès (Vice Chair), Paul Jorgensen, Judy McKay, Jamie Mitchell, Thomas Mueller, Klaus Olsen, Kenji Onishi, Meile Posthuma, Eric Riou du Cosquer, Jan Sabak, Hans Schaefer, Mike Smith (Chair), Geoff Thompson, Erik van Veenendaal, Tsuyoshi Yumoto.

The following persons participated in the reviewing, commenting and balloting of this document (alphabetical order):

Rex Black, Thomas Donner, Bernard Homès, Judy McKay, Rasluca Popescu, Jan Sabak, Mike Smith

This document was formally approved for release by the General Assembly of ISTQB® on October 19th, 2012.

E-book Search Instructions

Search Instructions

For the purposes of facilitating reference and review of the important terms used throughout, this page will instruct you in how to use the various search functions on the leading e-reading devices so that you may find Glossary definitions with ease.

How To Search in Kindle e-ink Devices

On a Kindle e-ink device, to look up a word in the dictionary, use the 5-way key to navigate to the specific word you wish to search.

1. Simply highlight the word, and the dictionary definition will appear at the bottom of the Kindle screen.

2. For a more wide-ranging search, start typing the word you wish to look up while the book is open on the screen.

3. When you’ve finished typing the word in the rounded box that will pop up, use the 5-way controller to move to the right. When the “find” button turns black, simply click it and it will search within the current book.

4. For more options, move to the right one more time. You’ll then see 5 options: “my items,” “store,” “google,” “wikipedia” and “dictionary.” Press the desired option to execute the search.

Please note that Kindle is fairly picky, so try to type accurately and competely, e.g., “fired” instead of “fire,” if “fired” is the word you’re seeking. A list of the search results will be returned, and you can navigate to the desired result. Simply then press the “back” toggle, once or twice, (beneath the 5-way button) to return to where you were reading.

How To Search in Kindle Fire Tablet Devices

1. On the Kindle Fire tablet, with the book open, simply highlight the word you wish to search.

2. A dialogue box will pop up, with a dictionary definition, a Note option, a Highlight option and a Search option.

3. Press “Search” and you’ll be asked “Search in Book,” “Search in Wikipedia” or “Search the Web.”

4. Pick the search you want’in this case, for the glossary definitions, you’d choose “Search in Book,” and a chapter-by-chapter list of search hits will be returned.

5. Select the item you want, (in many cases, simply the search result will give you the answer you seek), and you’ll be taken to the result, with your searched term highlighted in yellow.

When done reading, tap the page lightly, and when the bottom menu bar pops up, tap the BACK arrow to return to where you were reading.

How To Search on the Nook and NookColor Tablet Devices

On the Nook and NookColor tablets, with the book open, simply highlight the word you wish to search (using touch).

1. A dialogue box will pop up, with a Highlight option, a Notes option, a Share option (if enabled), a “Look Up” option and a “find” option.

2. Press “Lookup” for a dictionary search (you can also search Google and Wikipedia from this interface.)

3. For an in-book search, simply press “Find.”

4. A list of every occurrence of the term, by chapter, will pop-up in the lower-left-hand corner.

5. Select the choice to which you want to navigate with a thumb-press.

After reviewing the search result, press the double « button (occasionally you may need to press it twice) to return to the location at which you initiated the search.

How To Search on the iPad iBooks application

In the iBooks application, with the book open, simply highlight the word you wish to search (using touch).

1. A dialogue box will pop up, with a Define, Highlight, Note and Search option, with the word you identified highighted in blue.

2. Press “Search” for an in-book search.

3. A list of every occurence of the term, by page number, will pop-up in the upper-right-hand corner.

4. Select the choice to which you want to navigate with a thumb-press.

After reviewing the search result, look down at the lower-left-hand corner of the page, where you’ll see, in very small text, beneath the progress bar, text that says “Back to page X.” Simply press that text to return to the location at which you initiated the search.

How To Search on the Kobo device

Back Button

There is no Back button as such on the Kobo Touch. There is one hardware button that will take you from anywhere to the Kobo Home Page which is a collection of thumbnails of the last five books you’ve been reading. At this page, if you touch the cover image for a particular book, you will be positioned at the last page you read to.

How to Search on the Kobo Touch

To search for a word or phrase in the book:

1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.

2. Touch the open book to pop up a menu with five items, the third of which is “Search in Book.”

3. Touch “Search in Book” and an input box appears at the top of the screen. Underneath is a touchable list of the last six words or phrases that you searched for. On the bottom half of the screen is a keyboard.

4. Type the word or phrase you want to search for with the keyboard and touch “Go” in the lower right corner.

5. You will be presented with a list of phrases from the book text where the word or phrase occurs, organized by the section of the book where they occur.

To find a definition for a word:

1. Touch the bottom of the screen and bring up the menu bar that has icons including the open book.

2. Touch the open book to pop up a menu with five items, the fourth of which is “Definition.”

3. Touch “Definition” and an input box appears at the top of the screen. On the bottom half of the screen is a keyboard.

4. Type the word for which you want a definition with the keyboard and touch “Go” in the lower right corner.

5. You will be presented with a list of definitions.

1. Introduction to the Advanced Level

This overview document is intended for anyone with an interest in the ISTQB Advanced Level who wants a high-level introduction to its leading principles and an overview of the individual Advanced Level syllabi.

The Advanced Level certifications will be examined with three major role descriptions in mind, each representing basic responsibilities and expectations within an organization. In any organization, responsibilities and associated tasks may be split between different individuals or covered by a single individual.

The following Advanced syllabi are defined:

- Test Manager

- Test Analyst

- Technical Test Analyst

In this document each Advanced Level syllabus is summarized and associated Business Outcomes are stated. These Business Outcomes provide a specific statement of what can be expected from a person who achieves Advanced Level Certification in a particular subject area, (e.g., an Advanced Test Manager), and will outline the benefits for companies who are considering the development of specific testing skills at this level.

For stakeholders who are already familiar with or use the 2007 version of the Advanced Level Syllabus, a summary of principal changes is provided in the appendix.

1.1 Career Paths for Testers

Building on the Foundation Level, the Advanced Level supports the definition of career paths for professional testers. A person with the Advanced Level Certificate has extended the broad understanding of testing acquired at the Foundation Level to enable the role of Test Manager or Test Analyst to be performed. Test Analyst skills may be further extended into the role of Technical Test Analyst.

The Advanced Level establishes a platform from which further skills and knowledge may be acquired at the Expert Level. After achieving experience as a Test Manager, for example, a person may choose to develop their testing career further by acquiring Expert Level certifications in the subjects of test management and improving the test process.

People possessing an ISTQB Advanced Level Certificate may use the Certified Tester Advanced Level acronyms CTAL-TM, CTAL-TA, and CTAL-TTA, according to the subject for which they are certified. If all three certifications are acquired, the CTAL-Full acronym may be used.

1.2 Intended Audience

The Advanced Level qualification is aimed at people who have achieved an advanced point in their careers in software testing. This includes people in roles such as Testers, Test Analysts, Test Engineers, Test Consultants, Test Managers, User Acceptance Testers and Software Developers.

This Advanced Level qualification is also appropriate for anyone who wants a deeper understanding of software testing, such as Project Managers, Quality Managers, Software Development Managers, Business Analysts, IT Directors and Management Consultants.

1.3 Learning Objectives

In general the Foundation Level syllabus and all parts of the specific Advanced Level syllabus are examinable at a K1 level, i.e. the candidate will recognize, remember and recall terms and concepts stated in the Foundation and the specific Advanced Level syllabus.

The relevant Learning Objectives at K2, K3 and K4 levels are provided at the beginning of each chapter within each particular Advanced Level Syllabus.

1.4 Entry Requirements

To be able to obtain an Advanced Level certification, candidates must hold the Foundation Certificate and satisfy the board which examines them that they have sufficient practical experience to be considered Advanced Level qualified. Refer to the relevant Exam Board and/or National Board to check the specific criteria used to evaluate practical experience.

1.5 Structure and Course Duration

The Advanced Level consists of three separate syllabi. Each syllabus is self-contained; there are no shared or common elements.

The syllabi must be taught in the following minimum number of days:

Syllabus

Days

Test Manager

5

Test Analyst

4

Technical Test Analyst

3

The following figure shows the structure of the Advanced Level and its relationship to the Foundation and Expert Levels. Note that at the time of writing the Advanced Level Test Analyst certification has no direct dependency link to an Expert Level certification. This may change in the future as new syllabi are introduced at the Expert Level.

1.6 Handling of Standards

Standards (IEEE, ISO, etc.) are referenced in these syllabi. The purpose of these references is to provide a framework (as in the references to ISO 9126/ISO 25000 regarding quality characteristics) or to provide a source of additional information if desired by the reader. Please note that only the items from these standards that are referenced specifically in the syllabi are eligible for examination. The standards documents themselves are not intended for examination and are included only for reference.

1.7 Keeping It Current

The software industry changes rapidly. To deal with these changes and to provide the stakeholders with access to relevant and current information, the ISTQB working groups have created links on the www.istqb.org web site which refer to supporting documents, changes to standards and new occurrences in the industry. This information is not examinable under these syllabi.

2. Overview of Advanced Level Syllabi

2.1 Advanced Level: Test Manager (TM)

2.1.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Test Manager certification.

An Advanced Test Manager can…

TM1

Manage a testing project by implementing the mission, goals and testing processes established for the testing organization.

TM2

Organize and lead risk identification and risk analysis sessions and use the results of such sessions for test estimation, planning, monitoring and control.

TM3

Create and implement test plans consistent with organizational policies and test strategies.

TM4

Continuously monitor and control the test activities to achieve project objectives.

TM5

Assess and report relevant and timely test status to project stakeholders.

TM6

Identify skills and resource gaps in their test team and participate in sourcing adequate resources.

TM7

Identify and plan necessary skills development within their test team.

TM8

Propose a business case for test activities which outlines the costs and benefits expected.

TM9

Ensure proper communication within the test team and with other project stakeholders.

TM10

Participate in and lead test process improvement initiatives.

In general, an Advanced Test Manager is expected to have acquired the necessary skills to enable further development at the Expert Level in the areas of test management and improving the test process.

2.1.2 Content

Chapter 1: Testing Process

- The Test Manager’s main activities are covered for each step in the fundamental test process. Emphasis is placed on the tasks of Test Planning, Monitoring and Control. In addition, the Test Manager learns how to implement a project retrospective in order to validate processes and discover areas to improve.

Chapter 2: Test Management

- This is the core chapter of the syllabus and accounts for over two days of training out of the five days course.

- The Test Manager should understand how to define test management tasks according to the context of a project. This involves taking all stakeholder needs into account and adjusting test activities to the software lifecycle model.

- Risk-based testing (RBT) for test prioritization and effort allocation is covered in detail. Emphasis is placed on the Test Manager’s tasks in identifying, analyzing and assessing risks for the product and the project. The Test Manager learns specific techniques and understands how to take stakeholder views into account when assessing risk levels and when defining the set of test activities needed to mitigate the risks. By performing appropriate risk control activities, the Test Manager learns how to determine residual risks and can report them to project stakeholders such that informed release decisions can be made.

- The Test Manager needs to know how to evaluate different types of test documentation and tailor them to meet project and organization needs. This includes an understanding of relevant standards.

- Estimation of testing effort using specific metrics and based on experience is covered. Techniques required for estimating are covered and the Test Manager should learn how to use available data to communicate the value of testing.

- Test Managers should have an understanding for distributed, outsourced and insourced forms of test organizations.

Chapter 3: Reviews

- The activities of the Test Manager focus on defining an appropriate review plan and setting up the review to achieve the best results. The Test Manager should learn how to use metrics to optimize the review results and to show return on investment.

- The Test Manager should understand how to lead a review team through a formal review.

Chapter 4: Defect Management

- The Test Manager should know how to set up a defect lifecycle tailored for the adopted software lifecycle.

- The Test Manager should understand the skills to acquire in analyzing defect reports with the objective of evaluating the capability of the testing and software development processes.

Chapter 5: Improving the Test Process

- The Test Manager should be aware of the generic steps for conducting a standard test process improvement initiative and how to create a test improvement plan based on these steps.

- Information is provided on the test process improvement models TMMi, TPI Next, CTP and STEP.

Chapter 6: Test Tools and Automation

- The Test Manager should be able to plan and implement the selection of different types of tools (including open-source and custom-built tools) such that risks, costs, benefits and opportunities are adequately considered.

- An understanding of using tools for metric collection and evaluation is acquired.

Chapter 7: People Skills - Team Composition

- The Test Manager should understand how to assess the availability of soft skills along with the technical, business domain and testing skills required for a testing team, and can define a growth plan for skill development.

- The Test Manager should understand the relevant team motivation and communication skills required.

2.2 Advanced Level: Test Analyst (TA)

2.2.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Test Analyst certification.

An Advanced Test Analyst can…

TA1

Perform the appropriate testing activities based on the software development lifecycle being used.

TA2

Determine the proper prioritization of the testing activities based on the information provided by the risk analysis.

TA3

Select and apply appropriate testing techniques to ensure that tests provide an adequate level of confidence, based on defined coverage criteria.

TA4

Provide the appropriate level of documentation relevant to the testing activities.

TA5

Determine the appropriate types of functional testing to be performed.

TA6

Assume responsibility for the usability testing for a given project.

TA7

Effectively participate in formal and informal reviews with stakeholders, applying knowledge of typical mistakes made in work products.

TA8

Design and implement a defect classification scheme.

TA9

Apply tools to support an efficient testing process.

2.2.2 Content

Chapter 1: Testing Process

- A Test Analyst should understand the importance of involvement throughout the test process, with particular focus on the role and contribution of the Test Analyst, and how those align with the roles of the Test Manager and Technical Test Analyst.

- The Test Analyst’s tasks of test analysis and design are described. This includes determining the appropriate uses for concrete and logical test cases as well as defining the pre-conditions and criteria used for starting test execution and determining the status of test completion criteria. The impact of different lifecycle models is a central aspect to these tasks.

Chapter 2: Test Management: Responsibilities for the Test Analyst

- A Test Analyst should understand the purpose and expected contribution toward the project metrics.

- A Test Analyst should understand how to prioritize on the basis of risks and can schedule business domain-based tests appropriately. This includes understanding the risk assessment impact on test case selection, test coverage and test data tasks.

Chapter 3: Test Techniques

- Testing techniques belong to the core competencies of the Test Analyst. Specification-based and defect- and experience-based techniques are covered.

- The specification-based techniques introduced at the Foundation level are developed further. These include equivalence partitioning, boundary value analysis, decision tables, state transition testing and use case testing.

- Additional specification-based techniques introduced include classification tree testing, use of orthogonal arrays, pairwise testing, domain analysis and user stories.

- Defect-based techniques, exploratory testing and the use of defect taxonomies are additional techniques covered in the area of defect- and experience-based techniques.

- A Test Analyst should understand how to select the best technique for a given testing situation as well as how to effectively mix techniques to achieve the best testing result.

Chapter 4: Testing Software Quality Characteristics

- The specific software quality characteristics that apply to the Test Analyst are covered in this section. These include the functional areas of accuracy, suitability and interoperability as well as the non-functional areas of usability and accessibility.

- A Test Analyst should understand how to approach these areas and the testing techniques that can be applied.

Chapter 5: Reviews

- The activities of the Test Analyst focus on using checklists to identify defects in use cases and in requirements specifications from a tester’s perspective. In addition, the Test Analyst learns how to present problems found in a review meeting.

- Several sample checklists are supplied to help guide review sessions for various work products.

Chapter 6: Defect Management

- Test Analysts should understand how to define defect classification values to be used in a defect management system and how to apply this classification to defects found.

- A discussion of the importance of capturing, refining and using root cause information for process improvement is included in this section. Test Analyst should know how to perform preliminary root cause analysis to help provide good classification information for each reported defect.

Chapter 7: Test Tools

- This short chapter focuses on the tools and automation issues which are relevant to the Test Analyst. This includes building awareness of business process modeling tools and knowledge of the interactions between the tools typically used by a Test Analyst.

2.3 Advanced Level: Technical Test Analyst (TTA)

2.3.1 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Advanced Technical Test Analyst certification.

An Advanced Technical Test Analyst can…

TTA1

Recognize and classify the typical risks associated with the performance, security, reliability, portability and maintainability of software systems.

TTA2

Create test plans which detail the planning, design and execution of tests for mitigating performance, security, reliability, portability and maintainability risks.

TTA3

Select and apply appropriate structural design techniques to ensure that tests provide an adequate level of confidence, based on code coverage and design coverage.

TTA4

Effectively participate in technical reviews with developers and software architects applying knowledge of typical mistakes made in code and architecture.

TTA5

Recognize risks in code and software architecture and create test plan elements to mitigate those risks through dynamic analysis.

TTA6

Propose improvements to the security, maintainability and testability of code by applying static analysis.

TTA7

Outline the costs and benefits to be expected from introducing particular types of test automation.

TTA8

Select appropriate tools to automate technical testing tasks.

TTA9

Understand the technical issues and concepts in applying test automation.

2.3.2 Content

Chapter 1: The Technical Test Analyst’s Tasks in Risk-Based Testing

- A Technical Test Analyst should understand how to identify, assess and mitigate technical risks.

Chapter 2: Structure-Based Testing

- Structural testing techniques belong to the core competencies of the Technical Test Analyst.

- This section builds on the Foundation techniques of statement and decision coverage.

- The structure-based techniques covered are condition testing, decision/condition testing, modified condition/decision coverage (MC/DC), multiple condition testing, basis path testing and API coverage.

- In general, Technical Test Analysts should understand how to choose appropriate structural test technique(s).

Chapter 3: Analytical Techniques

- Technical Test Analysts should understand how to apply static analysis to detect potential security, maintainability and testability defects in code.

- The planning of dynamic analysis to mitigate risks in code and software architecture is covered.

Chapter 4: Quality Characteristics for Technical Testing

- A Technical Test Analyst should understand how to design high-level test cases for security, performance and reliability quality attributes and to support the Test Manager in creating test strategies to mitigate the identified risks.

- A Technical Test Analyst should understand how to include coverage of maintainability, portability and resource utilization quality attributes in a testing strategy.

Chapter 5: Reviews

- The activities of the Technical Test Analyst focus on using checklists to identify defects in code and architecture.

Chapter 6 Test Tools and Automation

- This major chapter focuses on the tools and automation issues which are relevant to Technical Test Analysts.

- Several tools are covered, including those used for web-based testing, for supporting model-based testing, for fault seeding and fault injection, for unit testing and the build process and for performance testing.

- A Technical Test Analyst should be able to recognize common technical issues that cause high failure rates in automation projects and to appreciate different automation techniques.

- Specific issues resulting from the use of open-source and custom-built tools are covered.

3. Appendix: Main Changes in the 2012 Syllabi

Introduction

In this 2012 version of the Advanced Level syllabi, feedback was considered from stakeholders in the market such as training providers and attendees.

Improved Scoping

A principal objective of the new Advanced Level Syllabi is to ensure that there is no overlap between any current ISTQB syllabi.

The Advanced Syllabi add to the Foundation Level and fit neatly with the topics managed at the Expert Level (Test Management, Improving the Test Process, Test Automation, etc.).

In addition, the consistency and clear scoping between the three Advanced Level syllabi has been substantially improved.

Clearer Organization of the Syllabus

The 2012 Advanced Level Syllabi have been created as three stand-alone syllabi with a single overview document (this document).

This structure makes it clearer for everyone (students, training providers and exam creation teams) to understand what is expected and what is to be covered. This is also consistent with the structure of the Expert Level. The individual Advanced syllabi may evolve separately if needed.

Reduced Duration of TA and TTA Syllabi

Stakeholder feedback has been carefully considered in setting the minimum course duration. As a result of careful scoping and a policy of “no overlaps”, the duration of the Test Analyst syllabus has been reduced from five to four days and the duration of the Technical Test Analyst has been reduced from five to three days.

Business Outcomes Now Form the Basis for the Advanced syllabi

Each Business Outcome provides a statement of what can be expected from a person who achieves the Advanced Level in the particular subject area, (e.g. an Advanced Test Manager). The BOs are listed in this document.

Business Outcomes are specifically directed to the business needs of industry and will particularly benefit businesses who are considering investing in developing the skills of their staff at the Advanced level.

The Learning Objectives of each Advanced Syllabus implement (and are traceable to) the Business Outcomes.

Improved Learning Objectives

Learning Objectives have been improved by removing potential misinterpretations and by splitting certain “compound” learning objectives into individual parts.

Learning Objectives are now uniquely numbered and placed at the start of each syllabus chapter, in line with Foundation and Expert Level syllabi.

The following example demonstrates these improvements:

2007 syllabus:

(K3) Use the algorithms “Control flow analysis”, “Data flow analysis” to verify if code has not any control or data flow anomaly

2012 syllabus (TTA):

TTA-3.2.1 (K3) Use control flow analysis to detect if code has any control flow anomalies

TTA-3.2.2 (K3) Use data flow analysis to detect if code has any data flow anomalies

Changes to the Test Management Syllabus

The principal subjects covered remain the same. However, the content has been improved.

Redundancies with FL have been removed (e.g. reviews).

The Expert Level syllabi “Test Management“ and “Improving the Test Process“ are now available. Scoping and alignment with these syllabi has taken place.

The defect management chapter is no longer based on IEEE-1044 and focuses more on setting up a defect management lifecycle and using defect data for process improvement.

Changes to the Test Analyst Syllabus

The major concentration of this syllabus remains on the testing techniques and test process.

Domain analysis and user stories are new to the specification-based techniques section.

Content has been significantly revised to align the scope with Foundation, other Advanced Level syllabi and Expert Level Test Automation.

The Test Management and Tools chapters are relatively short and cover only the specific issues relating to the Test Analyst.

The defect management chapter is no longer based on IEEE-1044 and focuses more on defect categorization and performing initial root cause analysis of defects.

Changes to the Technical Test Analyst Syllabus

A significant refocus on the technical aspects of testing has been completed. As a result it is now expected that candidates must be able to read and understand pseudo-code.

The basic aspects of testing and the testing process are now covered in other syllabi.

The test management chapter is relatively short and covers only the specific issues relating to the Technical Test Analyst.

Testing techniques remains a significant part of the syllabus and accounts for approximately one third of the training time. Six techniques are covered, including basis path testing and API coverage. The LCSAJ technique has been removed from the syllabus.

Content has been significantly reduced due to scoping with the other syllabi (in particular Test Analyst).

4. Abbreviations

Abbreviation

Meaning

BO

Business Outcome

ISTQB

International Software Testing Qualifications Board

LO

Learning Objective

TA

Test Analyst

TM

Test Manager

TTA

Technical Test Analyst

5. References

5.1 Trademarks

The following registered trademarks and service marks are used in this document:

ISTQB® is a registered trademark of the International Software Testing Qualifications Board

5.2 Documents and Web Sites

Identifier

Reference

[ISTQB-Web]

Web site of the International Software Testing Qualifications Board. Refer to this website for the latest ISTQB Glossary and syllabi. (www.istqb.org)

Certified Tester

Advanced Level Syllabus
Technical Test Analyst

Version 2012

International Software Testing Qualifications Board

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®).

Advanced Level Technical Test Analyst Working Group: Graham Bath (Chair), Paul Jorgensen, Jamie Mitchell; 2010-2012.

Revision History

Version

Date

Remarks

ISEB v1.1

04SEP01

ISEB Practitioner Syllabus

ISTQB 1.2E

SEP03

ISTQB Advanced Level Syllabus from EOQ-SG

V2007

12OCT07

Certified Tester Advanced Level syllabus version 2007

D100626

26JUN10

Incorporation of changes as accepted in 2009, separation of each chapters for the separate modules

D101227

27DEC10

Acceptance of changes to format and corrections that have no impact on the meaning of the sentences.

Draft V1

17SEP11

First version of the new separate TTA syllabus based on agreed scoping document. AL WG Review

Draft V2

20NOV11

NB Review version

Alpha 2012

09MAR12

Incorporation of all comments from NBs received from October release.

Beta 2012

07APR12

Beta Version submitted to GA

Beta 2012

08JUN12

Copy edited version released to NBs

Beta 2012

27JUN12

EWG and Glossary comments incorporated

RC 2012

15AUG12

Release candidate version - final NB edits included

RC 2012

02SEP12

Comments from BNLTB and Stuart Reid incorporated Paul Jorgensen cross-check

GA 2012

19OCT12

Final edits and cleanup for GA release

Table of Contents

Revision History

Acknowledgements

0.    Introduction to this Syllabus

0.1    Purpose of this Document

0.2    Overview

0.3    Examinable Learning Objectives

0.4    Expectations

1.    The Technical Test Analyst’s Tasks in Risk-Based Testing - 30 mins.

1.1    Introduction

1.2    Risk Identification

1.3    Risk Assessment

1.4    Risk Mitigation

2.    Structure-Based Testing - 225 mins.

2.1    Introduction

2.2    Condition Testing

2.3    Decision Condition Testing

2.4    Modified Condition/Decision Coverage (MC/DC) Testing

2.5    Multiple Condition Testing

2.6    Path Testing

2.7    API Testing

2.8    Selecting a Structure-Based Technique

3.    Analytical Techniques - 255 mins.

3.1    Introduction

3.2    Static Analysis

3.2.1    Control Flow Analysis

3.2.2    Data Flow Analysis

3.2.3    Using Static Analysis for Improving Maintainability

3.2.4    Call Graphs

3.3    Dynamic Analysis

3.3.1    Overview

3.3.2    Detecting Memory Leaks

3.3.3    Detecting Wild Pointers

3.3.4    Analysis of Performance

4.    Quality Characteristics for Technical Testing - 405 mins.

4.1    Introduction

4.2    General Planning Issues

4.2.1    Stakeholder Requirements

4.2.2    Required Tool Acquisition and Training

4.2.3    Test Environment Requirements

4.2.4    Organizational Considerations

4.2.5    Data Security Considerations

4.3    Security Testing

4.3.1    Introduction

4.3.2    Security Test Planning

4.3.3    Security Test Specification

4.4    Reliability Testing

4.4.1    Measuring Software Maturity

4.4.2    Tests for Fault Tolerance

4.4.3    Recoverability Testing

4.4.4    Reliability Test Planning

4.4.5    Reliability Test Specification

4.5    Performance Testing

4.5.1    Introduction

4.5.2    Types of Performance Testing

4.5.3    Performance Test Planning

4.5.4    Performance Test Specification

4.6    Resource Utilization

4.7    Maintainability Testing

4.7.1    Analyzability, Changeability, Stability and Testability

4.8    Portability Testing

4.8.1    Installability Testing

4.8.2    Co-existence/Compatibility Testing

4.8.3    Adaptability Testing

4.8.4    Replaceability Testing

5.    Reviews - 165 mins

5.1    Introduction

5.2    Using Checklists in Reviews

5.2.1    Architectural Reviews

5.2.2    Code Reviews

6.    Test Tools and Automation - 195 mins.

6.1    Integration and Information Interchange Between Tools

6.2    Defining the Test Automation Project

6.2.1    Selecting the Automation Approach

6.2.2    Modeling Business Processes for Automation

6.3    Specific Test Tools

6.3.1    Fault Seeding/Fault Injection Tools

6.3.2    Performance Testing Tools

6.3.3    Tools for Web-Based Testing

6.3.4    Tools to Support Model-Based Testing

6.3.5    Component Testing and Build Tools

7.    References

7.1    Standards

7.2    ISTQB Documents

7.3    Books

7.4    Other References

8.    Index

Acknowledgements

This document was produced by a core team from the International Software Testing Qualifications Board Advanced Level Working Group - Technical Test Analyst: Graham Bath (Chair), Paul Jorgensen, Jamie Mitchell.

The core team thanks the review team and the National Boards for their suggestions and input.

At the time the Advanced Level Syllabus was completed the Advanced Level Working Group had the following membership (alphabetical order):

Graham Bath, Rex Black, Maria Clara Choucair, Debra Friedenberg, Bernard Homès (Vice Chair), Paul Jorgensen, Judy McKay, Jamie Mitchell, Thomas Mueller, Klaus Olsen, Kenji Onishi, Meile Posthuma, Eric Riou du Cosquer, Jan Sabak, Hans Schaefer, Mike Smith (Chair), Geoff Thompson, Erik van Veenendaal, Tsuyoshi Yumoto.

The following persons participated in the reviewing, commenting and balloting of this syllabus:

Dani Almog, Graham Bath, Franz Dijkman, Erwin Engelsma, Mats Grindal, Dr. Suhaimi Ibrahim, Skule Johansen, Paul Jorgensen, Kari Kakkonen, Eli Margolin, Rik Marselis, Judy McKay, Jamie Mitchell, Reto Mueller, Thomas Müller, Ingvar Nordstrom, Raluca Popescu, Meile Posthuma, Michael Stahl, Chris van Bael, Erik van Veenendaal, Rahul Verma, Paul Weymouth, Hans Weiberg, Wenqiang Zheng, Shaomin Zhu.

This document was formally released by the General Assembly of the ISTQB® on October 19th, 2012.

0. Introduction to this Syllabus

0.1 Purpose of this Document

This syllabus forms the basis for the International Software Testing Qualification at the Advanced Level for the Technical Test Analyst. The ISTQB® provides this syllabus as follows:

1. To National Boards, to translate into their local language and to accredit training providers. National Boards may adapt the syllabus to their particular language needs and modify the references to adapt to their local publications.

2. To Exam Boards, to derive examination questions in their local language adapted to the learning objectives for each syllabus.

3. To training providers, to produce courseware and determine appropriate teaching methods.

4. To certification candidates, to prepare for the exam (as part of a training course or independently).

5. To the international software and systems engineering community, to advance the profession of software and systems testing, and as a basis for books and articles.

The ISTQB® may allow other entities to use this syllabus for other purposes, provided they seek and obtain prior written permission.

0.2 Overview

The Advanced Level is comprised of three separate syllabi:

- Test Manager

- Test Analyst

- Technical Test Analyst

The Advanced Level Overview document [ISTQB_AL_OVIEW] includes the following information:

- Business Outcomes for each syllabus

- Summary for each syllabus

- Relationships between the syllabi

- Description of cognitive levels (K-levels)

- Appendices

0.3 Examinable Learning Objectives

The Learning Objectives support the Business Outcomes and are used to create the examination for achieving the Advanced Technical Test Analyst Certification. In general all parts of this syllabus are examinable at a K1 level. That is, the candidate will recognize, remember and recall a term or concept. The learning objectives at K2, K3 and K4 levels are shown at the beginning of the pertinent chapter.

0.4 Expectations

Some of the learning objectives for the Technical Test Analyst assume that basic experience is available in the following areas:

- General programming concepts

- General concepts of system architectures

1. The Technical Test Analyst’s Tasks in Risk-Based Testing - 30 mins.

Keywords

product risk, risk analysis, risk assessment, risk identification, risk level, risk mitigation, risk-based testing

Learning Objectives for the Technical Test Analyst’s Tasks in Risk-Based Testing

1.3 Risk Assessment

TTA-1.3.1

(K2) Summarize the generic risk factors that the Technical Test Analyst typically needs to consider

Common Learning Objectives

The following learning objective relates to content covered in more than one section of this chapter.

TTA-1.x.1

(K2) Summarize the activities of the Technical Test Analyst within a risk-based approach for planning and executing testing

1.1 Introduction

The Test Manager has overall responsibility for establishing and managing a risk-based testing strategy. The Test Manager usually will request the involvement of the Technical Test Analyst to ensure the risk-based approach is implemented correctly.

Because of their particular technical expertise, Technical Test Analysts are actively involved in the following risk-based testing tasks:

- Risk identification

- Risk assessment

- Risk mitigation

These tasks are performed iteratively throughout the project to deal with emerging product risks and changing priorities, and to regularly evaluate and communicate risk status.

Technical Test Analysts work within the risk-based testing framework established by the Test Manager for the project. They contribute their knowledge of the technical risks that are inherent in the project, such as risks related to security, system reliability and performance.

1.2 Risk Identification

By calling on the broadest possible sample of stakeholders, the risk identification process is most likely to detect the largest possible number of significant risks. Because Technical Test Analysts possess unique technical skills, they are particularly well-suited for conducting expert interviews, brainstorming with co-workers and also analyzing the current and past experiences to determine where the likely areas of product risk lie. In particular, the Technical Test Analysts work closely with their technical peers (e.g., developers, architects, operations engineers) to determine the areas of technical risk.

Sample risks that might be identified include:

- Performance risks (e.g., inability to achieve response times under high load conditions)

- Security risks (e.g., disclosure of sensitive data through security attacks)

- Reliability risks (e.g., application unable to meet availability specified in the Service Level Agreement)

Risk areas relating to specific software quality characteristics are covered in the relevant chapters of this syllabus.

1.3 Risk Assessment

While risk identification is about identifying as many pertinent risks as possible, risk assessment is the study of those identified risks in order to categorize each risk and determine the likelihood and impact associated with each risk.

Determining the level of risk typically involves assessing, for each risk item, the likelihood of occurrence and the impact upon occurrence. The likelihood of occurrence is usually interpreted as the likelihood that the potential problem can exist in the system under test.

The Technical Test Analyst contributes to finding and understanding the potential technical risk for each risk item whereas the Test Analyst contributes to understanding the potential business impact of the problem should it occur.

Generic factors that typically need to be considered include:

- Complexity of technology

- Complexity of code structure

- Conflict between stakeholders regarding technical requirements

- Communication problems resulting from the geographical distribution of the development organization

- Tools and technology

- Time, resource and management pressure

- Lack of earlier quality assurance

- High change rates of technical requirements

- Large number of defects found relating to technical quality characteristics

- Technical interface and integration issues

Given the available risk information, the Technical Test Analyst establishes the levels of risk according to the guidelines established by the Test Manager. For example, the Test Manager may determine that risks should be categorized with a value from 1 to 10, with 1 being highest risk.

1.4 Risk Mitigation

During the project, Technical Test Analysts influence how testing responds to the identified risks. This generally involves the following:

- Reducing risk by executing the most important tests and by putting into action appropriate mitigation and contingency activities as stated in the test strategy and test plan

- Evaluating risks based on additional information gathered as the project unfolds, and using that information to implement mitigation actions aimed at decreasing the likelihood or impact of the risks previously identified and analyzed

2. Structure-Based Testing - 225 mins.

Keywords

atomic condition, condition testing, control flow testing, decision condition testing, multiple condition testing, path testing, short-circuiting, statement testing, structure-based technique

Learning Objectives for Structure-Based Testing

2.2 Condition Testing

TTA-2.2.1

(K2) Understand how to achieve condition coverage and why it may be less rigorous testing than decision coverage

2.3 Decision Condition Testing

TTA-2.3.1

(K3) Write test cases by applying the Decision Condition testing test design technique to achieve a defined level of coverage

2.4 Modified Condition/Decision Coverage (MC/DC) Testing

TTA-2.4.1

(K3) Write test cases by applying the Modified Condition/Decision Coverage (MC/DC) testing test design technique to achieve a defined level of coverage

2.5 Multiple Condition Testing

TTA-2.5.1

(K3) Write test cases by applying the Multiple Condition testing test design technique to achieve a defined level of coverage

2.6 Path Testing

TTA-2.6.1

(K3) Write test cases by applying the Path testing test design technique

2.7 API Testing

TTA-2.7.1

(K2) Understand the applicability of API testing and the kinds of defects it finds

2.8 Selecting a Structure-Based Technique

TTA-2.8.1

(K4) Select an appropriate structure-based technique according to a given project situation

2.1 Introduction

This chapter principally describes structure-based test design techniques, which are also known as white box or code-based test techniques. These techniques use the code, the data and the architecture and/or system flow as the basis for test design. Each specific technique enables test cases to be derived systematically and focuses on a particular aspect of the structure to be considered. The techniques provide coverage criteria which have to be measured and associated with an objective defined by each project or organization. Achieving full coverage does not mean that the entire set of tests is complete, but rather that the technique being used no longer suggests any useful tests for the structure under consideration.

With the exception of condition coverage, the structure-based test design techniques considered in this syllabus are more rigorous than the statement and decision coverage techniques covered in the Foundation Syllabus [ISTQB_FL_SYL].

The following techniques are considered in this syllabus:

- Condition testing

- Decision Condition testing

- Modified Condition/Decision Coverage (MC/DC) testing

- Multiple Condition testing

- Path testing

- API testing

The first four of the techniques listed above are based on decision predicates and broadly find the same type of defects. No matter how complex a decision predicate may be, it will evaluate to either TRUE or FALSE, with one path being taken through the code and the other not. A defect is detected when the intended path is not taken because a complex decision predicate does not evaluate as expected.

In general the first four techniques are successively more thorough; they require more tests to be defined in order to achieve their intended coverage and to find more subtle instances of this type of defect.

Refer to [Bath08], [Beizer90], [Beizer95], [Copeland03] and [Koomen06].

2.2 Condition Testing

Compared to decision (branch) testing, which considers the entire decision as a whole and evaluates the TRUE and FALSE outcomes in separate test cases, condition testing considers how a decision is made. Each decision predicate is made up of one or more simple “atomic” conditions, each of which evaluates to a discrete Boolean value. These are logically combined to determine the final outcome of the decision. Each atomic condition must be evaluated both ways by the test cases to achieve this level of coverage.

Applicability

Condition testing is probably interesting only in the abstract because of the difficulties noted below. Understanding it, however, is necessary to achieving greater levels of coverage that build upon it.

Limitations/Difficulties

When there are two or more atomic conditions in a decision, an unwise choice in selecting test data during test design can result in achieving condition coverage while failing to achieve decision coverage. For example, assume the decision predicate, “A and B”.

A

B

A and B

Test 1

FALSE

TRUE

FALSE

Test 2

TRUE

FALSE

FALSE

To achieve 100% condition coverage, run the two tests shown in the table above. While these two tests achieve 100% condition coverage, they fail to achieve decision coverage, since in both cases the predicate evaluates to FALSE.

When a decision consists of a single atomic condition, condition testing is identical to decision testing.

2.3 Decision Condition Testing

Decision Condition testing specifies that testing must achieve condition coverage (see above), and also requires that decision coverage (see Foundation syllabus [ISTQB_FL_SYL]) also be satisfied. A thoughtful choice of test data values for atomic conditions may result in achieving this level of coverage without adding extra test cases beyond that needed to achieve condition coverage.

The example below tests the same decision predicate seen above, “A and B”. Decision Condition coverage can be achieved with the same number of tests by selecting different test values.

A

B

A and B

Test 1

TRUE

TRUE

TRUE

Test 2

FALSE

FALSE

FALSE

This technique therefore may have an efficiency advantage.

Applicability

This level of coverage should be considered when the code being tested is important but not critical.

Limitations/Difficulties

Because it may require more test cases than testing to the decision level, it may be problematic when time is an issue.

2.4 Modified Condition/Decision Coverage (MC/DC) Testing

This technique provides a stronger level of control flow coverage. Assuming N unique atomic conditions, MC/DC can usually be achieved in N+1 unique test cases. MC/DC achieves Decision Condition coverage, but then requires the following also be fulfilled:

1. At least one test where the decision outcome would change if the atomic condition X were TRUE

2. At least one test where the decision outcome would change if the atomic condition X were FALSE

3. Each different atomic condition has tests that meet requirements 1 and 2.

A

B

C

(A or B) and C

Test 1

TRUE

FALSE

TRUE

TRUE

Test 2

FALSE

TRUE

TRUE

TRUE

Test 3

FALSE

FALSE

TRUE

FALSE

Test 4

TRUE

FALSE

FALSE

FALSE

In the example above, decision coverage is achieved (the outcome of the decision predicate is both TRUE and FALSE), and condition coverage is achieved (A, B, and C are all evaluated both TRUE and FALSE).

In Test 1, A is TRUE and the output is TRUE. If A is changed to FALSE (as in Test 3, holding other values unchanged) the result turns to FALSE.

In Test 2, B is TRUE and the output is TRUE. If B is changed to FALSE (as in Test 3, holding other values unchanged) the result turns to FALSE.

In Test 1, C is TRUE and the output is TRUE. If C is changed to FALSE (as in Test 4, holding other values unchanged) the result turns to FALSE.

Applicability

This technique is used extensively in the aerospace software industry and many other safety-critical systems. It should be used when dealing with safety critical software where any failure may cause a catastrophe.

Limitations/Difficulties

Achieving MC/DC coverage may be complicated when there are multiple occurrences of a specific term in an expression; when this occurs, the term is said to be “coupled”. Depending on the decision statement in the code, it may not be possible to vary the value of the coupled term such that it alone causes the decision outcome to change. One approach in addressing this issue is to specify that only uncoupled atomic conditions must be tested to the MC/DC level. The other approach is to analyze each decision in which coupling occurs on a case-by-case basis.

Some programming languages and/or interpreters are designed such that they exhibit short-circuiting behavior when evaluating a complex decision statement in the code. That is, the executing code may not evaluate an entire expression if the final outcome of the evaluation can be determined after evaluating only a portion of the expression. For example, if evaluating the decision “A and B”, there is no reason to evaluate B if A evaluates to FALSE. No value of B can change the final value so the code may save execution time by not evaluating B. Short-circuiting may affect the ability to attain MC/DC coverage since some required tests may not be achievable.

2.5 Multiple Condition Testing

In rare instances, it might be required to test all possible combinations of values that a decision may contain. This exhaustive level of testing is called multiple condition coverage. The number of required tests is dependent on the number of atomic conditions in the decision statement and can be determined by calculating 2n where n is the number of uncoupled atomic conditions. Using the same example as before, the following tests are required to achieve multiple condition coverage:

A

B

C

(A or B) and C

Test 1

TRUE

TRUE

TRUE

TRUE

Test 2

TRUE

TRUE

FALSE

FALSE

Test 3

TRUE

FALSE

TRUE

TRUE

Test 4

TRUE

FALSE

FALSE

FALSE

Test 5

FALSE

TRUE

TRUE

TRUE

Test 6

FALSE

TRUE

FALSE

FALSE

Test 7

FALSE

FALSE

TRUE

FALSE

Test 8

FALSE

FALSE

FALSE

FALSE

If the language uses short-circuiting, the number of actual test cases will often be reduced, depending on the order and grouping of logical operations that are performed on the atomic conditions.

Applicability

Traditionally, this technique was used to test embedded software which was expected to run reliably without crashing for long periods of time (e.g., telephone switches that were expected to last 30 years). This type of testing is likely to be replaced by MC/DC testing in most critical applications.

Limitations/Difficulties

Because the number of test cases can be derived directly from a truth table containing all of the atomic conditions, this level of coverage can easily be determined. However, the sheer number of test cases required makes MC/DC coverage more applicable to most situations.

2.6 Path Testing

Path testing consists of identifying paths through the code and then creating tests to cover them. Conceptually, it would be useful to test every unique path through the system. In any non-trivial system, however, the number of test cases could become excessively large due to the nature of looping structures.

By setting aside the issue of indefinite looping, however, it is realistic to perform some path testing. To apply this technique [Beizer90] recommends that tests are created which follow many paths through the module, from entry to exit. To simplify what might be a complex task, he recommends that this can be done systematically, using the following procedure:

1. Pick the first path as the simplest, functionally sensible path from entry to exit.

2. Pick each additional path as a small variation on the previous path. Try to change only one branch in the path that is different for each successive test. Favor short paths over long paths when possible. Favor paths that make functional sense over ones that do not.

3. Pick paths that don’t make functional sense only when required for coverage. Beizer notes in this rule that such paths might be extraneous and should be questioned.

4. Use intuition when choosing paths (i.e., which paths are most likely to be executed).

Note that some path segments are likely to be executed more than once using this strategy. The key point of this strategy is to test every possible branch through the code at least once and possibly many times.

Applicability

Partial path testing—as defined above—is often performed in safety critical software. It is a good addition to the other methods covered in this chapter because it looks at paths through the software rather than just at the way decisions are made,

Limitations/Difficulties

While it is possible to use a control flow graph to determine the paths, realistically a tool is required to calculate them for complex modules.

Coverage

Creating sufficient tests to cover all paths (disregarding loops) guarantees that both statement and branch coverage is achieved. Path testing provides more thorough testing than branch coverage, with a relatively small increase in the number of tests. [NIST 96]

2.7 API Testing

An Application Programming Interface (API) is code which enables communication between different processes, programs and/or systems. APIs are often utilized in a client/server relationship where one process supplies some kind of functionality to other processes.

In certain respects API testing is quite similar to testing a graphical user interface (GUI). The focus is on the evaluation of input values and returned data.

Negative testing is often crucial when dealing with APIs. Programmers who use APIs to access services external to their own code may try to use API interfaces in ways for which they were not intended. That means that robust error handling is essential to avoid incorrect operation. Combinatorial testing of many different interfaces may be required because APIs are often used in conjunction with other APIs, and because a single interface may contain several parameters, where values of these may be combined in many ways.

APIs frequently are loosely coupled, resulting in the very real possibility of lost transactions or timing glitches. This necessitates thorough testing of the recovery and retry mechanisms. An organization that provides an API interface must ensure that all services have a very high availability; this often requires strict reliability testing by the API publisher as well as infrastructure support.

Applicability

API testing is becoming more important as more systems become distributed or use remote processing as a way of off-loading some work to other processors. Examples include operating systems calls, service-oriented architectures (SOA), remote procedure calls (RPC), web services, and virtually every other distributed application. API testing is particularly applicable for testing systems of systems.

Limitations/Difficulties

Testing an API directly usually requires a Technical Test Analyst to use specialized tools. Because there is typically no direct graphical interface associated with an API, tools may be required to setup the initial environment, marshal the data, invoke the API, and determine the result.

Coverage

API testing is a description of a type of testing; it does not denote any specific level of coverage. At a minimum the API test should include exercise of all calls to the API as well as all valid and reasonable invalid values.

Types of Defects

The types of defects that can be found by testing APIs are quite disparate. Interface issues are common, as are data handling issues, timing problems, loss of transactions and duplication of transactions.

2.8 Selecting a Structure-Based Technique

The context of the system under test will determine the level of structure-based testing coverage that should be achieved. The more critical the system, the higher the level of coverage needed. In general, the higher the level of coverage required, the more time and resources will be needed to achieve that level.

Sometimes the level of coverage required may be derived from applicable standards that apply to the software system. For example, if the software were to be used in an airborne environment, it may be required to conform to standard DO-178B (in Europe, ED-12B.) This standard contains the following five failure conditions:

1. Catastrophic: failure may cause lack of critical function needed to safely fly or land the plane

2. Hazardous: failure may have a large negative impact on safety or performance

3. Major: failure is significant, but less serious than A or B

4. Minor: failure is noticeable, but with less impact than C

5. No effect: failure has no impact on safety

If the software system is categorized as level A, it must be tested to MC/DC coverage. If it is level B, it must be tested to decision level coverage although MC/DC is optional. Level C requires statement coverage at minimum.

Likewise, IEC-61508 is an international standard for the functional safety of programmable, electronic, safety-related systems. This standard has been adapted in many different areas, including automotive, rail, manufacturing process, nuclear power plants, and machinery. Criticality is defined using a graduated Safety Integrity Level (SIL) continuum (1 being the least critical, 4 being the most) and coverage is recommended as follows:

1. Statement and branch coverage recommended

2. Statement coverage highly recommended, branch coverage recommended

3. Statement and branch coverage highly recommended

4. MC/DC highly recommended

In modern systems, it is rare that all processing will be done on a single system. API testing should be instituted anytime some of the processing is going to be done remotely. The criticality of the system should determine how much effort should be invested in API testing.

As always, the context of the software system under test should guide the Technical Test Analyst on the methods used in testing.

3. Analytical Techniques - 255 mins.

Keywords

control flow analysis, cyclomatic complexity, data flow analysis, definition-use pairs, dynamic analysis, memory leak, pairwise integration testing, neighborhood integration testing, static analysis, wild pointer

Learning Objectives for Analytical Techniques

3.2 Static Analysis

TTA-3.2.1

(K3) Use control flow analysis to detect if code has any control flow anomalies

TTA-3.2.2

(K3) Use data flow analysis to detect if code has any data flow anomalies

TTA-3.2.3

(K3) Propose ways to improve the maintainability of code by applying static analysis

TTA-3.2.4

(K2) Explain the use of call graphs for establishing integration testing strategies

3.3 Dynamic Analysis

TTA-3.3.1

(K3) Specify goals to be achieved by the use of dynamic analysis

3.1 Introduction

There are two types of analysis: static analysis and dynamic analysis.

Static analysis (Section 3.2) encompasses the analytical testing that can occur without executing the software. Because the software is not executing, it is examined either by a tool or by a person to determine if it will process correctly when it is executed. This static view of the software allows detailed analysis without having to create the data and preconditions that would cause the scenario to be exercised.

Note that the different forms of review which are relevant for the Technical Test Analyst are covered in Chapter 5.

Dynamic analysis (Section 3.3) requires the actual execution of the code and is used to find coding faults which are more easily detected when the code is executing (e.g., memory leaks). Dynamic analysis, as with static analysis, may rely on tools or may rely on an individual monitoring the executing system watching for such indicators as rapid memory growth.

3.2 Static Analysis

The objective of static analysis is to detect actual or potential faults in code and system architecture and to improve their maintainability. Static analysis is generally supported by tools.

3.2.1 Control Flow Analysis

Control flow analysis is the static technique where the control flow through a program is analyzed, either through the use of a control flow graph or a tool. There are a number of anomalies which can be found in a system using this technique, including loops that are badly designed (e.g., having multiple entry points), ambiguous targets of function calls in certain languages (e.g., Scheme), incorrect sequencing of operations, etc.

One of the most common uses of control flow analysis is to determine cyclomatic complexity. The cyclomatic complexity value is a positive integer which represents the number of independent paths in a strongly connected graph with loops and iterations ignored as soon as they have been traversed once. Each independent path, from entry to exit, represents a unique path through the module. Each unique path should be tested.

The cyclomatic complexity value is generally used to understand the overall complexity of a module. Thomas McCabe’s theory [McCabe 76] was that the more complex the system, the harder it would be to maintain and the more defects it would contain. Many studies over the years have noted this correlation between complexity and the number of contained defects. The NIST (National Institute of Standards and Technology) recommends a maximum complexity value of 10. Any module that is measured with a higher complexity may need to be divided into multiple modules.

3.2.2 Data Flow Analysis

Data flow analysis covers a variety of techniques which gather information about the use of variables in a system. Scrutiny is given to the lifecycle of the variables, (i.e., where they are declared, defined, read, evaluated and destroyed), since anomalies can occur during any of those operations.

One common technique is called define-use notation where the lifecycle of each variable is split into three different atomic actions:

- d: when the variable is declared, defined or initialized

- u: when the variable is used or read in either a computation or a decision predicate

- k: when the variable is killed, destroyed or goes out of scope

These three atomic actions are combined into pairs (“definition-use pairs”) to illustrate the data flow. For example, a “du-path” represents a fragment of the code where a data variable is defined and then subsequently used.

Possible data anomalies include performing the correct action on a variable at the wrong time or carrying out an incorrect action on data in a variable. These anomalies include:

- Assigning an invalid value to a variable

- Failing to assign a value to a variable before using it

- Taking an incorrect path due to an incorrect value in a control predicate

- Trying to use a variable after it is destroyed

- Referencing a variable when it is out of scope

- Declaring and destroying a variable without using it

- Redefining a variable before it has been used

- Failing to kill a dynamically allocated variable (causing a possible memory leak)

- Modifying a variable, which results in unexpected side effects (e.g., ripple effects when changing a global variable without considering all uses of the variable)

The development language being used may guide the rules used in data flow analysis. Programming languages may allow the programmer to perform certain actions with variables which are not illegal, but may cause the system to behave differently than the programmer expected under certain circumstances. For example, a variable might be defined twice without actually being used when a certain path is followed. Data flow analysis will often label these uses “suspicious”. While this may be a legal use of the variable assignment capability, it can lead to future maintainability issues in the code.

Data flow testing “uses the control flow graph to explore the unreasonable things that can happen to data” [Beizer90] and therefore finds different defects than control flow testing. A Technical Test Analyst should include this technique when planning testing since many of these defects cause intermittent failures that are difficult to find while performing dynamic testing.

However, data flow analysis is a static technique; it may miss some issues that occur to data in the run-time system. For example, the static data variable may contain a pointer into a dynamically created array that does not even exist until run-time. Multi-processor usage and pre-emptive multi-tasking may create race conditions which will not be found by data flow or control flow analysis.

3.2.3 Using Static Analysis for Improving Maintainability

Static analysis can be applied in a number of ways to improve the maintainability of code, architecture and web sites.

Poorly written, uncommented and unstructured code tends to be harder to maintain. It may require more effort for developers to locate and analyze defects in the code and the modification of the code to correct a defect or add a new feature may result in further defects being introduced.

Static analysis is used with tool support to improve code maintainability by verifying compliance to coding standards and guidelines. These standards and guidelines describe required coding practices such as naming conventions, commenting, indentation and code modularization. Note that static analysis tools generally flag warnings rather than errors even though the code may be syntactically correct.

Modular designs generally result in more maintainable code. Static analysis tools support the development of modular code in the following ways:

- They search for repeated code. These sections of code may be candidates for refactoring into modules (although the run-time overhead imposed by module calls may be an issue for real-time systems).

- They generate metrics which are valuable indicators of code modularization. These include measures of coupling and cohesion. A system that is to have good maintainability is more likely to have a low measure of coupling (the degree to which modules rely on each other during execution) and a high measure of cohesion (the degree to which a module is self-contained and focused on a single task).

- They indicate, in object-oriented code, where derived objects may have too much or too little visibility into parent classes.

- They highlight areas in code or architecture with a high level of structural complexity, which is generally considered to be an indicator for poor maintainability and a higher potential for containing faults. Acceptable levels of cyclomatic complexity (see Section 3.2.1.) may be specified in guidelines to ensure that code is developed in a modular fashion with maintainability and defect prevention in mind. Code with high levels of cyclomatic complexity may be candidates for modularization.

Maintenance of a web site can also be supported using static analysis tools. Here the objective is to check if the tree-like structure of the site is well-balanced or if there is an imbalance that will lead to:

- More difficult testing tasks

- Increased maintenance workload

- Difficult navigation for the user

3.2.4 Call Graphs

Call graphs are a static representation of communication complexity. They are directed graphs in which nodes represent program units and edges represent communication among units.

Call graphs may be used in unit testing where different functions or methods call each other, in integration and system testing when separate modules call each other, or in system integration testing when separate systems call each other.

Call graphs can be used for the following purposes:

- Designing tests that call a specific module or system

- Establishing the number of locations within the software from where a module or system is called

- Evaluating the structure of the code and of the system architecture

- Providing suggestions for the order of integration (pairwise and neighborhood integration. These are discussed in more detail below.

In the Foundation Level syllabus [ISTQB_FL_SYL], two different categories of integration testing were discussed: incremental (top-down, bottom-up, etc.) and non-incremental (big bang). The incremental methods were said to be preferred because they introduce code in increments thus making fault isolation easier since the amount of code involved is limited.

In this Advanced syllabus, three more non-incremental methods using call graphs are introduced. These may be preferable to incremental methods which likely will require additional builds to complete testing and non-shippable code to be written to support the testing. These three methods are:

- Pairwise integration testing (not to be confused with the black box test technique “pairwise testing”), targets pairs of components that work together as seen in the call graph for integration testing. While this method reduces the number of builds only by a small amount, it reduces the amount of test harness code needed.

- Neighborhood integration tests all of the nodes that connect to a given node as the basis for the integration testing. All predecessor and successor nodes of a specific node in the call graph are the basis for the test.

- McCabe’s design predicate approach uses the theory of cyclomatic complexity as applied to a call graph for modules. This requires the construction of a call graph that shows the different ways that modules can call each other, including:

- Unconditional call: the call of one module to another always happens

- Conditional call: the call of one module to another sometimes happens

- Mutually exclusive conditional call: a module will call one (and only one) of a number of different modules

- Iterative call: one module calls another at least once but may call it multiple times

- Iterative conditional call: one module can call another zero to many times

After creating the call graph, the integration complexity is calculated, and tests are created to cover the graph.

Refer to [Jorgensen07] for more information on using call graphs and pairwise integration testing.

3.3 Dynamic Analysis

3.3.1 Overview

Dynamic analysis is used to detect failures where the symptoms may not be immediately visible. For example, the possibility of memory leaks may be detectable by static analysis (finding code that allocates but never frees memory), but a memory leak is readily apparent with dynamic analysis.

Failures that are not immediately reproducible can have significant consequences on the testing effort and on the ability to release or productively use software. Such failures may be caused by memory leaks, incorrect use of pointers and other corruptions (e.g., of the system stack) [Kaner02]. Due to the nature of these failures, which may include the gradual worsening of system performance or even system crashes, testing strategies must consider the risks associated with such defects and, where appropriate, perform dynamic analysis to reduce them (typically by using tools). Since these failures often are the most expensive failures to find and to correct, it is recommended to perform dynamic analysis early in the project.

Dynamic analysis may be applied to accomplish the following:

- Prevent failures from occurring by detecting wild pointers and loss of system memory

- Analyze system failures which cannot easily be reproduced

- Evaluate network behavior

- Improve system performance by providing information on run-time system behavior

Dynamic analysis may be performed at any test level and requires technical and system skills to do the following:

- Specify the testing objectives of dynamic analysis

- Determine the proper time to start and stop the analysis

- Analyze the results

During system testing, dynamic analysis tools can be used even if the Technical Test Analysts have minimal technical skills; the tools utilized usually create comprehensive logs which can be analyzed by those with the needed technical skills.

3.3.2 Detecting Memory Leaks

A memory leak occurs when the areas of memory (RAM) available to a program are allocated by that program but are not subsequently released when no longer needed. This memory area is left as allocated and is not available for re-use. When this occurs frequently or in low memory situations, the program may run out of usable memory. Historically, memory manipulation was the responsibility of the programmer. Any dynamically allocated areas of memory had to be released by the allocating program within the correct scope to avoid a memory leak. Many modern programming environments include automatic or semi-automatic “garbage collection” where allocated memory is freed without the programmer’s direct intervention. Isolating memory leaks can be very difficult in cases where existing allocated memory is freed by the automatic garbage collection.

Memory leaks cause problems which develop over time and may not always be immediately obvious. This may be the case if, for example, the software has been recently installed or the system restarted, which often occurs during testing. For these reasons, the negative effects of memory leaks may often first be noticed when the program is in production.

The symptoms of a memory leak are a steadily worsening of system response time which may ultimately result in system failure. While such failures may be resolved by re-starting (re-booting) the system, this may not always be practical or even possible.

Many dynamic analysis tools identify areas in the code where memory leaks occur so that they can be corrected. Simple memory monitors can also be used to obtain a general impression of whether available memory is declining over time, although a follow-up analysis would still be required to determine the exact cause of the decline.

There are other sources for leaks that also should be considered. Examples include file handles, semaphores and connection pools for resources.

3.3.3 Detecting Wild Pointers

“Wild” pointers within a program are pointers which must not be used. For example, a wild pointer may have “lost” the object or function to which it should be pointing or it does not point to the area of memory intended (e.g., it points to an area that is beyond the allocated boundaries of an array). When a program uses wild pointers, a variety of consequences may occur:

- The program may perform as expected. This may be the case where the wild pointer accesses memory which is currently not used by the program and is notionally “free” and/or contains a reasonable value.

- The program may crash. In this case the wild pointer may have caused a part of the memory to be incorrectly used which is critical to the running of the program (e.g., the operating system).

- The program does not function correctly because objects required by the program cannot be accessed. Under these conditions the program may continue to function, although an error message may be issued.

- Data in the memory location may be corrupted by the pointer and incorrect values subsequently used.

Note that any changes made to the program’s memory usage (e.g., a new build following a software change) may trigger any of the four consequences listed above. This is particularly critical where initially the program performs as expected despite the use of wild pointers, and then crashes unexpectedly (perhaps even in production) following a software change. It is important to note that such failures are often symptoms of an underlying defect (i.e., the wild pointer) (Refer to [Kaner02], “Lesson 74”). Tools can help identify wild pointers as they are used by the program, irrespective of their impact on the program’s execution. Some operating systems have built-in functions to check for memory access violations during run-time. For instance, the operating system may throw an exception when an application tries to access a memory location that is outside of that application’s allowed memory area.

3.3.4 Analysis of Performance

Dynamic analysis is not just useful for detecting failures. With the dynamic analysis of program performance, tools help identify performance bottlenecks and generate a wide range of performance metrics which can be used by the developer to tune the system performance. For example, information can be provided about the number of times a module is called during execution. Modules which are frequently called would be likely candidates for performance enhancement.

By merging the information about the dynamic behavior of the software with information obtained from call graphs during static analysis (see Section 3.2.4), the tester can also identify the modules which might be candidates for detailed and extensive testing (e.g., modules which are frequently called and have many interfaces).

Dynamic analysis of program performance is often done while conducting system tests, although it may also be done when testing a single sub-system in earlier phases of testing using test harnesses.

4. Quality Characteristics for Technical Testing - 405 mins.

Keywords

adaptability, analyzability, changeability, co-existence, efficiency, installability, maintainability testing, maturity, operational acceptance test, operational profile, performance testing, portability testing, recoverability testing, reliability growth model, reliability testing, replaceability, resource utilization testing, robustness, security testing, stability, testability

Learning Objectives for Quality Characteristics for Technical Testing

4.2 General Planning Issues

TTA-4.2.1

(K4) For a particular project and system under test, analyze the non-functional requirements and write the respective sections of the test plan

4.3 Security Testing

TTA-4.3.1

(K3) Define the approach and design high-level test cases for security testing

4.4 Reliability Testing

TTA-4.4.1

(K3) Define the approach and design high-level test cases for the reliability quality characteristic and its corresponding ISO 9126 sub-characteristics

4.5 Performance Testing

TTA-4.5.1

(K3) Define the approach and design high-level operational profiles for performance testing

Common Learning Objectives

The following learning objectives relate to content covered in more than one section of this chapter.

TTA-4.x.1

(K2) Understand and explain the reasons for including maintainability, portability and resource utilization tests in a testing strategy and/or test approach

TTA-4.x.2

(K3) Given a particular product risk, define the particular non-functional test type(s) which are most appropriate

TTA-4.x.3

(K2) Understand and explain the stages in an application’s lifecycle where non-functional tests should be applied

TTA-4.x.4

(K3) For a given scenario, define the types of defects you would expect to find by using non-functional testing types

4.1 Introduction

In general, the Technical Test Analyst focuses testing on “how” the product works, rather than the functional aspects of “what” it does. These tests can take place at any test level. For example, during component testing of real time and embedded systems, conducting performance benchmarking and testing resource usage is important. During system test and Operational Acceptance Test (OAT), testing for reliability aspects, such as recoverability, is appropriate. The tests at this level are aimed at testing a specific system, i.e., combinations of hardware and software. The specific system under test may include various servers, clients, databases, networks and other resources. Regardless of the test level, testing should be conducted according to the risk priorities and the available resources.

The description of product quality characteristics provided in ISO 9126 is used as a guide to describing the characteristics. Other standards, such as the ISO 25000 series (which has superseded ISO 9126) may also be of use. The ISO 9126 quality characteristics are divided into characteristics, each of which may have sub-characteristics. These are shown in the table below, together with an indication of which characteristics/sub-characteristics are covered by the Test Analyst and Technical Test Analyst syllabi.

While this allocation of work may vary in different organizations, it is this one that is followed in these ISTQB syllabi.

The sub-characteristic of compliance is shown for each of the quality characteristics. In the case of certain safety-critical or regulated environments, each quality characteristic may have to comply with specific standards and regulations. Because those standards can vary widely depending on the industry, they will not be discussed in depth here. If the Technical Test Analyst is working in an environment that is affected by compliance requirements, it is important to understand those requirements and to ensure that both the testing and the test documentation will fulfill the compliance requirements.

For all of the quality characteristics and sub-characteristics discussed in this section, the typical risks must be recognized so that an appropriate testing strategy can be formed and documented. Quality characteristic testing requires particular attention to lifecycle timing, required tools, software and documentation availability and technical expertise. Without planning a strategy to deal with each characteristic and its unique testing needs, the tester may not have adequate planning, preparation and test execution time built into the schedule [Bath08]. Some of this testing, e.g., performance testing, requires extensive planning, dedicated equipment, specific tools, specialized testing skills and, in most cases, a significant amount of time. Testing of the quality characteristics and sub-characteristics must be integrated into the overall testing schedule, with adequate resources allocated to the effort. Each of these areas has specific needs, targets specific issues and may occur at different times during the software lifecycle, as discussed in the sections below.

While the Test Manager will be concerned with compiling and reporting the summarized metric information concerning quality characteristics and sub-characteristics, the Test Analyst or the Technical Test Analyst (according to the table above) gathers the information for each metric.

Measurements of quality characteristics gathered in pre-production tests by the Technical Test Analyst may form the basis for Service Level Agreements (SLA) between the supplier and the stakeholders (e.g., customers, operators) of the software system. In some cases, the tests may continue to be executed after the software has entered production, often by a separate team or organization. This is usually seen for efficiency and reliability testing which may show different results in the production environment than in the testing environment.

4.2 General Planning Issues

Failure to plan for non-functional tests can put the success of an application at considerable risk. The Technical Test Analyst may be requested by the Test Manager to identify the principal risks for the relevant quality characteristics (see table in Section 4.1) and address any planning issues associated with the proposed tests. These may be used in creating the Master Test Plan. The following general factors are considered when performing these tasks:

- Stakeholder requirements

- Required tool acquisition and training

- Test environment requirements

- Organizational considerations

- Data security considerations

4.2.1 Stakeholder Requirements

Non-functional requirements are often poorly specified or even non-existent. At the planning stage, Technical Test Analysts must be able to obtain expectation levels relating to technical quality characteristics from affected stakeholders and evaluate the risks that these represent.

A common approach is to assume that if the customer is satisfied with the existing version of the system, they will continue to be satisfied with new versions, as long as the achieved quality levels are maintained. This enables the existing version of the system to be used as a benchmark. This can be a particularly useful approach to adopt for some of the non-functional quality characteristics such as performance, where stakeholders may find it difficult to specify their requirements.

It is advisable to obtain multiple viewpoints when capturing non-functional requirements. They must be elicited from stakeholders such as customers, users, operations staff and maintenance staff; otherwise some requirements are likely to be missed.

4.2.2 Required Tool Acquisition and Training

Commercial tools or simulators are particularly relevant for performance and certain security tests. Technical Test Analysts should estimate the costs and timescales involved for acquiring, learning and implementing the tools. Where specialized tools are to be used, planning should account for the learning curves for new tools and/or the cost of hiring external tool specialists.

The development of a complex simulator may represent a development project in its own right and should be planned as such. In particular the testing and documentation of the developed tool must be accounted for in the schedule and resource plan. Sufficient budget and time should be planned for upgrading and retesting the simulator as the simulated product changes. The planning for simulators to be used in safety-critical applications must take into account the acceptance testing and possible certification of the simulator by an independent body.

4.2.3 Test Environment Requirements

Many technical tests (e.g., security tests, performance tests) require a production-like test environment in order to provide realistic measures. Depending on the size and complexity of the system under test, this can have a significant impact on the planning and funding of the tests. Since the cost of such environments may be high, the following alternatives may be considered:

- Using the production environment

- Using a scaled-down version of the system. Care must then be taken that the test results obtained are sufficiently representative of the production system.

The timing of such test executions must be planned carefully and it is quite likely that such tests can only be executed at specific times (e.g., at low usage times).

4.2.4 Organizational Considerations

Technical tests may involve measuring the behavior of several components in a complete system (e.g., servers, databases, networks). If these components are distributed across a number of different sites and organizations, the effort required to plan and co-ordinate the tests may be significant. For example, certain software components may only be available for system testing at particular times of day or year, or organizations may only offer support for testing for a limited number of days. Failing to confirm that system components and staff (i.e., “borrowed” expertise) from other organizations are available “on call” for testing purposes may result in severe disruption to the scheduled tests.

4.2.5 Data Security Considerations

Specific security measures implemented for a system should be taken into account at the test planning stage to ensure that all testing activities are possible. For example, the use of data encryption may make the creation of test data and the verification of results difficult.

Data protection policies and laws may preclude the generation of any required test data based on production data. Making test data anonymous is a non-trivial task which must be planned for as part of the test implementation.

4.3 Security Testing

4.3.1 Introduction

Security testing differs from other forms of functional testing in two significant areas:

1. Standard techniques for selecting test input data may miss important security issues

2. The symptoms of security defects are very different from those found with other types of functional testing

Security testing assesses a system’s vulnerability to threats by attempting to compromise the system’s security policy. The following is a list of potential threats which should be explored during security testing:

- Unauthorized copying of applications or data

- Unauthorized access control (e.g., ability to perform tasks for which the user does not have rights). User rights, access and privileges are the focus of this testing. This information should be available in the specifications for the system.

- Software which exhibits unintended side-effects when performing its intended function. For example, a media player which correctly plays audio but does so by writing files out to unencrypted temporary storage exhibits a side-effect which may be exploited by software pirates.

- Code inserted into a web page which may be exercised by subsequent users (cross-site scripting or XSS). This code may be malicious.

- Buffer overflow (buffer overrun) which may be caused by entering strings into a user interface input field which are longer than the code can correctly handle. A buffer overflow vulnerability represents an opportunity for running malicious code instructions.

- Denial of service, which prevents users from interacting with an application (e.g., by overloading a web server with “nuisance” requests).

- The interception, mimicking and/or altering and subsequent relaying of communications (e.g., credit card transactions) by a third party such that a user remains unaware of that third party’s presence (“Man in the Middle” attack)

- Breaking the encryption codes used to protect sensitive data

- Logic bombs (sometimes called Easter Eggs), which may be maliciously inserted into code and which activate only under certain conditions (e.g., on a specific date). When logic bombs activate, they may perform malicious acts such as the deletion of files or formatting of disks.

4.3.2 Security Test Planning

In general the following aspects are of particular relevance when planning security tests:

- Because security issues can be introduced during the architecture, design and implementation of the system, security testing may be scheduled for the unit, integration and system testing levels. Due to the changing nature of security threats, security tests may also be scheduled on a regular basis after the system has entered production.

- The test strategies proposed by the Technical Test Analyst may include code reviews and static analysis with security tools. These can be effective in finding security issues in architecture, design documents and code that are easily missed during dynamic testing.

- The Technical Test Analyst may be called upon to design and execute certain security “attacks” (see below) which require careful planning and coordination with stakeholders. Other security tests may be performed in cooperation with developers or with Test Analysts (e.g., testing user rights, access and privileges). Planning of the security tests must include careful consideration of organizational issues such as these.

- An essential aspect of security test planning is obtaining approvals. For the Technical Test Analyst, this means obtaining explicit permission from the Test Manager to perform the planned security tests. Any additional, unplanned tests performed could appear to be actual attacks and the person conducting those tests could be at risk for legal action. With nothing in writing to show intent and authorization, the excuse “We were performing a security test” may be difficult to explain convincingly.

- It should be noted that improvements which may be made to the security of a system may affect its performance. After making security improvements it is advisable to consider the need for conducting performance tests (see Section 4.5 below).

4.3.3 Security Test Specification

Particular security tests may be grouped [Whittaker04] according to the origin of the security risk:

- User interface related - unauthorized access and malicious inputs

- File system related - access to sensitive data stored in files or repositories

- Operating system related - storage of sensitive information such as passwords in non-encrypted form in memory which could be exposed when the system is crashed through malicious inputs

- External software related - interactions which may occur among external components that the system utilizes. These may be at the network level (e.g., incorrect packets or messages passed) or at the software component level (e.g., failure of a software component on which the software relies).

The following approach [Whittaker04] may be used to develop security tests:

- Gather information which may be useful in specifying tests, such as names of employees, physical addresses, details regarding the internal networks, IP numbers, identity of software or hardware used, and operating system version.

- Perform a vulnerability scan using widely available tools. Such tools are not used directly to compromise the system(s), but to identify vulnerabilities that are, or that may result in, a breach of security policy. Specific vulnerabilities can also be identified using checklists such as those provided by the National Institute of Standards and Technology (NIST) [Web-2].

- Develop “attack plans” (i.e., a plan of testing actions intended to compromise a particular system’s security policy) using the gathered information. Several inputs via various interfaces (e.g., user interface, file system) need to be specified in the attack plans to detect the most severe security faults. The various “attacks” described in [Whittaker04] are a valuable source of techniques developed specifically for security testing.

Security issues can also be exposed by reviews (see Chapter 5) and/or the use of static analysis tools (see Section 3.2). Static analysis tools contain an extensive set of rules which are specific to security threats and against which the code is checked. For example, buffer overflow issues, caused by failure to check buffer size before data assignment, can be found by the tool.

Static analysis tools can be used for web code to check for possible exposure to security vulnerabilities such as code injection, cookie security, cross site scripting, resource tampering and SQL code injection.

4.4 Reliability Testing

The ISO 9126 classification of product quality characteristics defines the following sub-characteristics of reliability:

- Maturity

- Fault tolerance

- Recoverability

4.4.1 Measuring Software Maturity

An objective of reliability testing is to monitor a statistical measure of software maturity over time and compare this to a desired reliability goal which may be expressed as a Service Level Agreement (SLA). The measures may take the form of a Mean Time Between Failures (MTBF), Mean Time to Repair (MTTR) or any other form of failure intensity measurement (e.g., number of failures of a particular severity occurring per week). These may be used as exit criteria (e.g., for production release).

4.4.2 Tests for Fault Tolerance

In addition to the functional testing that evaluates the software’s tolerance to faults in terms of handling unexpected input values (so-called negative tests), additional testing is needed to evaluate a system’s tolerance to faults which occur externally to the application under test. Such faults are typically reported by the operating system (e.g., disk full, process or service not available, file not found, memory not available). Tests of fault tolerance at the system level may be supported by specific tools.

Note that the terms “robustness” and “error tolerance” are also commonly used when discussing fault tolerance (see [ISTQB_GLOSSARY] for details).

4.4.3 Recoverability Testing

Further forms of reliability testing evaluate the software system’s ability to recover from hardware or software failures in a predetermined manner which subsequently allows normal operations to be resumed. Recoverability tests include Failover and Backup and Restore tests.

Failover tests are performed where the consequences of a software failure are so negative that specific hardware and/or software measures have been implemented to ensure system operation even in the event of failure. Failover tests may be applicable, for example, where the risk of financial losses is extreme or where critical safety issues exist. Where failures may result from catastrophic events, this form of recoverability testing may also be called “disaster recovery” testing.

Typical preventive measures for hardware failures might include load balancing across several processors and clustering servers, processors or disks so that one can immediately take over from another if it should fail (redundant systems). A typical software measure might be the implementation of more than one independent instance of a software system (for example, an aircraft’s flight control system) in so-called redundant dissimilar systems. Redundant systems are typically a combination of software and hardware measures and may be called duplex, triplex or quadruplex systems, depending on the number of independent instances (two, three or four respectively). The dissimilar aspect for the software is achieved when the same software requirements are provided to two (or more) independent and not connected development teams, with the objective of having the same services provided with different software. This protects the redundant dissimilar systems in that a similar defective input is less likely to have the same result. These measures taken to improve the recoverability of a system may directly influence its reliability as well and should also be considered when performing reliability testing.

Failover testing is designed to explicitly test systems by simulating failure modes or actually causing failures in a controlled environment. Following a failure, the failover mechanism is tested to ensure that data is not lost or corrupted and that any agreed service levels are maintained (e.g., function availability or response times). For more information on failover testing, see [Web-1].

Backup and Restore tests focus on the procedural measures set up to minimize the effects of a failure. Such tests evaluate the procedures (usually documented in a manual) for taking different forms of backup and for restoring that data if data loss or corruption should occur. Test cases are designed to ensure that critical paths through each procedure are covered. Technical reviews may be performed to “dry-run” these scenarios and validate the manuals against the actual procedures. Operational Acceptance Tests (OAT) exercise the scenarios in a production or production-like environment to validate their actual use.

Measures for Backup and Restore tests may include the following:

- Time taken to perform different types of backup (e.g., full, incremental)

- Time taken to restore data

- Levels of guaranteed data backup (e.g., recovery of all data no more than 24 hours old, recovery of specific transaction data no more than one hour old)

4.4.4 Reliability Test Planning

In general the following aspects are of particular relevance when planning reliability tests:

- Reliability can continue to be monitored after the software has entered production. The organization and staff responsible for operation of the software must be consulted when gathering reliability requirements for test planning purposes.

- The Technical Test Analyst may select a reliability growth model which shows the expected levels of reliability over time. A reliability growth model can provide useful information to the Test Manager by enabling comparison of the expected and achieved reliability levels.

- Reliability tests should be conducted in a production-like environment. The environment used should remain as stable as possible to enable reliability trends to be monitored over time.

- Because reliability tests often require use of the entire system, reliability testing is most commonly done as part of system testing. However, individual components can be subjected to reliability testing as well as integrated sets of components. Detailed architecture, design and code reviews can also be used to remove some of the risk of reliability issues occurring in the implemented system.

- In order to produce test results that are statistically significant, reliability tests usually require long execution times. This may make it difficult to schedule within other planned tests.

4.4.5 Reliability Test Specification

Reliability testing may take the form of a repeated set of predetermined tests. These may be tests selected at random from a pool or test cases generated by a statistical model using random or pseudo-random methods. Tests may also be based on patterns of use which are sometimes referred to as “Operational Profiles” (see Section 4.5.4).

Certain reliability tests may specify that memory-intensive actions be executed repeatedly so that possible memory leaks can be detected.

4.5 Performance Testing

4.5.1 Introduction

The ISO 9126 classification of product quality characteristics includes performance (time behavior) as a sub-characteristic of efficiency. Performance testing focuses on the ability of a component or system to respond to user or system inputs within a specified time and under specified conditions.

Performance measurements vary according to the objectives of the test. For individual software components, performance may be measured according to CPU cycles, while for client-based systems performance may be measured according to the time taken to respond to a particular user request. For systems whose architectures consist of several components (e.g., clients, servers, databases) performance measurements are taken for transactions between individual components so that performance “bottlenecks” can be identified.

4.5.2 Types of Performance Testing

4.5.2.1 Load Testing

Load testing focuses on the ability of a system to handle increasing levels of anticipated realistic loads resulting from the transaction requests generated by numbers of concurrent users or processes. Average response times for users under different scenarios of typical use (operational profiles) can be measured and analyzed. See also [Splaine01].

4.5.2.2 Stress Testing

Stress testing focuses on the ability of a system or component to handle peak loads at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as accessible computer capacity and available bandwidth. Performance levels should degrade slowly and predictably without failure as stress levels are increased. In particular, the functional integrity of the system should be tested while the system is under stress in order to find possible faults in functional processing or data inconsistencies.

One possible objective of stress testing is to discover the limits at which a system actually fails so that the “weakest link in the chain” can be determined. Stress testing allows additional capacity to be added to the system in a timely manner (e.g., memory, CPU capability, database storage).

4.5.2.3 Scalability Testing

Scalability testing focuses on the ability of a system to meet future efficiency requirements, which may be beyond those currently required. The objective of the tests is to determine the system’s ability to grow (e.g., with more users, larger amounts of data stored) without exceeding the currently specified performance requirements or failing. Once the limits of scalability are known, threshold values can be set and monitored in production to provide a warning of impending problems. In addition the production environment may be adjusted with appropriate amounts of hardware.

4.5.3 Performance Test Planning

In addition to the general planning issues described in Section 4.2, the following factors can influence the planning of performance tests:

- Depending on the test environment used and the software being tested, (see Section 4.2.3) performance tests may require the entire system to be implemented before effective testing can be done. In this case, performance testing is usually scheduled to occur during system test. Other performance tests which can be conducted effectively at the component level may be scheduled during unit testing.

- In general it is desirable to conduct initial performance tests as early as possible, even if a production-like is not yet available. These early tests may find performance problems (e.g. bottlenecks) and reduce project risk by avoiding time-consuming corrections in the later stages of software development or production.

- Code reviews, in particular those which focus on database interaction, component interaction and error handling, can identify performance issues (particularly regarding “wait and retry” logic and inefficient queries) and should be scheduled early in the software lifecycle.

- The hardware, software and network bandwidth needed to run the performance tests should be planned and budgeted. Needs depend primarily on the load to be generated, which may be based on the number of virtual users to be simulated and the amount of network traffic they are likely to generate. Failure to account for this may result in unrepresentative performance measurements being taken. For example, verifying the scalability requirements of a much-visited Internet site may require the simulation of hundreds of thousands of virtual users.

- Generating the required load for performance tests may have a significant influence on hardware and tool acquisition costs. This must be considered in the planning of performance tests to ensure that adequate funding is available.

- The costs of generating the load for performance tests may be minimized by renting the required test infrastructure. This may involve, for example, renting “top-up” licenses for performance tools or by using the services of a third-party provider for meeting hardware needs (e.g., cloud-services). If this approach is taken, the available time for conducting the performance tests may be limited and must therefore be carefully planned.

- Care should be taken at the planning stage to ensure that the performance tool to be used provides the required compatibility with the communications protocols used by the system under test.

- Performance-related defects often have significant impact on the system under test. When performance requirements are imperative, it is often useful to conduct performance tests on the critical components (via drivers and stubs) instead of waiting for system tests.

4.5.4 Performance Test Specification

The specification of tests for different performance test types such as load and stress are based on the definition of operational profiles. These represent distinct forms of user behavior when interacting with an application. There may be multiple operational profiles for a given application.

The numbers of users per operational profile may be obtained by using monitoring tools (where the actual or comparable application is already available) or by predicting usage. Such predictions may be based on algorithms or provided by the business organization, and are especially important for specifying the operational profile(s) to be used for scalability testing.

Operational profiles are the basis for the number and types of test cases to be used during performance testing. These tests are often controlled by test tools that create “virtual” or simulated users in quantities that will represent the profile under test (see Section 6.3.2).

4.6 Resource Utilization

The ISO 9126 classification of product quality characteristics includes resource utilization as a sub-characteristic of efficiency. Tests relating to resource utilization evaluate the usage of system resources (e.g., usage of memory, disk capacity, network bandwidth, connections) against a predefined benchmark. These are compared under both normal loads and stress situations, such as high levels of transaction and data volumes, to determine if unnatural growth in usage is occurring.

For example for real-time embedded systems, memory usage (sometimes referred to as a “memory footprint”) plays a significant role in performance testing. If the memory footprint exceeds the allowed measure, the system may have insufficient memory needed to perform its tasks within the specified time periods. This may slow down the system or even lead to a system crash.

Dynamic analysis may also be applied to the task of investigating resource utilization (see Section 3.3.4) and detecting performance bottlenecks.

4.7 Maintainability Testing

Software often spends substantially more of its lifetime being maintained than being developed. Maintenance testing is performed to test the changes to an operational system or the impact of a changed environment to an operational system. To ensure that the task of conducting maintenance is as efficient as possible, maintainability testing is performed to measure the ease with which code can be analyzed, changed and tested.

Typical maintainability objectives of affected stakeholders (e.g., the software owner or operator) include:

- Minimizing the cost of owning or operating the software

- Minimizing down time required for software maintenance

Maintainability tests should be included in a test strategy and/or test approach where one or more of the following factors apply:

- Software changes are likely after the software enters production (e.g., to correct defects or introduce planned updates)

- The benefits of achieving maintainability objectives (see above) over the software lifecycle are considered by the affected stakeholders to outweigh the costs of performing the maintainability tests and making any required changes

- The risks of poor software maintainability (e.g., long response times to defects reported by users and/or customers) justify conducting maintainability tests

Appropriate techniques for maintainability testing include static analysis and reviews as discussed in Sections 3.2 and 5.2. Maintainability testing should be started as soon as the design documents are available and should continue throughout the code implementation effort. Since maintainability is built into the code and the documentation for each individual code component, maintainability can be evaluated early in the lifecycle without having to wait for a completed and running system.

Dynamic maintainability testing focuses on the documented procedures developed for maintaining a particular application (e.g., for performing software upgrades). Selections of maintenance scenarios are used as test cases to ensure the required service levels are attainable with the documented procedures. This form of testing is particularly relevant where the underlying infrastructure is complex, and support procedures may involve multiple departments/organizations. This form of testing may take place as part of Operational Acceptance Testing (OAT). [Web-1]

4.7.1 Analyzability, Changeability, Stability and Testability

The maintainability of a system can be measured in terms of the effort required to diagnose problems identified within a system (analyzability), implement the code changes (changeability) and test the changed system (testability). Stability relates specifically to the system’s response to change. Systems with low stability exhibit large numbers of downstream problems (also known as the “ripple effect”) whenever a change is made. [ISO9126] [Web-1].

The effort required to perform maintenance tasks is dependent on a number of factors such as software design methodology (e.g., object orientation) and coding standards used.

Note that “stability” in this context should not be confused with the terms “robustness” and “fault tolerance”, which are covered in Section 4.4.2.

4.8 Portability Testing

Portability tests in general relate to the ease with which software can be transferred into its intended environment, either initially or from an existing environment. Portability tests include tests for installability, co-existence/compatibility, adaptability and replaceability. Portability testing can start with the individual components (e.g., replaceability of a particular component such as changing from one database management system to another) and will expand in scope as more code becomes available. Installability may not be testable until all the components of the product are functionally working. Portability must be designed and built into the product and so must be considered early in the design and architecture phases. Architecture and design reviews can be particularly productive for identifying potential portability requirements and issues (e.g., dependency on a particular operating system).

4.8.1 Installability Testing

Installability testing is conducted on the software and written procedures used to install the software on its target environment. This may include, for example, the software developed to install an operating system onto a processor, or an installation “wizard” used to install a product onto a client PC.

Typical installability testing objectives include:

- Validating that the software can be successfully installed by following the instructions in an installation manual (including the execution of any installation scripts), or by using an installation wizard. This includes exercising installation options for different hardware/software configurations and for various degrees of installation (e.g., initial or update).

- Testing whether failures which occur during installation (e.g., failure to load particular DLLs) are dealt with by the installation software correctly without leaving the system in an undefined state (e.g., partially installed software or incorrect system configurations)

- Testing whether a partial installation/de-installation can be completed

- Testing whether an installation wizard can successfully identify invalid hardware platforms or operating system configurations

- Measuring whether the installation process can be completed within a specified number of minutes or in less than a specified number of steps

- Validating that the software can be successfully downgraded or uninstalled

Functionality testing is normally conducted after the installation test to detect any faults which may have been introduced by the installation (e.g., incorrect configurations, functions not available). Usability testing is normally conducted in parallel with installability testing (e.g., to validate that users are provided with understandable instructions and feedback/error messages during the installation).

4.8.2 Co-existence/Compatibility Testing

Computer systems which are not related to each other are said to be compatible when they can run in the same environment (e.g., on the same hardware) without affecting each other’s behavior (e.g., resource conflicts). Compatibility should be performed when new or upgraded software will be rolled out into environments which already contain installed applications.

Compatibility problems may arise when the application is tested in an environment where it is the only installed application (where incompatibility issues are not detectable) and then deployed onto another environment (e.g., production) which also runs other applications.

Typical compatibility testing objectives include:

- Evaluation of possible adverse impact on functionality when applications are loaded in the same environment (e.g., conflicting resource usage when a server runs multiple applications)

- Evaluation of the impact to any application resulting from the deployment of operating system fixes and upgrades

Compatibility issues should be analyzed when planning the targeted production environment but the actual tests are normally performed after system and user acceptance testing have been successfully completed.

4.8.3 Adaptability Testing

Adaptability testing checks whether a given application can function correctly in all intended target environments (hardware, software, middleware, operating system, etc.). An adaptive system is therefore an open system that is able to fit its behavior according to changes in its environment or in parts of the system itself. Specifying tests for adaptability requires that combinations of the intended target environments are identified, configured and available to the testing team. These environments are then tested using a selection of functional test cases which exercise the various components present in the environment.

Adaptability may relate to the ability of the software to be ported to various specified environments by performing a predefined procedure. Tests may evaluate this procedure.

Adaptability tests may be performed in conjunction with installability tests and are typically followed by functional tests to detect any faults which may have been introduced in adapting the software to a different environment.

4.8.4 Replaceability Testing

Replaceability testing focuses on the ability of software components within a system to be exchanged for others. This may be particularly relevant for systems which use commercial off-the-shelf (COTS) software for specific system components.

Replaceability tests may be performed in parallel with functional integration tests where more than one alternative component is available for integration into the complete system. Replaceability may be evaluated by technical review or inspection at the architecture and design levels, where the emphasis is placed on the clear definition of interfaces to potential replaceable components.

5. Reviews - 165 mins.

Keywords

anti-pattern

Learning Objectives for Reviews

5.1 Introduction

TTA 5.1.1

(K2) Explain why review preparation is important for the Technical Test Analyst

5.2 Using Checklists in Reviews

TTA 5.2.1

(K4) Analyze an architectural design and identify problems according to a checklist provided in the syllabus

TTA 5.2.2

(K4) Analyze a section of code or pseudo-code and identify problems according to a checklist provided in the syllabus

5.1 Introduction

Technical Test Analysts must be active participants in the review process, providing their unique views. They should have formal review training to better understand their respective roles in any technical review process. All review participants must be committed to the benefits of a well-conducted technical review. For a complete description of technical reviews, including numerous review checklists, see [Wiegers02]. Technical Test Analysts normally participate in technical reviews and inspections where they bring an operational (behavioral) viewpoint that may be missed by developers. In addition, Technical Test Analysts play an important role in the definition, application, and maintenance of review checklists and defect severity information.

Regardless of the type of review being conducted, the Technical Test Analyst must be allowed adequate time to prepare. This includes time to review the work product, time to check cross-referenced documents to verify consistency, and time to determine what might be missing from the work product. Without adequate preparation time, the review can become an editing exercise rather than a true review. A good review includes understanding what is written, determining what is missing, and verifying that the described product is consistent with other products that are either already developed or are in development. For example, when reviewing an integration level test plan, the Technical Test Analyst must also consider the items that are being integrated. Are they ready for integration? Are there dependencies that must be documented? Is there data available to test the integration points? A review is not isolated to the work product being reviewed. It must also consider the interaction of that item with the others in the system.

It is not unusual for the authors of a product being reviewed to feel they are being criticized. The Technical Test Analyst should be sure to approach any review comments from the viewpoint of working together with the author to create the best product possible. By using this approach, comments will be worded constructively and will be oriented toward the work product and not the author. For example, if a statement is ambiguous, it is better to say “I don’t understand what I should be testing to verify that this requirement has been implemented correctly. Can you help me understand it?” rather than “This requirement is ambiguous and no one will be able to figure it out.” The Technical Test Analyst’s job in a review is to ensure that the information provided in the work product will be sufficient to support the testing effort. If the information is not there or is not clear, then this is likely a defect that needs to be corrected by the author. By maintaining a positive approach rather than a critical approach, comments will be better received and the meeting will be more productive.

5.2 Using Checklists in Reviews

Checklists are used during reviews to remind the participants to verify specific points during the review. Checklists can also help to de-personalize the review, e.g., “this is the same checklist we use for every review, and we are not targeting only your work product.” Checklists can be generic and used for all reviews or focused on specific quality characteristics or areas. For example, a generic checklist might verify the proper usage of the terms “shall” and “should”, verify proper formatting and similar conformance items. A targeted checklist might concentrate on security issues or performance issues.

The most useful checklists are those gradually developed by an individual organization, because they reflect:

- The nature of the product

- The local development environment

- Staff

- Tools

- Priorities

- History of previous successes and defects

- Particular issues (e.g., performance, security)

Checklists should be customized for the organization and perhaps for the particular project. The checklists provided in this chapter are meant only to serve as examples.

Some organizations extend the usual notion of a software checklist to include “anti-patterns” that refer to common mistakes, poor techniques, and other ineffective practices. The term derives from the popular concept of “design patterns” which are reusable solutions to common problems that have been shown to be effective in practical situations [Gamma94]. An anti-pattern, then, is a commonly made mistake, often implemented as an expedient short-cut.

It is important to remember that if a requirement is not testable, meaning that it is not defined in such a way that the Technical Test Analyst can determine how to test it, then it is a defect. For example, a requirement that states “The software should be fast” cannot be tested. How can the Technical Test Analyst determine if the software is fast? If, instead, the requirement said “The software must provide a maximum response time of three seconds under specific load conditions”, then the testability of this requirement is substantially better, if we define the “specific load conditions” (e.g. number of concurrent users, activities performed by the users). It is also an overarching requirement because this one requirement could easily spawn many individual test cases in a non-trivial application. Traceability from this requirement to the test cases is also critical because if the requirement should change, all the test cases will need to be reviewed and updated as needed.

5.2.1 Architectural Reviews

Software architecture consists of the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution. [ANSI/IEEE Std 1471-2000], [Bass03].

Checklists used for architecture reviews could, for example, include verification of the proper implementation of the following items, which are quoted from [Web-3]:

- “Connection pooling - reducing the execution time overhead associated with establishing database connections by establishing a shared pool of connections

- Load balancing – spreading the load evenly between a set of resources

- Distributed processing

- Caching – using a local copy of data to reduce access time

- Lazy instantiation

- Transaction concurrency

- Process isolation between Online Transactional Processing (OLTP) and Online Analytical Processing (OLAP)

- Replication of data”

More details, (not relevant for the certification exam), can be found in [Web-4], which refers to a paper that surveys 117 checklists from 24 sources. Different categories of checklist items are discussed and examples are provided of good checklist items as well as those that should be avoided.

5.2.2 Code Reviews

Checklists for code reviews are necessarily very detailed, and, as with checklists for architecture reviews, are most useful when they are language, project and company-specific. The inclusion of code-level anti-patterns is helpful, particularly for less experienced software developers.

Checklists used for code reviews could include the following items six items: (based on [Web-5]).

1. Structure

- Does the code completely and correctly implement the design?

- Does the code conform to any pertinent coding standards?

- Is the code well-structured, consistent in style, and consistently formatted?

- Are there any uncalled or unneeded procedures or any unreachable code?

- Are there any leftover stubs or test routines in the code?

- Can any code be replaced by calls to external reusable components or library functions?

- Are there any blocks of repeated code that could be condensed into a single procedure?

- Is storage use efficient?

- Are symbolics used rather than “magic number” constants or string constants?

- Are any modules excessively complex and should be restructured or split into multiple modules?

2. Documentation

- Is the code clearly and adequately documented with an easy-to-maintain commenting style?

- Are all comments consistent with the code?

- Does the documentation conform to applicable standards?

3. Variables

- Are all variables properly defined with meaningful, consistent, and clear names?

- Are there any redundant or unused variables?

4. Arithmetic Operations

- Does the code avoid comparing floating-point numbers for equality?

- Does the code systematically prevent rounding errors?

- Does the code avoid additions and subtractions on numbers with greatly different magnitudes?

- Are divisors tested for zero or noise?

5. Loops and Branches

- Are all loops, branches, and logic constructs complete, correct, and properly nested?

- Are the most common cases tested first in IF-ELSEIF chains?

- Are all cases covered in an IF-ELSEIF or CASE block, including ELSE or DEFAULT clauses?

- Does every case statement have a default?

- Are loop termination conditions obvious and invariably achievable?

- Are indices or subscripts properly initialized, just prior to the loop?

- Can any statements that are enclosed within loops be placed outside the loops?

- Does the code in the loop avoid manipulating the index variable or using it upon exit from the loop?

6. Defensive Programming

- Are indices, pointers, and subscripts tested against array, record, or file bounds?

- Are imported data and input arguments tested for validity and completeness?

- Are all output variables assigned?

- Is the correct data element operated on in each statement?

- Is every memory allocation released?

- Are timeouts or error traps used for external device access?

- Are files checked for existence before attempting to access them?

- Are all files and devices left in the correct state upon program termination?

For further examples of checklists used for code reviews at different testing levels see [Web-6].

6. Test Tools and Automation - 195 mins.

Keywords

data-driven testing, debugging tool, fault seeding tool, hyperlink test tool, keyword-driven testing, performance testing tool, record/playback tool, static analyzer, test execution tool, test management tool

Learning Objectives for Test Tools and Automation

6.1 Integration and Information Interchange Between Tools

TTA-6.1.1

(K2) Explain technical aspects to consider when multiple tools are used together

6.2 Defining the Test Automation Project

TTA-6.2.1

(K2) Summarize the activities that the Technical Test Analyst performs when setting up a test automation project

TTA-6.2.2

(K2) Summarize the differences between data-driven and keyword-driven automation

TTA-6.2.3

(K2) Summarize common technical issues that cause automation projects to fail to achieve the planned return on investment

TTA-6.2.4

(K3) Create a keyword table based on a given business process

6.3 Specific Test Tools

TTA-6.3.1

(K2) Summarize the purpose of tools for fault seeding and fault injection

TTA-6.3.2

(K2) Summarize the main characteristics and implementation issues for performance testing and monitoring tools

TTA-6.3.3

(K2) Explain the general purpose of tools used for web-based testing

TTA-6.3.4

(K2) Explain how tools support the concept of model-based testing

TTA-6.3.5

(K2) Outline the purpose of tools used to support component testing and the build process

6.1 Integration and Information Interchange Between Tools

While the responsibility for selecting and integrating tools belongs to the Test Manager, the Technical Test Analyst may be called upon to review the integration of a tool or set of tools to ensure accurate tracking of data resulting from various testing areas such as static analysis, test execution automation and configuration management. In addition, depending on the programming skills of the Technical Test Analyst, there may also be involvement in creating the code that will integrate tools together that do not integrate “out of the box”.

An ideal toolset should eliminate the duplication of information across the tools. It takes more effort and it is more error-prone to store test execution scripts both in a test management database and in the configuration management system. It would be better to have a test management system that includes a configuration management component or that is able to integrate with a configuration management tool already in place in the organization. Well-integrated defect tracking and test management tools will allow a tester to launch a defect report during test case execution without having to leave the test management tool. Well-integrated static analysis tools should be able to report any discovered incidents and warnings directly to the defect management system (although this should be configurable due to the many warnings that may be generated).

Buying a suite of test tools from the same vendor does not automatically mean that the tools work together adequately. When considering the approach to integrating tools together, a data-centric focus is preferable. The data must be exchanged without manual intervention in a timely manner with guaranteed accuracy including fault recovery. While it is helpful to have a consistent user experience, capture, storage, protection and presentation of data should be the primary focus of tool integration.

An organization should evaluate the cost of automating the information interchange compared to the risk of losing information or allowing the data to get out of synchronization due to necessary manual intervention. As integration may be expensive or difficult, it should be a prime consideration in the overall tool strategy.

Some integrated development environments (IDE) may simplify integration between tools that are capable of working in that environment. This helps to unify the look and feel of the tools that share the same framework. However, a similar user interface will not guarantee a smooth information exchange between components. Coding may be required to complete the integration.

6.2 Defining the Test Automation Project

In order to be cost-effective, test tools and particularly test automation tools, must be carefully architected and designed. Implementing a test automation strategy without a solid architecture usually results in a tool set that is costly to maintain, insufficient for the purpose and unable to achieve the target return on investment.

A test automation project should be considered a software development project. This includes the need for architecture documents, detailed design documents, design and code reviews, component and component integration testing as well as final system testing. Testing can be needlessly delayed or complicated when unstable or inaccurate test automation code is used. There are multiple activities that the Technical Test Analyst performs regarding test automation. These include:

- Determining who will be responsible for the test execution

- Selecting the appropriate tool for the organization, timeline, skills of the team, maintenance requirements (note this could mean deciding to create a tool to use rather than acquiring one)

- Defining the interface requirements between the automation tool and other tools such as the test management and defect management tools

- Selecting the automation approach, i.e., keyword-driven or data-driven (see Section 6.2.1 below)

- Working with the Test Manager to estimate the cost of the implementation, including training

- Scheduling the automation project and allocating the time for maintenance

- Training the Test Analysts and Business Analysts to use and supply data for the automation

- Determining how the automated tests will be executed

- Determining how the automated test results will be combined with the manual test results

These activities and the resulting decisions will influence the scalability and maintainability of the automation solution. Sufficient time must be spent researching the options, investigating available tools and technologies and understanding the future plans for the organization. Some of these activities require more consideration than others, particularly during the decision process. These are discussed in more detail in the following sections.

6.2.1 Selecting the Automation Approach

Test automation is not limited to testing through the GUI. Tools exist to help automate testing at the API level, through a Command Line Interface (CLI) and other interface points in the software under test. One of the first decisions the Technical Test Analyst must make is the most effective interface to be accessed to automate the testing.

One of the difficulties of testing through the GUI is the tendency for the GUI to change as the software evolves. Depending on the way the test automation code is designed, this can result in a significant maintenance burden. For example, using the record/playback capability of a test automation tool may result in automated test cases (often called test scripts) that no longer run as desired if the GUI changes. This is because the recorded script captures interactions with the graphical objects when the tester executes the software manually. If the objects being accessed change, the recorded scripts may also need updating to reflect those changes.

Capture and Playback tools may be used as a convenient starting point for developing automation scripts. The tester records a test session and recorded script is then modified to improve maintainability (e.g., by replacing sections in the recorded script with reusable functions).

Depending on the software being tested, the data used for each test may be different although the executed test steps are virtually identical (e.g., testing error handling for an input field by entering multiple invalid values and checking the error returned for each). It is inefficient to develop and maintain an automated test script for each of these values to be tested. A common technical solution to this problem is to move the data from the scripts to an external store such as a spreadsheet or a database. Functions are written to access the specific data for each execution of the test script, which enables a single script to work through a set of test data that supplies the input values and expected result values (e.g., a value shown in a text field or an error message). This approach is called data-driven. When using this approach, a test script is developed that will process the supplied data, as well as a harness and the infrastructure needed to support the execution of the script or set of scripts. The actual data held in the spreadsheet or database is created by Test Analysts who are familiar with the business function of the software. This division of labor allows those responsible for developing test scripts (e.g., the Technical Test Analyst) to focus on the implementation of intelligent automation scripts while the Test Analyst maintains ownership of the actual test. In most cases, the Test Analyst will be responsible for executing the test scripts once the automation is implemented and tested.

Another approach, called keyword- or action word-driven, goes a step further by also separating the action to be performed on the supplied data from the test script [Buwalda01]. In order to accomplish this further separation, a high-level meta language is created by domain experts (e.g., Test Analysts) which is descriptive rather than directly executable. Each statement of this language describes a full or partial business process of the domain that may require testing. For example, business process keywords could include “Login”, “CreateUser”, and “DeleteUser”. A keyword describes a high-level action that will be performed in the application domain. Lower level actions which denote interaction with the software interface itself, such as: “ClickButton”, “SelectFromList”, or “TraverseTree” may also be defined and may be used to test GUI capabilities that do not neatly fit into business process keywords.

Once the keywords and data to be used have been defined, the test automator (e.g., Technical Test Analyst) translates the business process keywords and lower level actions into test automation code. The keywords and actions, along with the data to be used, may be stored in spreadsheets or entered using specific tools which support keyword-driven test automation. The test automation framework implements the keyword as a set of one or more executable functions or scripts. Tools read test cases written with keywords and call the appropriate test functions or scripts which implement them. The executables are implemented in a highly modular manner to enable easy mapping to specific keywords. Programming skills are needed to implement these modular scripts.

This separation of the knowledge of the business logic from the actual programming required to implement the test automation scripts provides the most effective use of the test resources. The Technical Test Analyst, in the role as the test automator, can effectively apply programming skills without having to become a domain expert across many areas of the business.

Separating the code from the changeable data helps to insulate the automation from changes, improving the overall maintainability of the code and improving the return on the automation investment.

In any test automation design, it is important to anticipate and handle software failures. If a failure occurs, the automator must determine what the software should do. Should the failure be recorded and the tests continue? Should the tests be terminated? Can the failure be handled with a specific action (such as clicking a button in a dialog box) or perhaps by adding a delay in the test? Unhandled software failures may corrupt subsequent test results as well as causing a problem with the test that was executing when the failure occurred.

It is also important to consider the state of the system at the start and end of the tests. It may be necessary to ensure the system is returned to a pre-defined state after the test execution is completed. This will allow a suite of automated tests to be run repeatedly without manual intervention to reset the system to a known state. To do this, the test automation may have to, for example, delete the data it created or alter the status of records in a database. The automation framework should ensure that a proper termination has been accomplished at the end of the tests (i.e., logging out after the tests complete).

6.2.2 Modeling Business Processes for Automation

In order to implement a keyword-driven approach for test automation, the business processes to be tested must be modeled in the high-level keyword language. It is important that the language is intuitive to its users who are likely to be the Test Analysts working on the project.

Keywords are generally used to represent high-level business interactions with a system. For example, “Cancel_Order” may require checking the existence of the order, verifying the access rights of the person requesting the cancellation, displaying the order to be cancelled and requesting confirmation of the cancellation. Sequences of keywords (e.g., “Login”, “Select_Order”, “Cancel_Order”), and the relevant test data are used by the Test Analyst to specify test cases. The following is a simple keyword-driven input table that could be used to test the ability of the software to add, reset and delete user accounts:

The automation script that uses this table would look for the input values to be used by the automation script. For example, when it gets to the row with the keyword “Delete_User”, only the user name is required. To add a new user both user name and password are required. Input values may also be referenced from a data store as shown with the second “Add_User” keyword where a reference to the data is entered rather than the data itself providing more flexibility to access data that may be changing as the tests execute. This allows data-driven techniques to be combined with the keyword scheme.

Issues to consider include:

- The more granular the keywords, the more specific the scenarios that can be covered, but the high-level language may become more complex to maintain.

- Allowing Test Analysts to specify low-level actions (“ClickButton”, “SelectFromList”, etc.) makes the keyword tests much more capable of handling different situations. However, because these actions are tied directly to the GUI, it also may cause the tests to require more maintenance when changes occur.

- Use of aggregated keywords may simplify development but complicate maintenance. For example, there may be six different keywords that collectively create a record. Should a keyword that actually calls all six keywords consecutively be created to simplify that action?

- No matter how much analysis goes into the keyword language, there will often be times when new and different keywords will be needed. There are two separate domains to a keyword (i.e., the business logic behind it and the automation functionality to execute it). Therefore, a process must be created to deal with both domains.

Keyword-based test automation can significantly reduce the maintenance costs of test automation, but it is more costly, more difficult to develop, and takes more time to design correctly in order to gain the expected return on investment.

6.3 Specific Test Tools

This section contains information on tools that are likely to be used by a Technical Test Analyst beyond what is discussed in the Advanced Level Test Analyst [ISTQB_ALTA_SYL] and Foundation Level [ISTQB_FL_SYL] syllabi.

6.3.1 Fault Seeding/Fault Injection Tools

Fault seeding tools are mainly used at the code level to create single or limited types of code faults in a systematic way. These tools deliberately insert defects into the test object for the purpose of evaluating the quality of the test suites (i.e., their ability to detect the defects).

Fault injection is focused on testing any fault handling mechanism built into the test object by subjecting it to abnormal conditions. Fault injection tools deliberately supply incorrect inputs to the software to ensure the software can cope with the fault.

Both of these types of tools are generally used the Technical Test Analyst, but may also be used by the developer when testing newly developed code.

6.3.2 Performance Testing Tools

Performance test tools have two main functions:

- Load generation

- Measurement and analysis of the system response to a given load

Load generation is performed by implementing a pre-defined operational profile (see Section 4.5.4) as a script. The script may initially be captured for a single user (possibly using a record/playback tool) and is then implemented for the specified operational profile using the performance test tool. This implementation must take into account the variation of data per transaction (or sets of transactions).

Performance tools generate a load by simulating large numbers of multiple users (“virtual” users) following their designated operational profiles to generate specific volumes of input data. In comparison with individual test execution automation scripts, many performance testing scripts reproduce user interaction with the system at the communications protocol level and not by simulating user interaction via a graphical user interface. This usually reduces the number of separate “sessions” needed during the testing. Some load generation tools can also drive the application using its user interface to more closely measure response time while the system is under load.

A wide range of measurements are taken by a performance test tool to enable analysis during or after execution of the test. Typical metrics taken and reports provided include:

- Number of simulated users throughout the test

- Number and type of transactions generated by the simulated users and the arrival rate of the transactions

- Response times to particular transaction requests made by the users

- Reports and graphs of load against response times

- Reports on resource usage (e.g., usage over time with minimum and maximum values)

Significant factors to consider in the implementation of performance test tools include:

- The hardware and network bandwidth required to generate the load

- The compatibility of the tool with the communications protocol used by the system under test

- The flexibility of the tool to allow different operational profiles to be easily implemented

- The monitoring, analysis and reporting facilities required

Performance test tools are typically acquired rather than developed in-house due to the effort required to develop them. It may, however, be appropriate to develop a specific performance tool if technical restrictions prevent an available product being used, or if the load profile and facilities to be provided are relatively simple.

6.3.3 Tools for Web-Based Testing

A variety of open source and commercial specialized tools are available for web testing. The following list shows the purpose of some of the common web-based testing tools:

- Hyperlink test tools are used to scan and check that no broken or missing hyperlinks are present on a web site

- HTML and XML checkers are tools which check compliance to the HTML and XML standards of the pages that are created by a web site

- Load simulators to test how the server will react when large numbers of users connect

- Lightweight automation execution tools that work with different browsers

- Tools to scan through the server, checking for orphaned (unlinked) files

- HTML specific spell checkers

- Cascading Style Sheet (CSS) checking tools

- Tools to check for standards violations e.g., Section 508 accessibility standards in the U.S. or M/376 in Europe

- Tools that find a variety of security issues

A good source of open source web testing tools is [Web-7]. The organization behind this web site sets the standards for the Internet and it supplies a variety of tools to check for errors against those standards.

Some tools that include a web spider engine can also provide information on the size of the pages and on the time necessary to download them, and on whether a page is present or not (e.g., HTTP error 404). This provides useful information for the developer, the webmaster and the tester.

Test Analysts and Technical Test Analysts use these tools primarily during system testing.

6.3.4 Tools to Support Model-Based Testing

Model-Based Testing (MBT) is a technique whereby a formal model such as a finite state machine is used to describe the intended execution-time behavior of a software-controlled system. Commercial MBT tools (see [Utting 07]) often provide an engine that allows a user to “execute” the model. Interesting threads of execution can be saved and used as test cases. Other executable models such as Petri Nets and Statecharts also support MBT. MBT models (and tools) can be used to generate large sets of distinct execution threads.

MBT tools can help reduce the very large number of possible paths that can be generated in a model.

Testing using these tools can provide a different view of the software to be tested. This can result in the discovery of defects that might have been missed by functional testing.

6.3.5 Component Testing and Build Tools

While component testing and build automation tools are developer tools, in many instances, they are used and maintained by Technical Test Analysts, especially in the context of Agile development.

Component testing tools are often specific to the language that is used for programming a module. For example, if Java was used as the programming language, JUnit might be used to automate the unit testing. Many other languages have their own special test tools; these are collectively called xUnit frameworks. Such a framework generates test objects for each class that is created, thus simplifying the tasks that the programmer needs to do when automating the component testing.

Debugging tools facilitate manual component testing at a very low level, allowing developers and Technical Test Analysts to change variable values during execution and step through the code line by line while testing. Debugging tools are also used to help the developer isolate and identify problems in the code when a failure is reported by the test team.

Build automation tools often allow a new build to be automatically triggered any time a component is changed. After the build is completed, other tools automatically execute the component tests. This level of automation around the build process is usually seen in a continuous integration environment.

When set up correctly, this set of tools can have a very positive effect on the quality of builds being released into testing. Should a change made by a programmer introduce regression defects into the build, it will usually cause some of the automated tests to fail, triggering immediate investigation into the cause of the failures before the build is released into the test environment.

7. References

7.1 Standards

The following standards are mentioned in these respective chapters.

- ANSI/IEEE Std 1471-2000, Recommended Practice for Architectural Description of Software-Intensive Systems
Chapter 5

- IEC-61508
Chapter 2

- [ISO25000] ISO/IEC 25000:2005, Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) Chapter 4

- [ISO9126] ISO/IEC 9126-1:2001, Software Engineering – Software Product Quality Chapter 4

- [RTCA DO-178B/ED-12B]: Software Considerations in Airborne Systems and Equipment Certification, RTCA/EUROCAE ED12B.1992. Chapter 2

7.2 ISTQB Documents

- [ISTQB_AL_OVIEW]   ISTQB Advanced Level Overview, Version 2012

- [ISTQB_ALTA_SYL]   ISTQB Advanced Level Test Analyst Syllabus, Version 2012

- [ISTQB_FL_SYL]   ISTQB Foundation Level Syllabus, Version 2011

- [ISTQB_GLOSSARY]   ISTQB Glossary of Terms used in Software Testing, Version 2.2, 2012

7.3 Books

[Bass03] Len Bass, Paul Clements, Rick Kazman “Software Architecture in Practice (2nd edition) ”, Addison-Wesley 2003] ISBN 0-321-15495-9

[Bath08] Graham Bath, Judy McKay, “The Software Test Engineer’s Handbook”, Rocky Nook, 2008, ISBN 978-1-933952-24-6

[Beizer90] Boris Beizer, “Software Testing Techniques Second Edition”, International Thomson Computer Press, 1990, ISBN 1-8503-2880-3

[Beizer95] Boris Beizer, “Black-box Testing”, John Wiley & Sons, 1995, ISBN 0-471-12094-4

[Buwalda01]: Hans Buwalda, “Integrated Test Design and Automation” Addison-Wesley Longman, 2001, ISBN 0-201-73725-6

[Copeland03]: Lee Copeland, “A Practitioner’s Guide to Software Test Design”, Artech House, 2003, ISBN 1-58053-791-X

[Gamma94] Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, 1994, ISBN 0-201-63361-2

[Jorgensen07]: Paul C. Jorgensen, “Software Testing, a Craftsman’s Approach third edition”, CRC press, 2007, ISBN-13:978-0-8493-7475-3

[Kaner02]: Cem Kaner, James Bach, Bret Pettichord; “Lessons Learned in Software Testing”; Wiley, 2002, ISBN: 0-471-08112-4

[Koomen06]: Tim Koomen, Leo van der Aalst, Bart Broekman, Michael Vroon, “TMap Next for result-driven testing”; UTN Publishers, 2006, ISBN: 90-72194-79-9

[McCabe76] Thomas J. McCabe, “A Complexity Measure”, IEEE Transactions on Software Engineering, Vol. SE-2, No. 4, December 1976. PP 308-320

[NIST96] Arthur H. Watson and Thomas J. McCabe, “Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric”, NIST Special Publication 500-235, Prepared under NIST Contract 43NANB517266, September 1996.

[Splaine01]: Steven Splaine, Stefan P. Jaskiel, “The Web-Testing Handbook”, STQE Publishing, 2001, ISBN 0-970-43630-0

[Utting 07] Mark Utting, Bruno Legeard, “Practical Model-Based Testing: A Tools Approach ”, Morgan-Kaufmann, 2007, ISBN: 978-0-12-372501-1

[Whittaker04]: James Whittaker and Herbert Thompson, “How to Break Software Security”, Pearson / Addison-Wesley, 2004, ISBN 0-321-19433-0

[Wiegers02] Karl Wiegers, “Peer Reviews in Software: A Practical Guide”, Addison-Wesley, 2002, ISBN 0-201-73485-0

7.4 Other References

The following references point to information available on the Internet. Even though these references were checked at the time of publication of this Advanced Level Syllabus, the ISTQB can not be held responsible if the references are not available anymore.

[Web-1] www.testingstandards.co.uk

[Web-2] http://www.nist.gov NIST National Institute of Standards and Technology,

[Web-3] http://www.codeproject.com/KB/architecture/SWArchitectureReview.aspx

[Web-4] http://portal.acm.org/citation.cfm?id=308798

[Web-5] http://www.processimpact.com/pr_goodies.shtml

[Web-6] http://www.ifsq.org

[Web-7] http://www.W3C.org

Chapter 4: [Web-1], [Web-2]

Chapter 5: [Web-3], [Web-4], [Web-5], [Web-6]

Chapter 6: [Web-7]

8. Index

action word-driven, 43

adaptability, 25

adaptability testing, 36

analyzability, 25, 35

anti-pattern, 37, 39

Application Programming Interface (API), 16

architectural reviews, 39

atomic condition, 11, 12

attack, 30

backup and restore, 31

benchmark, 27

changeability, 25, 35

client/server, 16

code reviews, 39

co-existence, 25

co-existence/compatibility testing, 36

cohesion, 21

condition testing, 11, 12

control flow analysis, 18, 19

control flow coverage, 13

control flow graph, 19

control flow testing, 11

coupled, 14

coupling, 21

cyclomatic complexity, 18, 19

data flow analysis, 18, 19

data security considerations, 28

data-driven, 43

data-driven testing, 41

decision condition testing, 11, 13

decision predicates, 12

decision testing, 13

definition-use pairs, 18, 20

du-path, 20

dynamic analysis, 18, 22

memory leaks, 23

overview, 22

performance, 24

wild pointers, 23

dynamic maintainability testing, 35

efficiency, 25

failover, 31

installability, 25, 35

keyword-driven, 41, 43

load testing, 32

maintainability, 20, 25

maintainability testing, 34

master test plan, 27

maturity, 25

MC/DC, 13

McCabe’s design predicate, 22

memory leak, 18

metrics

performance, 24

modified condition/decision coverage (MC/DC), 13

MTBF, 30

MTTR, 30

multiple condition coverage, 14

multiple condition testing, 11

neighborhood integration testing, 18, 22

OAT, 31

operational acceptance test, 25, 31

operational profile, 25, 32, 33

organizational considerations, 28

pairwise integration testing, 18, 21

path segments, 15

path testing, 11, 15

performance, 25

performance test planning, 33

performance test specification, 33

performance testing, 32

portability testing, 25, 35

product quality characteristics, 26

product risk, 8

quality attributes for technical testing, 25

record/playback tool, 41, 43

recoverability, 25

recoverability testing, 31

redundant dissimilar software, 31

reliability, 25

reliability growth model, 25

reliability test planning, 31

reliability test specification, 32

reliability testing, 30

remote procedure calls (RPC), 16

replaceability, 25

replaceability testing, 36

required tooling, 27

resource utilization, 25

reviews, 37

checklists, 38

risk analysis, 8

risk assessment, 8, 9

risk identification, 8, 9

risk level, 8

risk mitigation, 8, 10

risk-based testing, 8

robustness, 25

Safety Integrity Level (SIL), 17

scalability testing, 33

security, 25

buffer overflow, 29

cross-site scripting, 29

denial of service, 29

logic bombs, 29

man in the middle, 29

security test planning, 29

security test specification, 29

security testing, 28

service-oriented architectures (SOA), 16

short-circuiting, 11, 14

simulators, 27

stability, 25, 35

stakeholder requirements, 27

standards

DO-178B, 17

ED-12B, 17

IEC-61508, 17

ISO 25000, 26

ISO 9126, 26, 32, 35

statement testing, 11

static analysis, 18, 19, 20

call graph, 21

tools, 20

stress testing, 32

structure-based technique, 11

systems of systems, 16

test automation project, 42

test environment, 28

test for robustness, 30

test of resource utilization, 34

test tools

build automation, 47

component testing, 47

debugging, 41, 47

fault injection, 45

fault seeding, 41, 45

hyperlink verification, 41, 46

integration & information interchange, 42

model-based testing, 47

performance, 41, 46

static analyzer, 41

test execution, 41

test management, 41, 45

unit testing, 47

web tools, 46

testability, 25, 35

virtual users, 46

wild pointer, 18

Standard glossary of terms used in Software Testing

Version 2.2 (dd. October 19th, 2012)

Produced by the ‘Glossary Working Party’

International Software Testing Qualifications Board

Editor: Erik van Veenendaal (The Netherlands)

Copyright Notice

This document may be copied in its entirety, or extracts made, if the source is acknowledged.

Copyright © 2012, International Software Testing Qualifications Board (hereinafter called ISTQB®).

Contributors

Rex Black (USA)

Josie Crawford (Australia)

Enst Düring (Norway)

Sigrid Eldh (Sweden)

Isabel Evans (UK)

Simon Frankish (UK)

David Fuller (Australia)

Annu George (India)

Dorothy Graham (UK)

Mats Grindal (Sweden)

Matthias Hamburg (Germany)

Julian Harty (UK)

David Hayman (UK)

Bernard Homes (France)

Ian Howles (UK)

Juha Itkonen (Finland)

Paul Jorgensen (USA)

Vipul Kocher (India)

Gerard Kruijff (The Netherlands)

Fernando Lamas de Oliveira (Portugal)

Tilo Linz (Germany)

Gustavo Marquez Sosa (Spain)

Judy McKay (US)

Don Mills (UK)

Peter Morgan (UK)

Thomas Müller (Switzerland)

Avi Ofer (Israel)

Ana Paiva (Portugal)

Dale Perry (USA)

Ander Pettersson (Sweden)

Horst Pohlmann (Germany)

Juha Pomppu (Finland)

Meile Posthuma (The Netherlands)

Erkki Pöyhönen (Finland)

Maaret Pyhäjärvi (Finland)

Andy Redwood (UK)

Stuart Reid (UK)

Piet de Roo (The Netherlands)

Steve Sampson (UK)

Shane Saunders (UK)

Hans Schaefer (Norway)

Jurriën Seubers (The Netherlands)

Dave Sherratt (UK)

Mike Smith (UK)

Andreas Spillner (Germany)

Lucjan Stapp (Poland)

Richard Taylor (UK)

Geoff Thompson (UK)

Stephanie Ulrich (Germany)

Matti Vuori (Finland)

Gearrel Welvaart (The Netherlands)

Paul Weymouth (UK)

Pete Williams (UK)

Change History

Version 2.2 d.d. MM-DD-2012
This new version has been developed to support the Advanced level syllabi (Version 2012) and the Expert level syllabus Test Management (Version 2011). In addition a new of change request have been implemented in version 2.2 of the ISTQB Glossary.

New terms added:

Terms changed;

- actor

- action word driven testing

- analytical testing

- accuracy

- anti-pattern

- agile testing

- API (Application Programming Interface) testing

- big-bang testing

- atomic condition

- Capability Maturity Model (CMM)

- attack-based testing

- Capability Maturity Model Integration (CMMI)

- combinatorial testing

- classification tree method

- CMMI

- Commercial Off-The-Shelft software

- confidence interval

- condition

- consultative testing

- condition determination coverage

- control chart

- condition determination testing

- control flow testing

- critical success factor

- convergence metric

- Critical Testing Processes

- custom tool

- cyclomatic complexity

- data quality

- daily build

- defect category

- data-driven testing

- defect management committee

- data flow testing

- defect triage committee

- dd-path

- defect type

- defect-based technique

- domain analysis

- defect-based test design technique

- effectiveness

- definition-use pair

- embedded iterative development model

- efficiency

- experience-based testing

- elementary comparison testing

- factory acceptance testing

- extreme programming

- failover testing

- fault seeding

- fault injection

- heuristic evaluation

- feature-driven development

- keyword-driven testing

- hardware-software integration testing

- lead assessor

- insourced testing

- load testing tool

- man in the middle attack

- maturity

- methodical testing

- mind map

- model-based testing

- modified condition decision coverage

- Myers-Briggs Type Indicator (MBTI)

- modified condition decision testing

- neighborhood integration testing

- modified multiple condition coverage

- open source tool

- multiple condition coverage

- operational profiling

- performance profiling

- outsourced testing

- resumption criteria

- pairwise integration testing

- root cause

- phase containment

- software quality

- planning poker

- SPI

- predicate

- static analysis

- process-compliant testing

- static testing

- quality control

- stress testing

- quality risk

- technical review

- RACI matrix

- test case specification

- reactive testing

- test design specification

- regression-averse testing

- test execution schedule

- resumption requirements

- Test Process Improvement (TPI)

- review plan

- Test Maturity Model (TMM)

- risk assessment

- Test Maturity Model integration

- risk impact

- test procedure specification

- risk likelihood

- testable requirement

- Shewhart chart

- thread testing

- short-circuiting

- Total Quality Management

- S.M.A.R.T. goal methodology

- software integrity level

- standard-compliant testing

- structure-based technique

- SUMI

- test architect

- test data management

- test director

- test mission

- three point estimation

- TMMi

- TPI Next

- user story

- user story testing

- WAMMI

- Website Analysis and MeasureMent Inventory (WAMMI)

Table of Contents

Foreword

1. Introduction

2. Scope

3. Glossary structure

4. Trademarks

5. Definitions

A

B

C

D

E

F

G

H

I

K

L

M

N

O

P

Q

R

S

T

U

V

W

Annex A References

Foreword

In compiling this glossary the working party has sought the views and comments of as broad a spectrum of opinion as possible in industry, commerce and government bodies and organizations, with the aim of producing an international testing standard which would gain acceptance in as wide a field as possible. Total agreement will rarely, if ever, be achieved in compiling a document of this nature. Contributions to this glossary have been received from the testing communities in Australia, Belgium, Finland, France, Germany, India, Israel, The Netherlands, Norway, Portugal, Spain, Sweden, Switzerland, United Kingdom, and USA.

Many software testers have used BS 7925-1, the British Standard Glossary of Software Testing Terms, since its original publication in 1998. The standard was initially developed with a bias towards component testing, but, since its publication, many comments and proposals for new definitions have been submitted to both improve and expand the standard to cover a wider range of software testing. The ISTQB testing glossary has incorporated many of these suggested updates. It is used as a reference document for the International Software Testing Qualification Board (ISTQB) software testing qualification scheme.

The ISTQB Glossary has two main objectives:

-   Support the understanding of ISTQB syllabi by defining the terms used in the various syllabi

-   Support communication within the international testing community and with its stakeholders by providing a standard testing vocabulary.

ISTQB National or Regional Boards can use the ISTQB Glossary, to translate into their local language. These boards may adapt the ISTQB Glossary to their particular language needs.

1. Introduction

Much time and effort is wasted both within and between industry, commerce, government and professional and academic institutions when ambiguities arise as a result of the inability to differentiate adequately between such terms as ‘statement coverage’ and ‘decision coverage’; ‘test suite’, ‘test specification’ and ‘test plan’ and similar terms which form an interface between various sectors of society. Moreover, the professional or technical use of these terms is often at variance, with different meanings attributed to them.

2. Scope

This document presents concepts, terms and definitions designed to aid communication in (software) testing and related disciplines.

3. Glossary structure

Arrangement

The glossary has been arranged in a single section of definitions ordered alphabetically. Some terms are preferred to other synonymous ones, in which case, the definition of the preferred term appears, with the synonymous ones referring to that. For example structural testing refers to white box testing. For synonyms, the “See” indicator is used

“See also” cross-references are also used. They assist the user to quickly navigate to the right index term. “See also” cross-references are constructed for relationships such as broader term to a narrower term, and overlapping meaning between two terms.

Keywords

The ISTQB Glossary contains many terms for different reasons. Some are provided to “just” support the reader of an ISTQB syllabus in understanding the text. Some are there because the term was used in a previous version of a syllabus and the principle of backwards compatibility is being applied. However, probably the most important terms are the (examinable) keywords that are explicitly identified by the various ISTQB syllabi. An important user group of these keywords are the (test)professional who are preparing for an ISTQB exam. To support them, the keywords that they need to understand for a particular exam are indicated in this glossary. Note that the principle of inheritance is applicable, e.g., at an ISTQB Advanced exam one still needs to understand all ISTQB Foundation keywords. The keywords are indicated in the following manner:

F:

Keyword ISTQB Foundation syllabus

ATM:

Keyword ISTQB Advanced – Test Management syllabus

ATA:

Keyword ISTQB Advanced – Test Analyst syllabus

ATT:

Keyword ISTQB Advanced – Technical Test Analyst syllabus

EITP:

Keyword ISTQB Expert – Improving the Testing Process syllabus

ETM:

Keyword ISTQB Expert – Test Management syllabus.

Note that if a keyword is identified in a syllabus, but is not the preferred term according the glossary both the keyword and the term it refers to (using the “See” indicator) are labeled with the appropriate syllabus indicator.

References

In this glossary references, are used in two ways.

-  Square brackets without the addition of “after”, e.g., [ISO 9126], mean that the exact text of the reference is used.

-  In case a definition from a reference has been adapted to the context of the ISTQB Glossary by minor changes, the addition “after” is used, e.g., [After ISO 9126].

4. Trademarks

In this document the following trademarks are used:

-   CMMI and IDEAL are registered trademarks of Carnegie Mellon University

-   EFQM is a registered trademark of the EFQM Foundation

-   Rational Unified Process is a registered trademark of Rational Software Corporation

-   STEP is a registered trademark of Software Quality Engineering

-   TMap, TPA and TPI Next are registered trademarks of Sogeti Nederland BV

-   TMMi is a registered trademark of the TMMi Foundation

5. Definitions

A

abstract test case: See high level test case.

acceptance: See acceptance testing.

acceptance criteria: The exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. [IEEE 610]

acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610]

ATA

accessibility testing: Testing to determine the ease by which users with disabilities can use a component or system. [Gerrard]

accuracy: The capability of the software product to provide the right or agreed results or effects with the needed degree of precision. [ISO 9126] See also functionality.

ATA

accuracy testing: The process of testing to determine the accuracy of a software product

EITP

acting (IDEAL): The phase within the IDEAL model where the improvements are developed, put into practice, and deployed across the organization. The acting phase consists of the activities: create solution, pilot/test solution, refine solution and implement solution. See also IDEAL.

action word driven testing: See keyword-driven testing

actor: User or any other person or system that interacts with the system under test in a specific way.

actual outcome: See actual result.

actual result: The behavior produced/observed when a component or system is tested.

ad hoc review: See informal review.

ad hoc testing: Testing carried out informally; no formal test preparation takes place, no recognized test design technique is used, there are no expectations for results and arbitrariness guides the test execution activity.

ATT

adaptability: The capability of the software product to be adapted for different specified environments without applying actions or means other than those provided for this purpose for the software considered. [ISO 9126] See also portability.

agile manifesto: A statement on the values that underpin agile software development. The values are:

- individuals and interactions over processes and tools

- working software over comprehensive documentation

- customer collaboration over contract negotiation

- responding to change over following a plan.

EITP

agile software development: A group of software development methodologies based on iterative incremental development, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams.

EITP

agile testing: Testing practice for a project using agile software development methodologies, incorporating techniques and methods, such as extreme programming (XP), treating development as the customer of testing and emphasizing the test-first design paradigm. See also test driven development.

algorithm test: [TMap] See branch testing.

F

alpha testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers’ site, but outside the development organization. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing.

ETM

analytical testing: Testing based on a systematic analysis of e.g., product risks or requirements.

ATT

analyzability: The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified. [ISO 9126] See also maintainability.

analyzer: See static analyzer.

ATM

anomaly: Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc. or from someone’s perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation. [IEEE 1044] See also bug, defect, deviation, error, fault, failure, incident, problem.

ATT

anti-pattern: Repeated action, process, structure or reusable solution that initially appears to be beneficial and is commonly used but is ineffective and/or counterproductive in practice.

API (Application Programming Interface) testing: Testing the code which enables communication between different processes, programs and/or systems. API testing often involves negative testing, e.g., to validate the robustness of error handling. See also interface testing.

arc testing: See branch testing.

EITP

assessment report: A document summarizing the assessment results, e.g. conclusions, recommendations and findings. See also process assessment.

EITP

assessor: A person who conducts an assessment; any member of an assessment team.

ATT

atomic condition: A condition that cannot be decomposed, i.e., a condition that does not contain two or more single conditions joined by a logical operator (AND, OR, XOR).

F

attack: Directed and focused attempt to evaluate the quality, especially reliability, of a test object by attempting to force specific failures to occur. See also negative testing.

attack-based testing: An experience-based testing technique that uses software attacks to induce failures, particularly security related failures. See also attack.

ATA

attractiveness: The capability of the software product to be attractive to the user. [ISO 9126] See also usability.

ATM

audit: An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify:
(1) the form or content of the products to be produced
(2) the process by which the products shall be produced
(3) how compliance to standards or guidelines shall be measured. [IEEE 1028]

audit trail: A path by which the original input to a process (e.g. data) can be traced back through the process, taking the process output as a starting point. This facilitates defect analysis and allows a process audit to be carried out. [After TMap]

automated testware: Testware used in automated testing, such as tool scripts.

availability: The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage. [IEEE 610]

B

back-to-back testing: Testing in which two or more variants of a component or system are executed with the same inputs, the outputs compared, and analyzed in cases of discrepancies. [IEEE 610]

EITP

balanced scorecard: A strategic tool for measuring whether the operational activities of a company are aligned with its objectives in terms of business vision and strategy. See also corporate dashboard, scorecard.

baseline: A specification or software product that has been formally reviewed or agreed upon, that thereafter serves as the basis for further development, and that can be changed only through a formal change control process. [After IEEE 610]

basic block: A sequence of one or more consecutive executable statements containing no branches. Note: A node in a control flow graph represents a basic block.

basis test set: A set of test cases derived from the internal structure of a component or specification to ensure that 100% of a specified coverage criterion will be achieved.

bebugging: [Abbott] See fault seeding.

behavior: The response of a component or system to a set of input values and preconditions.

benchmark test: (1) A standard against which measurements or comparisons can be made. (2) A test that is be used to compare components or systems to each other or to a standard as in (1). [After IEEE 610]

bespoke software: Software developed specifically for a set of users or customers. The opposite is off-the-shelf software.

best practice: A superior method or innovative practice that contributes to the improved performance of an organization under given context, usually recognized as ‘best’ by other peer organizations.

F

beta testing: Operational testing by potential and/or existing users/customers at an external site not otherwise involved with the developers, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes. Beta testing is often employed as a form of external acceptance testing for off-the-shelf software in order to acquire feedback from the market.

big-bang testing: An integration testing approach in which software elements, hardware elements, or both are combined all at once into a component or an overall system, rather than in stages. [After IEEE 610] See also integration testing.

black box technique: See black box test design technique.

F

ATA

black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

F

black box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system.

blocked test case: A test case that cannot be executed because the preconditions for its execution are not fulfilled.

bottom-up testing: An incremental approach to integration testing where the lowest level components are tested first, and then used to facilitate the testing of higher level components. This process is repeated until the component at the top of the hierarchy is tested. See also integration testing.

boundary value: An input value or output value which is on the edge of an equivalence partition or at the smallest incremental distance on either side of an edge, for example the minimum or maximum value of a range.

F

ATA

boundary value analysis: A black box test design technique in which test cases are designed based on boundary values. See also boundary value.

boundary value coverage: The percentage of boundary values that have been exercised by a test suite.

boundary value testing: See boundary value analysis.

branch: A basic block that can be selected for execution based on a program construct in which one of two or more alternative program paths is available, e.g. case, jump, go to, if-then-else.

branch condition: See condition.

branch condition combination coverage: See multiple condition coverage.

branch condition combination testing: See multiple condition testing.

branch condition coverage: See condition coverage.

branch coverage: The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage.

branch testing: A white box test design technique in which test cases are designed to execute branches.

buffer: A device or storage area used to store data temporarily for differences in rates of data flow, time or occurrence of events, or amounts of data that can be handled by the devices or processes involved in the transfer or use of the data. [IEEE 610]

buffer overflow: A memory access failure due to the attempt by a process to store data beyond the boundaries of a fixed length buffer, resulting in overwriting of adjacent memory areas or the raising of an overflow exception. See also buffer.

F

bug: See defect.

bug report: See defect report.

bug taxonomy: See defect taxonomy.

bug tracking tool: See defect management tool.

business process-based testing: An approach to testing in which test cases are designed based on descriptions and/or knowledge of business processes.

C

call graph: An abstract representation of calling relationships between subroutines in a program.

ATM

EITP

Capability Maturity Model Integration: A framework that describes the key elements of an effective product development and maintenance process. The Capability Maturity Model Integration covers best-practices for planning, engineering and managing product development and maintenance. [CMMI]

ATT

capture/playback tool: A type of test execution tool where inputs are recorded during manual testing in order to generate automated test scripts that can be executed later (i.e. replayed). These tools are often used to support automated regression testing.

capture/replay tool: See capture/playback tool.

CASE: Acronym for Computer Aided Software Engineering.

CAST: Acronym for Computer Aided Software Testing. See also test automation.

EITP

causal analysis: The analysis of defects to determine their root cause. [CMMI]

cause-effect analysis: See cause-effect graphing.

cause-effect decision table: See decision table.

EITP

cause-effect diagram: A graphical representation used to organize and display the interrelationships of various possible root causes of a problem. Possible causes of a real or potential defect or failure are organized in categories and subcategories in a horizontal tree-structure, with the (potential) defect or failure as the root node. [After Juran]

EITP

cause-effect graph: A graphical representation of inputs and/or stimuli (causes) with their associated outputs (effects), which can be used to design test cases.

ATA

cause-effect graphing: A black box test design technique in which test cases are designed from cause-effect graphs. [BS 7925/2]

certification: The process of confirming that a component, system or person complies with its specified requirements, e.g. by passing an exam.

change control: See configuration control.

change control board: See configuration control board.

EITP

change management: (1) A structured approach to transitioning individuals, and organizations from a current state to a desired future state. (2) Controlled way to effect a change, or a proposed change, to a product or service. See also configuration management.

ATT

changeability: The capability of the software product to enable specified modifications to be implemented. [ISO 9126] See also maintainability.

charter: See test charter.

checker: See reviewer.

ATA

checklist-based testing: An experience-based test design technique whereby the experienced tester uses a high-level list of items to be noted, checked, or remembered, or a set of rules or criteria against which a product has to be verified.

Chow’s coverage metrics: See N-switch coverage. [Chow]

classification tree: A tree showing equivalence partitions hierarchically ordered, which is used to design test cases in the classification tree method. See also classification tree method.

ATA

classification tree method: A black box test design technique in which test cases, described by means of a classification tree, are designed to execute combinations of representatives of input and/or output domains. [Grochtmann]

clear-box testing: See white-box testing.

ATM

EITP

CMMI: See Capability Maturity Model Integration.

code: Computer instructions and data definitions expressed in a programming language or in a form output by an assembler, compiler or other translator. [IEEE 610]

code analyzer: See static code analyzer.

F

code coverage: An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage.

code-based testing: See white box testing.

EITP

codependent behavior: Excessive emotional or psychological dependence on another person, specifically in trying to change that person’s current (undesirable) behavior while supporting them in continuing that behavior. For example, in software testing, complaining about late delivery to test and yet enjoying the necessary “heroism” working additional hours to make up time when delivery is running late, therefore reinforcing the lateness.

ATT

co-existence: The capability of the software product to co-exist with other independent software in a common environment sharing common resources. [ISO 9126] See also portability.

ATA

combinatorial testing: A means to identify a suitable subset of test combinations to achieve a predetermined level of coverage when testing an object with multiple parameters and where those parameters themselves each have several values, which gives rise to more combinations than are feasible to test in the time allowed. See also classification tree method, pairwise testing, orthogonal array testing.

F

Commercial Off-The-Shelf software: See off-the-shelf software.

comparator: See test comparator.

compatibility testing: See interoperability testing.

F

compiler: A software tool that translates programs expressed in a high order language into their machine language equivalents. [IEEE 610]

complete testing: See exhaustive testing.

completion criteria: See exit criteria.

F

complexity: The degree to which a component or system has a design and/or internal structure that is difficult to understand, maintain and verify. See also cyclomatic complexity.

compliance: The capability of the software product to adhere to standards, conventions or regulations in laws and similar prescriptions. [ISO 9126]

compliance testing: The process of testing to determine the compliance of the component or system.

component: A minimal software item that can be tested in isolation.

component integration testing: Testing performed to expose defects in the interfaces and interaction between integrated components.

component specification: A description of a component’s function in terms of its output values for specified input values under specified conditions, and required non-functional behavior (e.g. resource-utilization).

F

component testing: The testing of individual software components. [After IEEE 610]

compound condition: Two or more single conditions joined by means of a logical operator (AND, OR or XOR), e.g. ‘A>B AND C>1000’.

ATA

concrete test case: See low level test case.

concurrency testing: Testing to determine how the occurrence of two or more activities within the same interval of time, achieved either by interleaving the activities or by simultaneous execution, is handled by the component or system. [After IEEE 610]

condition: A logical expression that can be evaluated as True or False, e.g. A>B. See also condition testing.

condition combination coverage: See multiple condition coverage.

condition combination testing: See multiple condition testing.

condition coverage: The percentage of condition outcomes that have been exercised by a test suite. 100% condition coverage requires each single condition in every decision statement to be tested as True and False.

condition determination coverage: See modified condition decision coverage.

condition determination testing: See modified condition decision testing.

condition outcome: The evaluation of a condition to True or False.

ATT

condition testing: A white box test design technique in which test cases are designed to execute condition outcomes.

ETM

confidence interval: In managing project risks, the period of time within which a contingency action must be implemented in order to be effective in reducing the impact of the risk.

confidence test: See smoke test.

configuration: The composition of a component or system as defined by the number, nature, and interconnections of its constituent parts.

configuration auditing: The function to check on the contents of libraries of configuration items, e.g. for standards compliance. [IEEE 610]

configuration control: An element of configuration management, consisting of the evaluation, co-ordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. [IEEE 610]

configuration control board (CCB): A group of people responsible for evaluating and approving or disapproving proposed changes to configuration items, and for ensuring implementation of approved changes. [IEEE 610]

configuration identification: An element of configuration management, consisting of selecting the configuration items for a system and recording their functional and physical characteristics in technical documentation. [IEEE 610]

configuration item: An aggregation of hardware, software or both, that is designated for configuration management and treated as a single entity in the configuration management process. [IEEE 610]

F

configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements. [IEEE 610]

F

configuration management tool: A tool that provides support for the identification and control of configuration items, their status over changes and versions, and the release of baselines consisting of configuration items.

configuration testing: See portability testing.

F

confirmation testing: See re-testing.

conformance testing: See compliance testing.

consistency: The degree of uniformity, standardization, and freedom from contradiction among the documents or parts of a component or system. [IEEE 610]

ETM

consultative testing: Testing driven by the advice and guidance of appropriate experts from outside the test team (e.g., technology experts and/or business domain experts).

EITP

content-based model: A process model providing a detailed description of good engineering practices, e.g. test practices.

EITP

continuous representation: A capability maturity model structure wherein capability levels provide a recommended order for approaching process improvement within specified process areas. [CMMI]

ETM

control chart: A statistical process control tool used to monitor a process and determine whether it is statistically controlled. It graphically depicts the average value and the upper and lower control limits (the highest and lowest values) of a process.

F

control flow: A sequence of events (paths) in the execution through a component or system.

ATT

control flow analysis: A form of static analysis based on a representation of unique paths (sequences of events) in the execution through a component or system. Control flow analysis evaluates the integrity of control flow structures, looking for possible control flow anomalies such as closed loops or logically unreachable process steps.

control flow graph: An abstract representation of all possible sequences of events (paths) in the execution through a component or system.

control flow path: See path.

ATT

control flow testing: An approach to structure-based testing in which test cases are designed to execute specific sequences of events. Various techniques exist for control flow testing, e.g., decision testing, condition testing, and path testing, that each have their specific approach and level of control flow coverage. See also decision testing, condition testing, path testing.

ETM

convergence metric: A metric that shows progress toward a defined criterion, e.g., convergence of the total number of test executed to the total number of tests planned for execution.

conversion testing: Testing of software used to convert data from existing systems for use in replacement systems.

EITP

corporate dashboard: A dashboard-style representation of the status of corporate performance data. See also balanced scorecard, dashboard.

cost of quality: The total costs incurred on quality activities and issues and often split into prevention costs, appraisal costs, internal failure costs and external failure costs.

F

COTS: Acronym for Commercial Off-The-Shelf software. See off-the-shelf software.

coverage: The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite.

coverage analysis: Measurement of achieved coverage to a specified coverage item during test execution referring to predetermined criteria to determine whether additional testing is required and if so, which test cases are needed.

coverage item: An entity or property used as a basis for test coverage, e.g. equivalence partitions or code statements.

coverage measurement tool: See coverage tool.

F

coverage tool: A tool that provides objective measures of what structural elements, e.g. statements, branches have been exercised by a test suite.

EITP

critical success factor: An element necessary for an organization or project to achieve its mission. Critical success factors are the critical factors or activities required for ensuring the success.

ATM

EITP

Critical Testing Processes: A content-based model for test process improvement built around twelve critical processes. These include highly visible processes, by which peers and management judge competence and mission-critical processes in which performance affects the company’s profits and reputation. See also content-based model.

ATM

EITP

CTP: See Critical Testing Processes.

custom software: See bespoke software.

ATM

custom tool: A software tool developed specifically for a set of users or customers.

ATT

cyclomatic complexity: The maximum number of linear, independent paths through a program. Cyclomatic complexity may be computed as: L – N + 2P, where

- L = the number of edges/links in a graph

- N = the number of nodes in a graph

- P = the number of disconnected parts of the graph (e.g. a called graph or subroutine) [After McCabe]

cyclomatic number: See cyclomatic complexity.

D

daily build: A development activity whereby a complete system is compiled and linked every day (often overnight), so that a consistent system is available at any time including all latest changes.

ETM

dashboard: A representation of dynamic measurements of operational performance for some organization or activity, using metrics represented via metaphores such as visual ‘dials’, ‘counters’, and other devices resembling those on the dashboard of an automobile, so that the effects of events or activities can be easily understood and related to operational goals. See also corporate dashboard, scorecard.

data definition: An executable statement where a variable is assigned a value.

F

ATT

data-driven testing: A scripting technique that stores test input and expected results in a table or spreadsheet, so that a single control script can execute all of the tests in the table. Data-driven testing is often used to support the application of test execution tools such as capture/playback tools. [Fewster and Graham] See also keyword-driven testing.

F

data flow: An abstract representation of the sequence and possible changes of the state of data objects, where the state of an object is any of: creation, usage, or destruction. [Beizer]

ATT

data flow analysis: A form of static analysis based on the definition and usage of variables.

data flow coverage: The percentage of definition-use pairs that have been exercised by a test suite.

data flow testing: A white box test design technique in which test cases are designed to execute definition-use pairs of variables.

data integrity testing: See database integrity testing.

data quality; An attribute of data that indicates correctness with respect to some pre-defined criteria, e.g., business expectations, requirements on data integrity, data consistency.

database integrity testing: Testing the methods and processes used to access and manage the data(base), to ensure access methods, processes and data rules function as expected and that during access to the database, data is not corrupted or unexpectedly deleted, updated or created.

dd-path: A path between two decisions of an algorithm, or two decision nodes of a corresponding graph, that includes no other decisions. See also path.

dead code: See unreachable code.

debugger: See debugging tool.

F

debugging: The process of finding, analyzing and removing the causes of failures in software.

F

ATT

debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs and find the corresponding defect. Debuggers enable programmers to execute programs step by step, to halt a program at any program statement and to set and examine program variables.

decision: A program point at which the control flow has two or more alternative routes. A node with two or more links to separate branches.

decision condition coverage: The percentage of all condition outcomes and decision outcomes that have been exercised by a test suite. 100% decision condition coverage implies both 100% condition coverage and 100% decision coverage.

ATT

decision condition testing: A white box test design technique in which test cases are designed to execute condition outcomes and decision outcomes.

F

decision coverage: The percentage of decision outcomes that have been exercised by a test suite. 100% decision coverage implies both 100% branch coverage and 100% statement coverage.

decision outcome: The result of a decision (which therefore determines the branches to be taken).

decision table: A table showing combinations of inputs and/or stimuli (causes) with their associated outputs and/or actions (effects), which can be used to design test cases.

F

ATA

decision table testing: A black box test design technique in which test cases are designed to execute the combinations of inputs and/or stimuli (causes) shown in a decision table. [Veenendaal04] See also decision table.

decision testing: A white box test design technique in which test cases are designed to execute decision outcomes.

F

ATM

defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

ATA

defect-based technique: See defect-based test design technique.

ATA

defect-based test design technique: A procedure to derive and/or select test cases targeted at one or more defect categories, with tests being developed from what is known about the specific defect category. See also defect taxonomy.

defect category: See defect type.

F

defect density: The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, e.g. lines-of-code, number of classes or function points).

EITP

Defect Detection Percentage (DDP): The number of defects found by a test phase, divided by the number found by that test phase and any other means afterwards.

defect management: The process of recognizing, investigating, taking action and disposing of defects. It involves recording defects, classifying them and identifying the impact. [After IEEE 1044]

ATM

defect management committee: A cross-functional team of stakeholders who manage reported defects from initial detection to ultimate resolution (defect removal, defect deferral, or report cancellation). In some cases, the same team as the configuration control board. See also configuration control board.

defect management tool: A tool that facilitates the recording and status tracking of defects and changes. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of defects and provide reporting facilities. See also incident management tool.

defect masking: An occurrence in which one defect prevents the detection of another. [After IEEE 610]

defect report: A document reporting on any flaw in a component or system that can cause the component or system to fail to perform its required function. [After IEEE 829]

ATA

defect taxonomy: A system of (hierarchical) categories designed to be a useful aid for reproducibly classifying defects.

defect tracking tool: See defect management tool.

ATM

defect triage committee: See defect management committee.

defect type: An element in a taxonomy of defects. Defect taxonomies can be identified with respect to a variety of considerations, including, but not limited to:

- Phase or development activity in which the defect is created, e.g., a specification error or a coding error

- Characterization of defects, e.g., an “off-by-one” defect

- Incorrectness, e.g., an incorrect relational operator, a programming language syntax error, or an invalid assumption

- Performance issues, e.g., excessive execution time, insufficient availability.

ATT

definition-use pair: The association of a definition of a variable with the subsequent use of that variable. Variable uses include computational (e.g. multiplication) or to direct the execution of a path (“predicate” use).

deliverable: Any (work) product that must be delivered to someone other than the (work) product’s author.

EITP

Deming cycle: An iterative four-step problem-solving process, (plan-do-check-act), typically used in process improvement. [After Deming]

design-based testing: An approach to testing in which test cases are designed based on the architecture and/or detailed design of a component or system (e.g. tests of interfaces between components or systems).

desk checking: Testing of software or a specification by manual simulation of its execution. See also static testing.

development testing: Formal or informal testing conducted during the implementation of a component or system, usually in the development environment by developers. [After IEEE 610]

deviation: See incident.

deviation report: See incident report.

EITP

diagnosing (IDEAL): The phase within the IDEAL model where it is determined where one is, relative to where one wants to be. The diagnosing phase consists of the activities: characterize current and desired states and develop recommendations. See also IDEAL.

dirty testing: See negative testing.

documentation testing: Testing the quality of the documentation, e.g. user guide or installation guide.

domain: The set from which valid input and/or output values can be selected.

ATA

domain analysis: A black box test design technique that is used to identify efficient and effective test cases when multiple variables can or should be tested together. It builds on and generalizes equivalence partitioning and boundary values analysis. See also boundary value analysis, equivalence partitioning.

F

driver: A software component or test tool that replaces a component that takes care of the control and/or the calling of a component or system. [After TMap]

ATT

dynamic analysis: The process of evaluating behavior, e.g. memory performance, CPU usage, of a system or component during execution. [After IEEE 610]

F

dynamic analysis tool: A tool that provides run-time information on the state of the software code. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic and to monitor the allocation, use and de-allocation of memory and to flag memory leaks.

dynamic comparison: Comparison of actual and expected results, performed while the software is being executed, for example by a test execution tool.

F

dynamic testing: Testing that involves the execution of the software of a component or system.

E

ATM

effectiveness: The capability of producing an intended result. See also efficiency.

ATM

ATT

efficiency: (1) The capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions. [ISO 9126]
(2) The capability of a process to produce the intended outcome, relative to the amount of resources used

efficiency testing: The process of testing to determine the efficiency of a software product.

EITP

EFQM (European Foundation for Quality Management) excellence model: A non-prescriptive framework for an organisation’s quality management system, defined and owned by the European Foundation for Quality Management, based on five ‘Enabling’ criteria (covering what an organisation does), and four ‘Results’ criteria (covering what an organisation achieves).

elementary comparison testing: A black box test design technique in which test cases are designed to execute combinations of inputs using the concept of modified condition decision coverage. [TMap]

embedded iterative development model: A development lifecycle sub-model that applies an iterative approach to detailed design, coding and testing within an overall sequential model. In this case, the high level design documents are prepared and approved for the entire project but the actual detailed design, code development and testing are conducted in iterations.

EITP

emotional intelligence: The ability, capacity, and skill to identify, assess, and manage the emotions of one’s self, of others, and of groups.

emulator: A device, computer program, or system that accepts the same inputs and produces the same outputs as a given system. [IEEE 610] See also simulator.

F

entry criteria: The set of generic and specific conditions for permitting a process to go forward with a defined task, e.g. test phase. The purpose of entry criteria is to prevent a task from starting which would entail more (wasted) effort compared to the effort needed to remove the failed entry criteria. [Gilb and Graham]

entry point: An executable statement or process step which defines a point at which a given process is intended to begin.

equivalence class: See equivalence partition.

equivalence partition: A portion of an input or output domain for which the behavior of a component or system is assumed to be the same, based on the specification.

equivalence partition coverage: The percentage of equivalence partitions that have been exercised by a test suite.

F

ATA

equivalence partitioning: A black box test design technique in which test cases are designed to execute representatives from equivalence partitions. In principle test cases are designed to cover each partition at least once.

F

error: A human action that produces an incorrect result. [After IEEE 610]

F

ATA

error guessing: A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them.

error seeding: See fault seeding.

error seeding tool: See fault seeding tool.

error tolerance: The ability of a system or component to continue normal operation despite the presence of erroneous inputs. [After IEEE 610].

EITP

establishing (IDEAL): The phase within the IDEAL model where the specifics of how an organization will reach its destination are planned. The establishing phase consists of the activities: set priorities, develop approach and plan actions. See also IDEAL.

evaluation: See testing.

exception handling: Behavior of a component or system in response to erroneous input, from either a human user or from another component or system, or to an internal failure.

executable statement: A statement which, when compiled, is translated into object code, and which will be executed procedurally when the program is running and may perform an action on data.

exercised: A program element is said to be exercised by a test case when the input value causes the execution of that element, such as a statement, decision, or other structural element.

F

exhaustive testing: A test approach in which the test suite comprises all combinations of input values and preconditions.

F

ATM

ATA

exit criteria: The set of generic and specific conditions, agreed upon with the stakeholders for permitting a process to be officially completed. The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham]

exit point: An executable statement or process step which defines a point at which a given process is intended to cease..

expected outcome: See expected result.

expected result: The behavior predicted by the specification, or another source, of the component or system under specified conditions.

ATA

experience-based technique: See experience-based test design technique.

F

ATA

experience-based test design technique: Procedure to derive and/or select test cases based on the tester’s experience, knowledge and intuition.

experience-based testing: Testing based on the tester’s experience, knowledge and intuition.

F

ATA

exploratory testing: An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. [After Bach]

EITP

extreme programming (XP): A software engineering methodology used within agile software development whereby core practices are programming in pairs, doing extensive code review, unit testing of all code, and simplicity and clarity in code. See also agile software development.

F

factory acceptance testing: Acceptance testing conducted at the site at which the product is developed and performed by employees of the supplier organization, to determine whether or not a component or system satisfies the requirements, normally including hardware as well as software. See also alfa testing.

fail: A test is deemed to fail if its actual result does not match its expected result.

failover testing: Testing by simulating failure modes or actually causing failures in a controlled environment. Following a failure, the failover mechanism is tested to ensure that data is not lost or corrupted and that any agreed service levels are maintained (e.g., function availability or response times). See also recoverability testing.

F

ATM

failure: Deviation of the component or system from its expected delivery, service or result. [After Fenton]

failure mode: The physical or functional manifestation of a failure. For example, a system in failure mode may be characterized by slow operation, incorrect outputs, or complete termination of execution. [IEEE 610]

EITP

Failure Mode and Effect Analysis (FMEA): A systematic approach to risk identification and analysis of identifying possible modes of failure and attempting to prevent their occurrence. See also Failure Mode, Effect and Criticality Analysis (FMECA).

Failure Mode, Effects, and Criticality Analysis (FMECA): An extension of FMEA, as in addition to the basic FMEA, it includes a criticality analysis, which is used to chart the probability of failure modes against the severity of their consequences. The result highlights failure modes with relatively high probability and severity of consequences, allowing remedial effort to be directed where it will produce the greatest value. See also Failure Mode and Effect Analysis (FMEA).

F

failure rate: The ratio of the number of failures of a given category to a given unit of measure, e.g. failures per unit of time, failures per number of transactions, failures per number of computer runs. [IEEE 610]

ATM

false-fail result: A test result in which a defect is reported although no such defect actually exists in the test object.

ATM

false-pass result: A test result which fails to identify the presence of a defect that is actually present in the test object.

ATM

false-positive result: See false-fail result.

ATM

false-negative result: See false-pass result.

F

fault: See defect.

F

fault attack: See attack.

fault density: See defect density.

Fault Detection Percentage (FDP): See Defect Detection Percentage (DDP).

fault injection: The process of intentionally adding defects to a system for the purpose of finding out whether the system can detect, and possibly recover from, a defect. Fault injection intended to mimic failures that might occur in the field. See also fault tolerance.

fault masking: See defect masking.

fault seeding: The process of intentionally adding defects to those already in the component or system for the purpose of monitoring the rate of detection and removal, and estimating the number of remaining defects. Fault seeding is typically part of development (pre-release) testing and can be performed at any test level (component, integration, or system). [After IEEE 610]

ATT

fault seeding tool: A tool for seeding (i.e. intentionally inserting) faults in a component or system.

fault tolerance: The capability of the software product to maintain a specified level of performance in cases of software faults (defects) or of infringement of its specified interface. [ISO 9126] See also reliability, robustness.

EITP

Fault Tree Analysis (FTA): A technique used to analyze the causes of faults (defects). The technique visually models how logical relationships between failures, human errors, and external events can combine to cause specific faults to disclose.

feasible path: A path for which a set of input values and preconditions exists which causes it to be executed.

feature: An attribute of a component or system specified or implied by requirements documentation (for example reliability, usability or design constraints). [After IEEE 1008]

ETM

feature-driven development: An iterative and incremental software development process driven from a client-valued functionality (feature) perspective. Feature-driven development is mostly used in agile software development. See also agile software development.

F

field testing: See beta testing.

finite state machine: A computational model consisting of a finite number of states and transitions between those states, possibly with accompanying actions. [IEEE 610]

finite state testing: See state transition testing.

fishbone diagram: See cause-effect diagram.

F

formal review: A review characterized by documented procedures and requirements, e.g. inspection.

frozen test basis: A test basis document that can only be amended by a formal change control process. See also baseline.

Function Point Analysis (FPA): Method aiming to measure the size of the functionality of an information system. The measurement is independent of the technology. This measurement may be used as a basis for the measurement of productivity, the estimation of the needed resources, and project control.

functional integration: An integration approach that combines the components or systems for the purpose of getting a basic functionality working early. See also integration testing.

F

functional requirement: A requirement that specifies a function that a component or system must perform. [IEEE 610]

functional test design technique: Procedure to derive and/or select test cases based on an analysis of the specification of the functionality of a component or system without reference to its internal structure. See also black box test design technique.

F

functional testing: Testing based on an analysis of the specification of the functionality of a component or system. See also black box testing.

functionality: The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. [ISO 9126]

functionality testing: The process of testing to determine the functionality of a software product.

G

glass box testing: See white box testing.

EITP

Goal Question Metric: An approach to software measurement using a three-level model: conceptual level (goal), operational level (question) and quantitative level (metric).

EITP

GQM: See Goal Question Metric.

H

hardware-software integration testing: Testing performed to expose defects in the interfaces and interaction between hardware and software components. See also integration testing.

hazard analysis: A technique used to characterize the elements of risk. The result of a hazard analysis will drive the methods used for development and testing of a system. See also risk analysis.

ATA

heuristic evaluation: A usability review technique that targets usability problems in the user interface or user interface design. With this technique, the reviewers examine the interface and judge its compliance with recognized usability principles (the “heuristics”).

ATA

high level test case: A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available. See also low level test case.

horizontal traceability: The tracing of requirements for a test level through the layers of test documentation (e.g. test plan, test design specification, test case specification and test procedure specification or test script).

hyperlink: A pointer within a web page that leads to other web pages.

ATT

hyperlink test tool: A tool used to check that no broken hyperlinks are present on a web site.

I

EITP

IDEAL: An organizational improvement model that serves as a roadmap for initiating, planning, and implementing improvement actions. The IDEAL model is named for the five phases it describes: initiating, diagnosing, establishing, acting, and learning.

F

impact analysis: The assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements.

F

incident: Any event occurring that requires investigation. [After IEEE 1008]

F

incident logging: Recording the details of any incident that occurred, e.g. during testing.

F

incident management: The process of recognizing, investigating, taking action and disposing of incidents. It involves logging incidents, classifying them and identifying the impact. [After IEEE 1044]

F

incident management tool: A tool that facilitates the recording and status tracking of incidents. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of incidents and provide reporting facilities. See also defect management tool.

F

incident report: A document reporting on any event that occurred, e.g. during the testing, which requires investigation. [After IEEE 829]

F

incremental development model: A development lifecycle where a project is broken into a series of increments, each of which delivers a portion of the functionality in the overall project requirements. The requirements are prioritized and delivered in priority order in the appropriate increment. In some (but not all) versions of this lifecycle model, each subproject follows a ‘mini V-model’ with its own design, coding and testing phases.

incremental testing: Testing where components or systems are integrated and tested one or some at a time, until all the components or systems are integrated and tested.

F

ATM

independence of testing: Separation of responsibilities, which encourages the accomplishment of objective testing. [After DO-178b]

EITP

indicator: A measure that can be used to estimate or predict another measure. [ISO 14598]

infeasible path: A path that cannot be exercised by any set of possible input values.

F

ATM

informal review: A review not based on a formal (documented) procedure.

EITP

initiating (IDEAL): The phase within the IDEAL model where the groundwork is laid for a successful improvement effort. The initiating phase consists of the activities: set context, build sponsorship and charter infrastructure. See also IDEAL.

input: A variable (whether stored within a component or outside) that is read by a component.

input domain: The set from which valid input values can be selected. See also domain.

input value: An instance of an input. See also input.

insourced testing: Testing performed by people who are co-located with the project team but are not fellow employees.

F

ATM

EITP

inspection: A type of peer review that relies on visual examination of documents to detect defects, e.g. violations of development standards and non-conformance to higher level documentation. The most formal review technique and therefore always based on a documented procedure. [After IEEE 610, IEEE 1028] See also peer review.

inspection leader: See moderator.

inspector: See reviewer.

ATT

installability: The capability of the software product to be installed in a specified environment [ISO 9126]. See also portability.

installability testing: The process of testing the installability of a software product. See also portability testing.

installation guide: Supplied instructions on any suitable media, which guides the installer through the installation process. This may be a manual guide, step-by-step procedure, installation wizard, or any other similar process description.

installation wizard: Supplied software on any suitable media, which leads the installer through the installation process. It normally runs the installation process, provides feedback on installation results, and prompts for options.

instrumentation: The insertion of additional code into the program in order to collect information about program behavior during execution, e.g. for measuring code coverage.

instrumenter: A software tool used to carry out instrumentation.

intake test: A special instance of a smoke test to decide if the component or system is ready for detailed and further testing. An intake test is typically carried out at the start of the test execution phase. See also smoke test.

F

integration: The process of combining components or systems into larger assemblies.

F

integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing.

integration testing in the large: See system integration testing.

integration testing in the small: See component integration testing.

interface testing: An integration test type that is concerned with testing the interfaces between components or systems.

interoperability: The capability of the software product to interact with one or more specified components or systems. [After ISO 9126] See also functionality.

F

ATA

interoperability testing: The process of testing to determine the interoperability of a software product. See also functionality testing.

invalid testing: Testing using input values that should be rejected by the component or system. See also error tolerance, negative testing.

ETM

Ishikawa diagram: See cause-effect diagram.

isolation testing: Testing of individual components in isolation from surrounding components, with surrounding components being simulated by stubs and drivers, if needed.

item transmittal report: See release note.

F

iterative development model: A development lifecycle where a project is broken into a usually large number of iterations. An iteration is a complete development loop resulting in a release (internal or external) of an executable product, a subset of the final product under development, which grows from iteration to iteration to become the final product.

K

key performance indicator: See performance indicator.

F

ATA

ATT

keyword-driven testing: A scripting technique that uses data files to contain not only test data and expected results, but also keywords related to the application being tested. The keywords are interpreted by special supporting scripts that are called by the control script for the test. See also data-driven testing.

L

LCSAJ: A Linear Code Sequence And Jump, consists of the following three items (conventionally identified by line numbers in a source code listing): the start of the linear sequence of executable statements, the end of the linear sequence, and the target line to which control flow is transferred at the end of the linear sequence.

LCSAJ coverage: The percentage of LCSAJs of a component that have been exercised by a test suite. 100% LCSAJ coverage implies 100% decision coverage.

LCSAJ testing: A white box test design technique in which test cases are designed to execute LCSAJs.

EITP

lead assessor: The person who leads an assessment. In some cases, for instance CMMi and TMMi when formal assessments are conducted, the lead assessor must be accredited and formally trained.

ATA

learnability: The capability of the software product to enable the user to learn its application. [ISO 9126] See also usability.

EITP

learning (IDEAL): The phase within the IDEAL model where one learns from experiences and improves one’s ability to adopt new processes and technologies in the future. The learning phase consists of the activities: analyze and validate, and propose future actions. See also IDEAL.

ATM

level test plan: A test plan that typically addresses one test level. See also test plan.

EITP

lifecycle model: A partitioning of the life of a product or project into phases. [CMMI] See also software lifecycle.

link testing: See component integration testing.

load profile: A specification of the activity which a component or system being tested may experience in production. A load profile consists of a designated number of virtual users who process a defined set of transactions in a specified time period and according to a predefined operational profile. See also operational profile.

F

load testing: A type of performance testing conducted to evaluate the behavior of a component or system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to determine what load can be handled by the component or system. See also performance testing, stress testing.

F

load testing tool: A tool to support load testing whereby it can simulate increasing load, e.g., numbers of concurrent users and/or transactions within a specified time-period. See also performance testing tool.

logic-coverage testing: See white box testing. [Myers]

logic-driven testing: See white box testing.

ATA

logical test case: See high level test case.

ATA

low level test case: A test case with concrete (implementation level) values for input data and expected results. Logical operators from high level test cases are replaced by actual values that correspond to the objectives of the logical operators. See also high level test case.

M

man in the middle attack: The interception, mimicking and/or altering and subsequent relaying of communications (e.g., credit card transactions) by a third party such that a user remains unaware of that third party’s presence.

maintainability: The ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment. [ISO 9126]

F

ATT

maintainability testing: The process of testing to determine the maintainability of a software product.

maintenance: Modification of a software product after delivery to correct defects, to improve performance or other attributes, or to adapt the product to a modified environment. [IEEE 1219]

F

maintenance testing: Testing the changes to an operational system or the impact of a changed environment to an operational system.

ATM

management review: A systematic evaluation of software acquisition, supply, development, operation, or maintenance process, performed by or on behalf of management that monitors progress, determines the status of plans and schedules, confirms requirements and their system allocation, or evaluates the effectiveness of management approaches to achieve fitness for purpose. [After IEEE 610, IEEE 1028]

EITP

manufacturing-based quality: A view of quality, whereby quality is measured by the degree to which a product or service conforms to its intended design and requirements. Quality arises from the process(es) used. [After Garvin] See also product-based quality, transcendent-based quality, user-based quality, value-based quality.

ATM

master test plan: A test plan that typically addresses multiple test levels. See also test plan.

ATT

maturity: (1) The capability of an organization with respect to the effectiveness and efficiency of its processes and work practices. See also Capability Maturity Model Integration, Test Maturity Model integration.
(2) The capability of the software product to avoid failure as a result of defects in the software. [ISO 9126] See also reliability.

EITP

maturity level: Degree of process improvement across a predefined set of process areas in which all goals in the set are attained. [TMMi]

maturity model: A structured collection of elements that describe certain aspects of maturity in an organization, and aid in the definition and understanding of an organization’s processes. A maturity model often provides a common language, shared vision and framework for prioritizing improvement actions.

Mean Time Between Failures: The arithmetic mean (average) time between failures of a system. The MTBF is typically part of a reliability growth model that assumes the failed system is immediately repaired, as a part of a defect fixing process. See also reliability growth model.

Mean Time To Repair: The arithmetic mean (average) time a system will take to recover from any failure. This typically includes testing to insure that the defect has been resolved.

EITP

measure: The number or category assigned to an attribute of an entity by making a measurement. [ISO 14598]

measurement: The process of assigning a number or category to an entity to describe an attribute of that entity. [ISO 14598]

measurement scale: A scale that constrains the type of data analysis that can be performed on it. [ISO 14598]

ATT

memory leak: A memory access failure due to a defect in a program’s dynamic store allocation logic that causes it to fail to release memory after it has finished using it, eventually causing the program and/or other concurrent processes to fail due to lack of memory.

ETM

methodical testing: Testing based on a standard set of tests, e.g., a checklist, a quality standard, or a set of generalized test cases.

F

EITP

metric: A measurement scale and the method used for measurement. [ISO 14598]

migration testing: See conversion testing.

milestone: A point in time in a project at which defined (intermediate) deliverables and results should be ready.

EITP

mind map: A diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central keyword or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.

F

mistake: See error.

ETM

model-based testing: Testing based on a model of the component or system under test, e.g., reliability growth models, usage models such as operational profiles or behavioural models such as decision table or state transition diagram.

F

modeling tool: A tool that supports the creation, amendment and verification of models of the software or system [Graham].

F

ATM

moderator: The leader and main person responsible for an inspection or other review process.

modified condition decision coverage: The percentage of all single condition outcomes that independently affect a decision outcome that have been exercised by a test case suite. 100% modified condition decision coverage implies 100% decision condition coverage.

modified condition decision testing: A white box test design technique in which test cases are designed to execute single condition outcomes that independently affect a decision outcome.

modified multiple condition coverage: See modified condition decision coverage.

modified multiple condition testing: See modified condition decision testing.

module: See component.

module testing: See component testing.

monitor: A software tool or hardware device that runs concurrently with the component or system under test and supervises, records and/or analyses the behavior of the component or system. [After IEEE 610]

F

monitoring tool: See monitor.

monkey testing: Testing by means of a random selection from a large range of inputs and by randomly pushing buttons, ignorant of how the product is being used.

MTBF: See Mean Time Between Failures.

MTTR: See Mean Time To Repair.

multiple condition: See compound condition.

multiple condition coverage: The percentage of combinations of all single condition outcomes within one statement that have been exercised by a test suite. 100% multiple condition coverage implies 100% modified condition decision coverage.

ATT

multiple condition testing: A white box test design technique in which test cases are designed to execute combinations of single condition outcomes (within one statement).

mutation analysis: A method to determine test suite thoroughness by measuring the extent to which a test suite can discriminate the program from slight variants (mutants) of the program.

mutation testing: See back-to-back testing.

ETM

Myers-Briggs Type Indicator (MBTI): An indicator of psychological preference representing the different personalities and communication styles of people.

N

N-switch coverage: The percentage of sequences of N+1 transitions that have been exercised by a test suite. [Chow]

N-switch testing: A form of state transition testing in which test cases are designed to execute all valid sequences of N+1 transitions. [Chow] See also state transition testing.

negative testing: Tests aimed at showing that a component or system does not work. Negative testing is related to the testers’ attitude rather than a specific test approach or test design technique, e.g. testing with invalid input values or exceptions. [After Beizer].

ATT

neighborhood integration testing: A form of integration testing where all of the nodes that connect to a given node are the basis for the integration testing.

non-conformity: Non fulfillment of a specified requirement. [ISO 9000]

F

non-functional requirement: A requirement that does not relate to functionality, but to attributes such as reliability, efficiency, usability, maintainability and portability.

non-functional test design technique: Procedure to derive and/or select test cases for non-functional testing based on an analysis of the specification of a component or system without reference to its internal structure. See also black box test design technique.

non-functional testing: Testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability.

O

off-the-shelf software: A software product that is developed for the general market, i.e. for a large number of customers, and that is delivered to many customers in identical format.

ATM

open source tool: A software tool that is available to all potential users in source code form, usually via the internet; its users are permitted, usually under licence, to study, change, improve and, at times, to distribute the software.

ATA

operability: The capability of the software product to enable the user to operate and control it. [ISO 9126] See also usability.

ATT

operational acceptance testing: Operational testing in the acceptance test phase, typically performed in a (simulated) operational environment by operations and/or systems administration staff focusing on operational aspects, e.g. recoverability, resource-behavior, installability and technical compliance. See also operational testing.

operational environment: Hardware and software products installed at users’ or customers’ sites where the component or system under test will be used. The software may include operating systems, database management systems, and other applications.

ETM

ATT

operational profile: The representation of a distinct set of tasks performed by the component or system, possibly based on user behavior when interacting with the component or system, and their probabilities of occurrence. A task is logical rather that physical and can be executed over several machines or be executed in non-contiguous time segments.

operational profile testing: Statistical testing using a model of system operations (short duration tasks) and their probability of typical use. [Musa]

ETM

operational profiling: The process of developing and implementing an operational profile. See also operational profile.

operational testing: Testing conducted to evaluate a component or system in its operational environment. [IEEE 610]

oracle: See test oracle.

ATA

orthogonal array: A 2-dimensional array constructed with special mathematical properties, such that choosing any two columns in the array provides every pair combination of each number in the array.

orthogonal array testing: A systematic way of testing all-pair combinations of variables using orthogonal arrays. It significantly reduces the number of all combinations of variables to test all pair combinations. See also pairwise testing.

outcome: See result.

output: A variable (whether stored within a component or outside) that is written by a component.

output domain: The set from which valid output values can be selected. See also domain.

output value: An instance of an output. See also output.

outsourced testing: Testing performed by people who are not co-located with the project team and are not fellow employees.

P

pair programming: A software development approach whereby lines of code (production and/or test) of a component are written by two programmers sitting at a single computer. This implicitly means ongoing real-time code reviews are performed.

pair testing: Two persons, e.g. two testers, a developer and a tester, or an end-user and a tester, working together to find defects. Typically, they share one computer and trade control of it while testing.

ATT

pairwise integration testing: A form of integration testing that targets pairs of components that work together, as shown in a call graph.

ATA

pairwise testing: A black box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters. See also orthogonal array testing.

EITP

Pareto analysis: A statistical technique in decision making that is used for selection of a limited number of factors that produce significant overall effect. In terms of quality improvement, a large majority of problems (80%) are produced by a few key causes (20%).

partition testing: See equivalence partitioning. [Beizer]

pass: A test is deemed to pass if its actual result matches its expected result.

pass/fail criteria: Decision rules used to determine whether a test item (function) or feature has passed or failed a test. [IEEE 829]

path: A sequence of events, e.g. executable statements, of a component or system from an entry point to an exit point.

path coverage: The percentage of paths that have been exercised by a test suite. 100% path coverage implies 100% LCSAJ coverage.

path sensitizing: Choosing a set of input values to force the execution of a given path.

ATT

path testing: A white box test design technique in which test cases are designed to execute paths.

F

peer review: A review of a software work product by colleagues of the producer of the product for the purpose of identifying defects and improvements. Examples are inspection, technical review and walkthrough.

performance: The degree to which a system or component accomplishes its designated functions within given constraints regarding processing time and throughput rate. [After IEEE 610] See also efficiency.

performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive development, e.g. lead-time slip for software development. [CMMI]

performance profiling: The task of analyzing, e.g., identifying performance bottlenecks based on generated metrics, and tuning the performance of a software component or system using tools.

F

ATT

performance testing: The process of testing to determine the performance of a software product. See also efficiency testing.

F

ATT

performance testing tool: A tool to support performance testing that usually has two main facilities: load generation and test transaction measurement. Load generation can simulate either multiple users or high volumes of input data. During execution, response time measurements are taken from selected transactions and these are logged. Performance testing tools normally provide reports based on test logs and graphs of load against response times.

ATA

ATM

phase containment: The percentage of defects that are removed in the same phase of the software lifecycle in which they were introduced.

phase test plan: A test plan that typically addresses one test phase. See also test plan.

ETM

planning poker: A consensus-based estimation technique, mostly used to estimate effort or relative size of user stories in agile software development. It is a variation of the Wide Band Delphi method using a deck of cards with values representing the units in which the team estimates. See also agile software development, Wide Band Delphi.

pointer: A data item that specifies the location of another data item; for example, a data item that specifies the address of the next employee record to be processed. [IEEE 610]

portability: The ease with which the software product can be transferred from one hardware or software environment to another. [ISO 9126]

F

ATT

portability testing: The process of testing to determine the portability of a software product.

postcondition: Environmental and state conditions that must be fulfilled after the execution of a test or test procedure.

post-execution comparison: Comparison of actual and expected results, performed after the software has finished running.

post-project meeting: See retrospective meeting.

precondition: Environmental and state conditions that must be fulfilled before the component or system can be executed with a particular test or test procedure.

predicate: A statement that can evaluate to true or false and may be used to determine the control flow of subsequent decision logic. See also decision.

predicted outcome: See expected result.

pretest: See intake test.

ATM

priority: The level of (business) importance assigned to an item, e.g. defect.

F

probe effect: The effect on the component or system by the measurement instrument when the component or system is being measured, e.g. by a performance testing tool or monitor. For example performance may be slightly worse when performance testing tools are being used.

problem: See defect.

problem management: See defect management.

problem report: See defect report.

procedure testing: Testing aimed at ensuring that the component or system can operate in conjunction with new or existing users’ business procedures or operational procedures.

process: A set of interrelated activities, which transform inputs into outputs. [ISO 12207]

EITP

process assessment: A disciplined evaluation of an organization’s software processes against a reference model. [after ISO 15504]

ETM

process-compliant testing: Testing that follows a set of defined processes, e.g., defined by an external party such as a standards committee. See also standard-compliant testing.

process cycle test: A black box test design technique in which test cases are designed to execute business procedures and processes. [TMap] See also procedure testing.

process improvement: A program of activities designed to improve the performance and maturity of the organization’s processes, and the result of such a program. [CMMI]

EITP

process model: A framework wherein processes of the same nature are classified into a overall model, e.g. a test improvement model.

EITP

product-based quality: A view of quality, wherein quality is based on a well-defined set of quality attributes. These attributes must be measured in an objective and quantitative way. Differences in the quality of products of the same type can be traced back to the way the specific quality attributes have been implemented. [After Garvin] See also manufacturing-based quality, quality attribute, transcendent-based quality, user-based quality, value-based quality.

F

ATM

ATA

ATT

product risk: A risk directly related to the test object. See also risk.

production acceptance testing: See operational acceptance testing.

program instrumenter: See instrumenter.

program testing: See component testing.

project: A project is a unique set of coordinated and controlled activities with start and finish dates undertaken to achieve an objective conforming to specific requirements, including the constraints of time, cost and resources. [ISO 9000]

EITP

project retrospective: A structured way to capture lessons learned and to create specific action plans for improving on the next project or next project phase.

F

ATM

project risk: A risk related to management and control of the (test) project, e.g. lack of staffing, strict deadlines, changing requirements, etc. See also risk.

project test plan: See master test plan.

pseudo-random: A series which appears to be random but is in fact generated according to some prearranged sequence.

Q

qualification: The process of demonstrating the ability to fulfill specified requirements. Note the term ‘qualified’ is used to designate the corresponding status. [ISO 9000]

F

quality: The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations. [After IEEE 610]

quality assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled. [ISO 9000]

quality attribute: A feature or characteristic that affects an item’s quality. [IEEE 610]

quality characteristic: See quality attribute.

quality control: The operational techniques and activities, part of quality management, that are focused on fulfilling quality requirements. [after ISO 8402]

quality gate: A special milestone in a project. Quality gates are located between those phases of a project strongly depending on the outcome of a previous phase. A quality gate includes a formal check of the documents of the previous phase.

quality management: Coordinated activities to direct and control an organization with regard to quality. Direction and control with regard to quality generally includes the establishment of the quality policy and quality objectives, quality planning, quality control, quality assurance and quality improvement. [ISO 9000]

ATM

quality risk: A risk related to a quality attribute. See also quality attribute, product risk.

R

ETM

RACI matrix: A matrix describing the participation by various roles in completing tasks or deliverables for a project or process. It is especially useful in clarifying roles and responsibilities. RACI is an acronym derived from the four key responsibilities most typically used: Responsible, Accountable, Consulted, and Informed.

random testing: A black box test design technique where test cases are selected, possibly using a pseudo-random generation algorithm, to match an operational profile. This technique can be used for testing non-functional attributes such as reliability and performance.

EITP

Rational Unified Process: A proprietary adaptable iterative software development process framework consisting of four project lifecycle phases: inception, elaboration, construction and transition.

ETM

reactive testing: Testing that dynamically responds to the actual system under test and test results being obtained. Typically reactive testing has a reduced planning cycle and the design and implementation test phases are not carried out until the test object is received.

recorder: See scribe.

ATT

record/playback tool: See capture/playback tool.

recoverability: The capability of the software product to re-establish a specified level of performance and recover the data directly affected in case of failure. [ISO 9126] See also reliability.

ATT

recoverability testing: The process of testing to determine the recoverability of a software product. See also reliability testing.

recovery testing: See recoverability testing.

ETM

regression-averse testing: Testing using various techniques to manage the risk of regression, e.g., by designing re-usable testware and by extensive automation of testing at one or more test levels.

F

regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed.

regulation testing: See compliance testing.

release note: A document identifying test items, their configuration, current status and other delivery information delivered by development to testing, and possibly other stakeholders, at the start of a test execution phase. [After IEEE 829]

reliability: The ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations. [ISO 9126]

ATT

reliability growth model: A model that shows the growth in reliability over time during continuous testing of a component or system as a result of the removal of defects that result in reliability failures.

F

ATT

reliability testing: The process of testing to determine the reliability of a software product.

ATT

replaceability: The capability of the software product to be used in place of another specified software product for the same purpose in the same environment. [ISO 9126] See also portability.

F

requirement: A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. [After IEEE 610]

ATA

requirements-based testing: An approach to testing in which test cases are designed based on test objectives and test conditions derived from requirements, e.g. tests that exercise specific functions or probe non-functional attributes such as reliability or usability.

F

requirements management tool: A tool that supports the recording of requirements, requirements attributes (e.g. priority, knowledge responsible) and annotation, and facilitates traceability through layers of requirements and requirements change management. Some requirements management tools also provide facilities for static analysis, such as consistency checking and violations to pre-defined requirements rules.

requirements phase: The period of time in the software lifecycle during which the requirements for a software product are defined and documented. [IEEE 610]

resource utilization: The capability of the software product to use appropriate amounts and types of resources, for example the amounts of main and secondary memory used by the program and the sizes of required temporary or overflow files, when the software performs its function under stated conditions. [After ISO 9126] See also efficiency.

ATT

resource utilization testing: The process of testing to determine the resource-utilization of a software product. See also efficiency testing.

result: The consequence/outcome of the execution of a test. It includes outputs to screens, changes to data, reports, and communication messages sent out. See also actual result, expected result.

resumption criteria: The criteria used to restart all or a portion of the testing activities that were suspended previously.

resumption requirements: The defined set of testing activities that must be repeated when testing is re-started after a suspension. [After IEEE 829]

F

re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions.

EITP

retrospective meeting: A meeting at the end of a project during which the project team members evaluate the project and learn lessons that can be applied to the next project.

F

ATM

review: An evaluation of a product or project status to ascertain discrepancies from planned results and to recommend improvements. Examples include management review, informal review, technical review, inspection, and walkthrough. [After IEEE 1028]

ATM

review plan: A document describing the approach, resources and schedule of intended review activities. It identifies, amongst others: documents and code to be reviewed, review types to be used, participants, as well as entry and exit criteria to be applied in case of formal reviews, and the rationale for their choice. It is a record of the review planning process.

F

review tool: A tool that provides support to the review process. Typical features include review planning and tracking support, communication support, collaborative reviews and a repository for collecting and reporting of metrics.

F

ATM

reviewer: The person involved in the review that identifies and describes anomalies in the product or project under review. Reviewers can be chosen to represent different viewpoints and roles in the review process.

F

ATM

risk: A factor that could result in future negative consequences; usually expressed as impact and likelihood.

ATM

ATA

ATT

risk analysis: The process of assessing identified risks to estimate their impact and probability of occurrence (likelihood).

ATM

ATT

risk assessment: The process of assessing a given project or product risk to determine its level of risk, typically by assigning likelihood and impact ratings and then aggregating those ratings into a single risk priority rating. See also product risk, project risk, risk, risk impact, risk level, risk likelihood.

F

ATM

ATA

ATT

risk-based testing: An approach to testing to reduce the level of product risks and inform stakeholders of their status, starting in the initial stages of a project. It involves the identification of product risks and the use of risk levels to guide the test process.

risk category: See risk type.

ATM

ATA

ATT

risk control: The process through which decisions are reached and protective measures are implemented for reducing risks to, or maintaining risks within, specified levels.

ATM

ATA

risk identification: The process of identifying risks using techniques such as brainstorming, checklists and failure history. ATT

risk impact: The damage that will be caused if the risk become an actual outcome or event.

ATM

ATA

ATT

risk level: The importance of a risk as defined by its characteristics impact and likelihood. The level of risk can be used to determine the intensity of testing to be performed. A risk level can be expressed either qualitatively (e.g. high, medium, low) or quantitatively.

risk likelihood: The estimated probability that a risk will become an actual outcome or event.

ATM

ATA

risk management: Systematic application of procedures and practices to the tasks of identifying, analyzing, prioritizing, and controlling risk.

ATM

ATA

ATT

risk mitigation: See risk control.

risk type: A set of risks grouped by one or more common factors such as a quality attribute, cause, location, or potential effect of risk;. A specific set of product risk types is related to the type of testing that can mitigate (control) that risk type. For example the risk of user-interactions being misunderstood can be mitigated by usability testing.

ATT

robustness: The degree to which a component or system can function correctly in the presence of invalid inputs or stressful environmental conditions. [IEEE 610] See also error-tolerance, fault-tolerance.

F

robustness testing: Testing to determine the robustness of the software product.

ATM

root cause: A source of a defect such that if it is removed, the occurrence of the defect type is decreased or removed. [CMMI]

ATA

root cause analysis: An analysis technique aimed at identifying the root causes of defects. By directing corrective measures at root causes, it is hoped that the likelihood of defect recurrence will be minimized.

EITP

RUP: See Rational Unified Process.

S

safety: The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use. [ISO 9126]

safety critical system: A system whose failure or malfunction may result in death or serious injury to people, or loss or severe damage to equipment, or environmental harm.

safety testing: Testing to determine the safety of a software product.

sanity test: See smoke test.

scalability: The capability of the software product to be upgraded to accommodate increased loads. [After Gerrard]

scalability testing: Testing to determine the scalability of the software product.

scenario testing: See use case testing.

scorecard: A representation of summarized performance measurements representing progress towards the implementation of long-term goals. A scorecard provides static measurements of performance over or at the end of a defined interval. See also balanced scorecard, dashboard.

F

scribe: The person who records each defect mentioned and any suggestions for process improvement during a review meeting, on a logging form. The scribe should ensure that the logging form is readable and understandable.

scripted testing: Test execution carried out by following a previously documented sequence of tests.

F

scripting language: A programming language in which executable test scripts are written, used by a test execution tool (e.g. a capture/playback tool).

EITP

SCRUM: An iterative incremental framework for managing projects commonly used with agile software development. See also agile software development.

security: Attributes of software products that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs and data. [ISO 9126] See also functionality.

F

ATT

security testing: Testing to determine the security of the software product. See also functionality testing.

security testing tool: A tool that provides support for testing security characteristics and vulnerabilities.

F

security tool: A tool that supports operational security.

serviceability testing: See maintainability testing.

session-based test management: A method for measuring and managing session-based testing, e.g. exploratory testing.

session-based testing: An approach to testing in which test activities are planned as uninterrupted sessions of test design and execution, often used in conjunction with exploratory testing.

ATM

severity: The degree of impact that a defect has on the development or operation of a component or system. [After IEEE 610]

ETM

Shewhart chart: See control chart.

ATT

short-circuiting: A programming language/interpreter technique for evaluating compound conditions in which a condition on one side of a logical operator may not be evaluated if the condition on the other side is sufficient to determine the final outcome.

simulation: The representation of selected behavioral characteristics of one physical or abstract system by another system. [ISO 2382/1]

simulator: A device, computer program or system used during testing, which behaves or operates like a given system when provided with a set of controlled inputs. [After IEEE 610, DO178b] See also emulator.

site acceptance testing: Acceptance testing by users/customers at their site, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes, normally including hardware as well as software.

ETM

S.M.A.R.T. goal methodology: A methodology whereby objectives are defined very specifically rather than generically. SMART is an acronym derived from the attributes of the objective to be defined: Specific, Measurable, Attainable, Relevant and Timely.

smoke test: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertaining that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. See also intake test.

software: Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system. [IEEE 610]

software attack: See attack.

Software Failure Mode and Effect Analysis (SFMEA): See Failure Mode and Effect Analysis (FMEA).

Software Failure Mode, Effects, and Criticality Analysis (SFMECA): See Failure Mode, Effects, and Criticality Analysis (FMECA).

Software Fault Tree Analysis (SFTA): See Fault Tree Analysis (FTA).

software feature: See feature.

software integrity level: The degree to which software complies or must comply with a set of stakeholder-selected software and/or software-based system characteristics (e.g., software complexity, risk assessment, safety level, security level, desired performance, reliability, or cost) which are defined to reflect the importance of the software to its stakeholders.

EITP

software lifecycle: The period of time that begins when a software product is conceived and ends when the software is no longer available for use. The software lifecycle typically includes a concept phase, requirements phase, design phase, implementation phase, test phase, installation and checkout phase, operation and maintenance phase, and sometimes, retirement phase. Note these phases may overlap or be performed iteratively.

EITP

Software Process Improvement: A program of activities designed to improve the performance and maturity of the organization’s software processes and the results of such a program. [After CMMI]

software product characteristic: See quality attribute.

software quality: The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs. [After ISO 9126] See also quality.

software quality characteristic: See quality attribute.

software test incident: See incident.

software test incident report: See incident report.

ATA

Software Usability Measurement Inventory (SUMI): A questionnaire-based usability test technique for measuring software quality from the end user’s point of view. [Veenendaal04]

source statement: See statement.

specification: A document that specifies, ideally in a complete, precise and verifiable manner, the requirements, design, behavior, or other characteristics of a component or system, and, often, the procedures for determining whether these provisions have been satisfied. [After IEEE 610]

specification-based testing: See black box testing.

ATA

specification-based technique: See black box test design technique.

specification-based test design technique: See black box test design technique.

specified input: An input for which the specification predicts a result.

EITP

SPI: See Software Process Improvement.

ATT

stability: The capability of the software product to avoid unexpected effects from modifications in the software. [ISO 9126] See also maintainability.

EITP

staged representation: A model structure wherein attaining the goals of a set of process areas establishes a maturity level; each level builds a foundation for subsequent levels. [CMMI]

EITP

standard: Formal, possibly mandatory, set of requirements developed and used to prescribe consistent approaches to the way of working or to provide guidelines (e.g., ISO/IEC standards, IEEE standards, and organizational standards). [After CMMI]

ETM

standard-compliant testing: Testing that complies to a set of requirements defined by a standard, e.g., an industry testing standard or a standard for testing safety-critical systems. See also process-compliant testing.

standard software: See off-the-shelf software.

standards testing: See compliance testing.

state diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610]

state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions.

state transition: A transition between two states of a component or system.

F

ATA

state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. See also N-switch testing.

statement: An entity in a programming language, which is typically the smallest indivisible unit of execution.

F

statement coverage: The percentage of executable statements that have been exercised by a test suite.

ATT

statement testing: A white box test design technique in which test cases are designed to execute statements.

F

ATT

static analysis: Analysis of software development artifacts, e.g. requirements or code, carried out without execution of these software development artifacts. Static analysis is usually carried out by means of a supporting tool.

F

static analysis tool: See static analyzer.

ATT

static analyzer: A tool that carries out static analysis.

static code analysis: Analysis of source code carried out without execution of that software.

static code analyzer: A tool that carries out static code analysis. The tool checks source code, for certain properties such as conformance to coding standards, quality metrics or data flow anomalies.

F

static testing: Testing of a software development artifact, e.g., requirements, design or code, without execution of these artifacts, e.g., reviews or static analysis.

statistical testing: A test design technique in which a model of the statistical distribution of the input is used to construct representative test cases. See also operational profile testing.

status accounting: An element of configuration management, consisting of the recording and reporting of information needed to manage a configuration effectively. This information includes a listing of the approved configuration identification, the status of proposed changes to the configuration, and the implementation status of the approved changes. [IEEE 610]

ATM

EITP

STEP: See Systematic Test and Evaluation Process.

storage: See resource utilization.

storage testing: See resource utilization testing.

F

stress testing: A type of performance testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers. [After IEEE 610] See also performance testing, load testing.

F

stress testing tool: A tool that supports stress testing.

structural coverage: Coverage measures based on the internal structure of a component or system.

structural test design technique: See white-box test design technique.

F

structural testing: See white-box testing.

structure-based test design technique: See white-box test design technique.

ATT

structure-based technique: See white-box test design technique.

F

structure-based testing: See white-box testing.

structured walkthrough: See walkthrough.

F

stub: A skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component. [After IEEE 610]

subpath: A sequence of executable statements within a component.

suitability: The capability of the software product to provide an appropriate set of functions for specified tasks and user objectives. [ISO 9126] See also functionality.

ATA

suitability testing: The process of testing to determine the suitability of a software product

ATA

SUMI: See Software Usability Measurement Inventory.

suspension criteria: The criteria used to (temporarily) stop all or a portion of the testing activities on the test items. [After IEEE 829]

syntax testing: A black box test design technique in which test cases are designed based upon the definition of the input domain and/or output domain.

system: A collection of components organized to accomplish a specific function or set of functions. [IEEE 610]

system integration testing: Testing the integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet).

system of systems: Multiple heterogeneous, distributed systems that are embedded in networks at multiple levels and in multiple interconnected domains, addressing large-scale inter-disciplinary common problems and purposes, usually without a common management structure.

F

system testing: The process of testing an integrated system to verify that it meets specified requirements. [Hetzel]

ATM

EITP

Systematic Test and Evaluation Process: A structured testing methodology, also used as a content-based model for improving the testing process. Systematic Test and Evaluation Process (STEP) does not require that improvements occur in a specific order. See also content-based model.

T

F

ATM

technical review: A peer group discussion activity that focuses on achieving consensus on the technical approach to be taken. [Gilb and Graham], [IEEE 1028] See also peer review.

test: A set of one or more test cases. [IEEE 829]

F

ATM

test approach: The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) project’s goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed

ETM

test architect : (1) A person who provides guidance and strategic direction for a test organization and for its relationship with other disciplines.
(2) A person who defines the way testing is structured for a given system, including topics such as test tools and test data management.

test automation: The use of software to perform or support test activities, e.g. test management, test design, test execution and results checking.

F

test basis: All documents from which the requirements of a component or system can be inferred. The documentation on which the test cases are based. If a document can be amended only by way of formal amendment procedure, then the test basis is called a frozen test basis. [After TMap]

test bed: See test environment.

F

ATM

test case: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

test case design technique: See test design technique.

F

test case specification: A document specifying a set of test cases (objective, inputs, test actions, expected results, and execution preconditions) for a test item. [After IEEE 829] See also test specification.

test case suite: See test suite.

ATA

test charter: A statement of test objectives, and possibly test ideas about how to test. Test charters are used in exploratory testing. See also exploratory testing.

ATM

test closure: During the test closure phase of a test process data is collected from completed activities to consolidate experience, testware, facts and numbers. The test closure phase consists of finalizing and archiving the testware and evaluating the test process, including preparation of a test evaluation report. See also test process.

F

test comparator: A test tool to perform automated test comparison of actual results with expected results.

test comparison: The process of identifying differences between the actual results produced by the component or system under test and the expected results for a test. Test comparison can be performed during test execution (dynamic comparison) or after test execution.

test completion criteria: See exit criteria.

F

ATM

test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.

F

ATM

ATA

test control: A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned. See also test management.

F

test coverage: See coverage.

test cycle: Execution of the test process against a single identifiable release of the test object.

F

test data: Data that exists (for example, in a database) before a test is executed, and that affects or is affected by the component or system under test.

F

ATA

test data preparation tool: A type of test tool that enables data to be selected from existing databases or created, generated, manipulated and edited for use in testing.

test data management: The process of analyzing test data requirements, designing test data structures, creating and maintaining test data.

test deliverable: Any test (work) product that must be delivered to someone other than the test (work) product’s author. See also deliverable.

F

ATM

ATA

test design: (1) See test design specification.
(2) The process of transforming general testing objectives into tangible test conditions and test cases.

test design specification: A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases. [After IEEE 829] See also test specification.

F

test design technique: Procedure used to derive and/or select test cases.

F

ATA

test design tool: A tool that supports the test design activity by generating test inputs from a specification that may be held in a CASE tool repository, e.g. requirements management tool, from specified test conditions held in the tool itself, or from code.

ATM

test director: A senior manager who manages test managers. See also test manager.

F

ETM

test driven development: A way of developing software where the test cases are developed, and often automated, before the software is developed to run those test cases.

test driver: See driver.

F

test environment: An environment containing hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test. [After IEEE 610]

ATM

test estimation: The calculated approximation of a result related to various aspects of testing (e.g. effort spent, completion date, costs involved, number of test cases, etc.) which is usable even if input data may be incomplete, uncertain, or noisy.

test evaluation report: A document produced at the end of the test process summarizing all testing activities and results. It also contains an evaluation of the test process and lessons learned.

F

ATM

ATA

test execution: The process of running a test on the component or system under test, producing actual result(s).

test execution automation: The use of software, e.g. capture/playback tools, to control the execution of tests, the comparison of actual results to expected results, the setting up of test preconditions, and other test control and reporting functions.

test execution phase: The period of time in a software development lifecycle during which the components of a software product are executed, and the software product is evaluated to determine whether or not requirements have been satisfied. [IEEE 610]

F

test execution schedule: A scheme for the execution of test procedures. Note: The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed.

test execution technique: The method used to perform the actual test execution, either manual or automated.

F

ATA

ATT

test execution tool: A type of test tool that is able to execute other software using an automated test script, e.g. capture/playback. [Fewster and Graham]

test fail: See fail.

test generator: See test data preparation tool.

F

test harness: A test environment comprised of stubs and drivers needed to execute a test.

ATM

ATA

test implementation: The process of developing and prioritizing test procedures, creating test data and, optionally, preparing test harnesses and writing automated test scripts.

EITP

test improvement plan: A plan for achieving organizational test process improvement objectives based on a thorough understanding of the current strengths and weaknesses of the organization’s test processes and test process assets. [After CMMI]

test incident: See incident.

test incident report: See incident report.

test infrastructure: The organizational artifacts needed to perform testing, consisting of test environments, test tools, office environment and procedures.

test input: The data received from an external source by the test object during test execution. The external source can be hardware, software or human.

test item: The individual element to be tested. There usually is one test object and many test items. See also test object.

test item transmittal report: See release note.

F

test leader: See test manager.

F

ATM

test level: A group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels are component test, integration test, system test and acceptance test. [After TMap]

F

ATM

test log: A chronological record of relevant details about the execution of tests. [IEEE 829]

test logging: The process of recording information about tests executed into a test log.

ATM

test management: The planning, estimating, monitoring and control of test activities, typically carried out by a test manager.

F

ATT

test management tool: A tool that provides support to the test management and control part of a test process. It often has several capabilities, such as testware management, scheduling of tests, the logging of results, progress tracking, incident management and test reporting.

F

test manager: The person responsible for project management of testing activities and resources, and evaluation of a test object. The individual who directs, controls, administers, plans and regulates the evaluation of a test object.

ATM

EITP

Test Maturity Model integration: A five level staged framework for test process improvement, related to the Capability Maturity Model Integration (CMMI), that describes the key elements of an effective test process.

ETM

test mission: The purpose of testing for an organization, often documented as part of the test policy. See also test policy.

F

ATM

ATA

test monitoring: A test management task that deals with the activities related to periodically checking the status of a test project. Reports are prepared that compare the actuals to that which was planned. See also test management.

test object: The component or system to be tested. See also test item.

F

test objective: A reason or purpose for designing and executing a test.

test oracle: A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code. [After Adrion]

test outcome: See result.

test pass: See pass.

test performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive test development, e.g. Defect Detection Percentage (DDP).

test phase: A distinct set of test activities collected into a manageable phase of a project, e.g. the execution activities of a test level. [After Gerrard]

F

ATM

test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process. [After IEEE 829]

ATM

test planning: The activity of establishing or updating a test plan.

ATA

Test Point Analysis (TPA): A formula based test estimation method based on function point analysis. [TMap]

F

ATM

EITP

ETM

test policy: A high level document describing the principles, approach and major objectives of the organization regarding testing.

F

ATM

test procedure: See test procedure specification.

F

ATM

test procedure specification: A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829] See also test specification.

test process: The fundamental test process comprises test planning and control, test analysis and design, test implementation and execution, evaluating exit criteria and reporting, and test closure activities.

EITP

Test Process Group: A collection of (test) specialists who facilitate the definition, maintenance, and improvement of the test processes used by an organization. [After CMMI]

EITP

test process improvement manifesto: A statement that echoes the agile manifesto, and defines values for improving the testing process. The values are:

- flexibility over detailed processes

- best practices over templates

- deployment orientation over process orientation

- peer reviews over quality assurance (departments)

- business driven over model driven. [Veenendaal08]

EITP

test process improver: A person implementing improvements in the test process based on a test improvement plan.

test progress report: A document summarizing testing activities and results, produced at regular intervals, to report progress of testing activities against a baseline (such as the original test plan) and to communicate risks and alternatives requiring a decision to management.

test record: See test log.

test recording: See test logging.

test report: See test summary report and test progress report.

test reproducibility: An attribute of a test indicating whether the same results are produced each time the test is executed.

test requirement: See test condition.

test result: See result.

test rig: See test environment.

test run: Execution of a test on a specific version of the test object.

test run log: See test log.

test scenario: See test procedure specification.

test schedule: A list of activities, tasks or events of the test process, identifying their intended start and finish dates and/or times, and interdependencies.

F

ATM

test script: Commonly used to refer to a test procedure specification, especially an automated one.

test session: An uninterrupted period of time spent in executing tests. In exploratory testing, each test session is focused on a charter, but testers can also explore new opportunities or issues during a session. The tester creates and executes test cases on the fly and records their progress. See also exploratory testing.

test set: See test suite.

test situation: See test condition.

test specification: A document that consists of a test design specification, test case specification and/or test procedure specification.

test specification technique: See test design technique.

test stage: See test level.

F

ATM

ATA

ETM

test strategy: A high-level description of the test levels to be performed and the testing within those levels for an organization or programme (one or more projects).

F

test suite: A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one.

F

ATM

test summary report: A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items against exit criteria. [After IEEE 829]

test target: A set of exit criteria.

test technique: See test design technique.

EITP

test tool: A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis. [TMap] See also CAST.

test type: A group of test activities aimed at testing a component or system focused on a specific test objective, i.e. functional test, usability test, regression test etc. A test type may take place on one or more test levels or test phases. [After TMap]

ATT

testability: The capability of the software product to enable modified software to be tested. [ISO 9126] See also maintainability.

testability review: A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. [After TMap]

testable requirement: A requirements that is stated in terms that permit establishment of test designs (and subsequently test cases) and execution of tests to determine whether the requirement has been met. [After IEEE 610]

F

tester: A skilled professional who is involved in the testing of a component or system.

F

testing: The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.

F

testware: Artifacts produced during the test process required to plan, design, and execute tests, such as documentation, scripts, inputs, expected results, set-up and clear-up procedures, files, databases, environment, and any additional software or utilities used in testing. [After Fewster and Graham]

thread testing: An approach to component integration testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by levels of a hierarchy.

three point estimation: A test estimation method using estimated values for the “best case”, “worst case”, and “most likely case” of the matter being estimated, to define the degree of certainty associated with the resultant estimate.

time behavior: See performance.

ATM

EITP

TMMi: See Test Maturity Model integration.

top-down testing: An incremental approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested. See also integration testing.

EITP

Total Quality Management: An organization-wide management approach centered on quality, based on the participation of all members of the organization and aiming at long-term success through customer satisfaction, and benefits to all members of the organization and to society. Total Quality Management consists of planning, organizing, directing, control, and assurance. [After ISO 8402]

ATM

EITP

TPI Next: A continuous business-driven framework for test process improvement that describes the key elements of an effective and efficient test process.

EITP

TPG: See Test Process Group.

EITP

TQM: See Total Quality Management.

F

traceability: The ability to identify related items in documentation and software, such as requirements with associated tests. See also horizontal traceability, vertical traceability.

EITP

transactional analysis: The analysis of transactions between people and within people’s minds; a transaction is defined as a stimulus plus a response. Transactions take place between people and between the ego states (personality segments) within one person’s mind.

EITP

transcendent-based quality: A view of quality, wherein quality cannot be precisely defined, but we know it when we see it, or are aware of its absence when it is missing. Quality depends on the perception and affective feelings of an individual or group of individuals towards a product. [After Garvin] See also manufacturing-based quality, product-based quality, user-based quality, value-based quality.

U

ATA

understandability: The capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use. [ISO 9126] See also usability.

unit: See component.

F

unit test framework: A tool that provides an environment for unit or component testing in which a component can be tested in isolation or with suitable stubs and drivers. It also provides other support for the developer, such as debugging capabilities. [Graham]

unit testing: See component testing.

unreachable code: Code that cannot be reached and therefore is impossible to execute.

usability: The capability of the software to be understood, learned, used and attractive to the user when used under specified conditions. [ISO 9126]

F

ATA

usability testing: Testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions. [After ISO 9126]

use case: A sequence of transactions in a dialogue between an actor and a component or system with a tangible result, where an actor can be a user or anything that can exchange information with the system.

F

ATA

use case testing: A black box test design technique in which test cases are designed to execute scenarios of use cases.

F

user acceptance testing: See acceptance testing.

EITP

user-based quality: A view of quality, wherein quality is the capacity to satisfy needs, wants and desires of the user(s). A product or service that does not fulfill user needs is unlikely to find any users. This is a context dependent, contingent approach to quality since different business characteristics require different qualities of a product. [after Garvin] See also manufacturing-based quality, product-based quality, transcendent-based quality, value-based quality.

user scenario testing: See use case testing

user story: A high-level user or business requirement commonly used in agile software development, typically consisting of one or more sentences in the everyday or business language capturing what functionality a user needs, any non-functional criteria, and also includes acceptance criteria. See also agile software development, requirement.

ATA

user story testing: A black box test design technique in which test cases are designed based on user stories to verify their correct implementation. See also user story.

user test: A test whereby real-life users are involved to evaluate the usability of a component or system.

V

F

V-model: A framework to describe the software development lifecycle activities from requirements specification to maintenance. The V-model illustrates how testing activities can be integrated into each phase of the software development lifecycle.

F

validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]

EITP

value-based quality: A view of quality, wherein quality is defined by price. A quality product or service is one that provides desired performance at an acceptable cost. Quality is determined by means of a decision process with stakeholders on trade-offs between time, effort and cost aspects. [After Garvin] See also manufacturing-based quality, product-based quality, transcendent-based quality, user-based quality.

variable: An element of storage in a computer that is accessible by a software program by referring to it by a name.

F

verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. [ISO 9000]

F

version control: See configuration control.

vertical traceability: The tracing of requirements through the layers of development documentation to components.

volume testing: Testing where the system is subjected to large volumes of data. See also resource-utilization testing.

W

F

ATM

walkthrough: A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content. [Freedman and Weinberg, IEEE 1028] See also peer review.

ATA

WAMMI: See Website Analysis and MeasureMent Inventory.

WBS: See Work Breakdown Structure.

ATA

Website Analysis and MeasureMent Inventory (WAMMI): A questionnaire-based usability test technique for measuring web site software quality from the end user’s point of view.

white-box technique: See white-box test design technique.

F

ATT

white-box test design technique: Procedure to derive and/or select test cases based on an analysis of the internal structure of a component or system.

F

white-box testing: Testing based on an analysis of the internal structure of the component or system.

ATM

Wide Band Delphi: An expert based test estimation technique that aims at making an accurate estimation using the collective wisdom of the team members.

ATT

wild pointer: A pointer that references a location that is out of scope for that pointer or that does not exist. See also pointer.

Work Breakdown Structure: An arrangement of work elements and their relationship to each other and to the end product. [CMMI]

Annex A References

Standards

[DO-178b] DO-178B:1992. Software Considerations in Airborne Systems and Equipment Certification, Requirements and Technical Concepts for Aviation (RTCA SC167).

[IEEE 610] IEEE 610.12:1990. Standard Glossary of Software Engineering Terminology.

[IEEE 829] IEEE 829:1998. Standard for Software Test Documentation.

[IEEE 1008] IEEE 1008:1993. Standard for Software Unit Testing.

[IEEE 1028] IEEE 1028:1997. Standard for Software Reviews and Audits.

[IEEE 1044] IEEE 1044:1993. Standard Classification for Software Anomalies.

[IEEE 1219] IEEE 1219:1998. Software Maintenance.

[ISO 2382/1] ISO/IEC 2382-1:1993. Data processing - Vocabulary - Part 1: Fundamental terms.

[ISO 9000] ISO 9000:2005. Quality Management Systems – Fundamentals and Vocabulary.

[ISO 9126] ISO/IEC 9126-1:2001. Software Engineering – Software Product Quality – Part 1: Quality characteristics and sub-characteristics.

[ISO 12207] ISO/IEC 12207:1995. Information Technology – Software Lifecycle Processes.

[ISO 14598] ISO/IEC 14598-1:1999. Information Technology – Software Product Evaluation - Part 1: General Overview.

[ISO 15504] ISO/IEC 15504-9: 1998. Information Technology – Software Process Assessment – Part 9: Vocabulary

Books and papers

[Abbott] J. Abbot (1986), Software Testing Techniques, NCC Publications.

[Adrion] W. Adrion, M. Branstad and J. Cherniabsky (1982), Validation, Verification and Testing of Computer Software, in: Computing Surveys, Vol. 14, No 2, June 1982.

[Bach] J. Bach (2004), Exploratory Testing, in: E. van Veenendaal, The Testing Practitioner – 2nd edition, UTN Publishing, ISBN 90-72194-65-9.

[Beizer] B. Beizer (1990), Software Testing Techniques, van Nostrand Reinhold, ISBN 0-442-20672-0

[Chow] T. Chow (1978), Testing Software Design Modelled by Finite-Sate Machines, in: IEEE Transactions on Software Engineering, Vol. 4, No 3, May 1978.

[CMM] M. Paulk, C. Weber, B. Curtis and M.B. Chrissis (1995), The Capability Maturity Model, Guidelines for Improving the Software Process, Addison-Wesley, ISBN 0-201-54664-7

[CMMI] M.B. Chrissis, M. Konrad and S. Shrum (2004), CMMI, Guidelines for Process Integration and Product Improvement, Addison Wesley, ISBN 0-321-15496-7

[Deming] D. W. Edwards (1986), Out of the Crisis, MIT Center for Advanced Engineering Study, ISBN 0-911379-01-0

[Fenton] N. Fenton (1991), Software Metrics: a Rigorous Approach, Chapman & Hall, ISBN 0-53249-425-1

[Fewster and Graham] M. Fewster and D. Graham (1999), Software Test Automation, Effective use of test execution tools, Addison-Wesley, ISBN 0-201-33140-3.

[Freedman and Weinberg] D. Freedman and G. Weinberg (1990), Walkthroughs, Inspections, and Technical Reviews, Dorset House Publishing, ISBN 0-932633-19-6.

[Garvin] D.A. Garvin (1984), What does product quality really mean?, in: Sloan Management Review, Vol. 26, nr. 1 1984

[Gerrard] P. Gerrard and N. Thompson (2002), Risk-Based E-Business Testing, Artech House Publishers, ISBN 1-58053-314-0.

[Gilb and Graham] T. Gilb and D. Graham (1993), Software Inspection, Addison-Wesley, ISBN 0-201-63181-4.

[Graham] D. Graham, E. van Veenendaal, I. Evans and R. Black (2007), Foundations of Software Testing, Thomson Learning, ISBN 978-1-84480-355-2

[Grochtmann] M. Grochtmann (1994), Test Case Design Using Classification Trees, in: Conference Proceedings STAR 1994.

[Hetzel] W. Hetzel (1988), The complete guide to software testing – 2nd edition, QED Information Sciences, ISBN 0-89435-242-3.

[Juran] J.M. Juran (1979), Quality Control Handbook, McGraw-Hill

[McCabe] T. McCabe (1976), A complexity measure, in: IEEE Transactions on Software Engineering, Vol. 2, pp. 308-320.

[Musa] J. Musa (1998), Software Reliability Engineering Testing, McGraw-Hill Education, ISBN 0-07913-271-5.

[Myers] G. Myers (1979), The Art of Software Testing, Wiley, ISBN 0-471-04328-1.

[TMap] M. Pol, R. Teunissen, E. van Veenendaal (2002), Software Testing, A guide to the TMap Approach, Addison Wesley, ISBN 0-201-745712.

[TMMi] E. van Veenendaal and J. Cannegieter (2011), The Little TMMi, UTN Publishing, ISBN 97-89490986-03-2

[Veenendaal04] E. van Veenendaal (2004), The Testing Practitioner – 2nd edition, UTN Publishing, ISBN 90-72194-65-9.

[Veenendaal08] E. van Veenendaal (2008), Test Improvement Manifesto, in: Testing Experience, Issue 04/08, December 2008