AI is transforming software testing at a speed most organizations have never seen. Teams that once spent days drafting test cases are now generating hundreds of them in hours. Backlogs are shrinking. Coverage reports look impressive. Stakeholders are happy.
But here is the uncomfortable question that ASTQB President Andrew Pollner posed at a recent industry conference: Does AI actually improve software quality, or does it just make it look that way?
The answer, backed by real-world data from organizations already using AI testing tools, is more nuanced than the hype suggests. That answer: Yes, AI adds speed. But it does not add inherent quality. Without trained human testers in the loop, the gap between those two things can lead to software that ships with significant, undetected defects.
The Two Paths Every Testing Team Faces
When organizations adopt Generative AI for testing, they typically go one of two directions.
- The first is the shortcut approach. AI is treated primarily as a cost-cutting tool. Staff and expertise are reduced, review cycles are minimized, and the focus shifts almost entirely to velocity. Test cases get written faster, and there are a lot more of them. There are so many test cases that everyone feels confident.
The outcome of this approach is faster delivery but also flawed delivery, with more defects reaching production than before. - The second is the strategic approach. Here, organizations treat AI as a force multiplier rather than a replacement. They invest in skills and training, pair AI capabilities with human testing expertise, enforce rigorous review processes, and align coverage with actual risk. The outcome is testing that is both efficient and effective, with measurably fewer defects.
The difference between these two paths is not the AI tool. It is the people using it, the training they have, and the discipline they bring to the work.
What AI Commonly Gets Wrong
Understanding why human oversight matters requires understanding where AI-generated test artifacts consistently fall short. There are several well-documented failure patterns.
The Happy Path Bias. AI tools tend to over-represent ideal user journeys. Tests follow the expected flow, the successful outcome – the clean scenario. What gets underrepresented or missed entirely are boundary conditions, negative tests, and the edge cases where software most commonly fails in the real world.
False Confidence from Volume. High test case counts create a psychological illusion of thorough coverage. Organizations see 500 tests where they previously had 50 and assume they are now 10 times better protected. In reality, 500 tests that all test the same happy path leave the same critical risks untested. Quantity does not equal quality.
Missing Boundary Conditions. Boundary Value Analysis (BVA) is one of the most reliable techniques for finding defects. Software fails at the edges of acceptable inputs – just inside, at, and just outside boundary conditions – far more often than it fails in the middle of a range. AI frequently generates tests that target arbitrary mid-range values and skip the boundaries entirely, leaving some of the most defect-prone areas of a system untested.
Incomplete State Transition Coverage. Complex software moves through states based on events and conditions. A payment that moves from initiated to authorized to settled, or a user session that moves from active to timed-out to re-authenticated, involves transitions that can fail in non-obvious ways. State Transition Testing ensures the system moves correctly through all of those transitions. AI-generated suites frequently address the most obvious paths and leave edge-case state transitions uncovered.
Skipping Equivalence Partitioning. Testing every possible input is not just inefficient – it is impossible. Equivalence Partitioning (EP) addresses this by identifying classes of inputs that the system should handle identically, then testing a representative sample from each class. This technique is especially important for AI systems, which often have vast input spaces. Without it, AI-generated tests can cover an enormous surface area while still missing whole categories of behavior.
Overlooking Non-Functional Risks. Security vulnerabilities, performance degradation under load, and accessibility failures rarely appear in AI-generated test suites unless a skilled tester explicitly prompts for them. These risks are not visible in user stories or acceptance criteria – they require expertise to identify and design tests for.
Building a Quality Engineering Framework That Actually Works
The antidote to these failure patterns is not abandoning AI. It is building a quality engineering framework that uses AI strategically, with human expertise embedded at every critical point.
That framework has four components. First, skill development — testers who are trained in test design techniques and AI prompting. Second, review standards – defined criteria for evaluating AI-generated artifacts before they enter a test suite. Third, quality metrics, measurement of defect detection effectiveness, not just test case volume. And fourth, governance, clear policies about how AI tools are used, who reviews the output, and how coverage decisions are made.
The throughline across all four components is human expertise. Not to slow AI down, but to direct it toward the things that matter: gaps, risks, traceability, and the kinds of defects that cost the most to find late.
How ISTQB Certifications Build That Expertise
This is where ASTQB-administered ISTQB certifications become directly relevant to every organization navigating the AI testing transition.
ISTQB Testing with Generative AI Certification was designed precisely for this moment. It prepares testers to understand AI and ML concepts, trends, and how testers can influence model quality. It addresses the specific challenges AI introduces, such as bias, ethics, non-determinism, and explainability, and teaches practical prompting techniques for software testing tasks. Critically, it prepares testers to identify risks and contribute to Generative AI testing strategies within their organizations. This is the foundation for the strategic path.
ISTQB AI Testing Certification focuses on the AI itself as the system under test, which is a fundamentally different challenge from testing conventional software. Because AI systems are non-deterministic and their behavior is shaped by data as much as by code, certified professionals learn to design tests tailored for machine learning models, validate training data for bias, noise, and imbalance, apply proper metrics like precision and recall, and monitor AI model drift and performance degradation. For organizations building or deploying AI-based systems, this certification produces testers who understand what it means for an AI to fail and how to design tests that surface it.
ISTQB Advanced Level Test Analyst Certification equips testers with exactly the techniques that AI misses on its own. Risk-based testing, Boundary Value Analysis, State Transition Testing, Equivalence Partitioning — these are not just theoretical frameworks. They are practical skills that trained Test Analysts use to find the defects AI-generated tests leave behind. This certification also covers functional and non-functional testing, tools, test data strategy, and defect prevention — building the kind of complete technical foundation that makes AI a genuine accelerator rather than a liability.
ISTQB Advanced Level Test Management Certification builds the organizational layer on top. Test Managers certified at this level can plan, manage, and monitor testing across projects and development lifecycles. They can define test strategies, apply risk-based approaches, report progress to stakeholders, and drive continuous improvement in test processes. In an AI-augmented testing environment, that management capability is what ensures the strategic approach stays strategic — that governance policies are enforced, that coverage decisions are intentional, and that the team’s use of AI scales without sacrificing quality.
ASTQB AI Assurance Pro™ is a designation for professionals who have earned all three of these certifications: ISTQB Foundation Level, ISTQB Testing with Generative AI, and ISTQB AI Testing. Together, these credentials signal that a tester understands not just how to use AI tools, but how to govern them, including evaluating AI-generated artifacts, identifying coverage gaps, and contributing to an organization’s broader AI quality strategy. Because AI changes so often, the designation is valid for 12 months and requires annual continuing education through AT*SQA AI-related micro-credentials to maintain. It is a credible way for organizations to identify and build the human expertise that makes the strategic path possible.
What You Need to Know and Do
AI has made testing faster. That is not in dispute. The question every testing leader should be asking right now is whether faster is actually translating into better software quality – or whether it is producing an increasingly convincing illusion of it.
Speed is measurable. Quality impact is harder to prove. Without testers who understand how to evaluate the artifacts AI generates, how to fill the gaps it reliably leaves, and how to align coverage with genuine risk, organizations are making a quiet trade: trading expertise for velocity and calling it progress.
ASTQB and AT*SQA offer the full suite of ISTQB certifications, including Testing with Generative AI, Advanced Level Test Analyst, and Advanced Level Test Management, to help testing software quality professionals and the organizations they work for take the strategic path. The goal was never more tests. It was always better software.
See all of the ISTQB software testing certifications and download the free ISTQB exam preparation materials. See why software testers take their ISTQB exams through the non-profit organizations ASTQB & AT*SQA.
