AI Testing
AI testing covers two things: using AI to generate and maintain tests, and testing AI systems whose output is non-deterministic.
/ quick answer
Testing with AI means generating unit, integration and edge-case tests from code and specs — cheap coverage, but it must be reviewed or it just encodes current behaviour as correct. Testing AI means grading probabilistic output: assertion-based checks for structure, rubric or model-graded checks for quality, and regression suites of real failures. Snapshot tests break immediately on model…
/ frequently asked
What is AI Testing?
Testing with AI means generating unit, integration and edge-case tests from code and specs — cheap coverage, but it must be reviewed or it just encodes current behaviour as correct. Testing AI means grading probabilistic output: assertion-based checks for structure, rubric or model-graded checks for quality, and regression suites of real failures. Snapshot tests break immediately on model output; behavioural assertions survive.
What is an example of AI Testing?
An extraction feature is tested with 40 stored documents asserting schema validity and field-level accuracy above 95%, not exact string equality.
Why does AI Testing matter?
AI testing covers two things: using AI to generate and maintain tests, and testing AI systems whose output is non-deterministic.
What should I learn after AI Testing?
Open the related workflows and prompts further down this page: they show how this idea is applied in a real task, step by step.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →AI Evals
Reproducible test suites that measure LLM output quality across model, prompt and code changes.
- →Evals
Automated tests that grade LLM outputs against expected behavior.
- →LLM-as-Judge
Using a strong model to grade another model's output.
- →Coding Agent
A coding agent reads a repository, plans a change, edits files, runs tests and iterates until the task passes — instead of just suggesting snippets.
Related workflows
Turn this into a repeatable process.
- →Build a Test Suite for a Non-Deterministic AI Feature
Grade probabilistic output without brittle snapshot tests.
- →Build an AI Code Review Loop
Catch what agents get wrong before a human reads the PR.
- →Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
- →Autonomous QA Testing
This workflow utilizes AutoGen to create a multi-agent system that autonomously generates test cases, executes them, identifies bugs, and suggests fixes for software applications, enhancing development efficiency.
Related tool stacks
The tools that run it in production.
- →AI Testing Stack
Test deterministic code and probabilistic AI output in one pipeline.
- →Coding Agent Stack
Run coding agents with executable feedback and reviewable diffs.
Related prompts
Reusable prompts for this job.
- →AI Test Generation Prompt
Generates behavioural tests and edge cases, not tautologies.
- →AI Code Review Rubric Prompt
A reviewer prompt that finds real defects instead of style opinions.
- →Coding Agent Task Spec Prompt
Writes the task brief that keeps a coding agent inside scope.
Related use cases
How people apply it, and what came out.
- →Two-Founder Startup Ships Like a Team of Six
Coding agents plus a hard verification gate replaced three planned hires.
- →Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.
Comparisons & alternatives
Pick between the options.
- →Coding Agent vs Autocomplete Copilot
Copilots accelerate typing; coding agents complete tasks. Different unit of work, different review burden.