AI Testing Stack
Test deterministic code and probabilistic AI output in one pipeline.
/ quick answer
Keep quality measurable when part of the system is non-deterministic. Test deterministic code and probabilistic AI output in one pipeline.
- Vitest or Pytest (unit and integration)
- Playwright (end-to-end)
- Langfuse or Braintrust (eval runs and scoring)
- GitHub Actions (gates on every commit)
- A stored regression set of real production failures
- Promptfoo
- DeepEval
- Custom eval harness
/ frequently asked
What is the AI Testing Stack stack for?
Keep quality measurable when part of the system is non-deterministic.
Which tools are in this stack?
Vitest or Pytest (unit and integration), Playwright (end-to-end), Langfuse or Braintrust (eval runs and scoring), GitHub Actions (gates on every commit), A stored regression set of real production failures.
Are there alternatives to this stack?
Yes — Promptfoo, DeepEval, Custom eval harness.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →AI Software Engineering
AI software engineering is the practice of building software where agents write most of the code and humans own architecture, review and verification.
- →AI Testing
AI testing covers two things: using AI to generate and maintain tests, and testing AI systems whose output is non-deterministic.
- →AI Evaluation
AI evaluation is the measurement layer of an AI system: a fixed set of cases, a scoring method and a tracked pass rate you can regress against.
Related workflows
Turn this into a repeatable process.
- →Build a Test Suite for a Non-Deterministic AI Feature
Grade probabilistic output without brittle snapshot tests.
- →Build an AI Code Review Loop
Catch what agents get wrong before a human reads the PR.
- →Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
Related tool stacks
The tools that run it in production.
- →Coding Agent Stack
Run coding agents with executable feedback and reviewable diffs.
- →Indie SaaS Launch Stack
Everything a solo founder needs to ship and monetize a SaaS in weeks.
- →AI Marketing Ops Stack
The control center for an AI-augmented marketing team of one to five.
- →Agent Architecture Stack
The minimum tooling to design, run and observe a production agent.
Related prompts
Reusable prompts for this job.
- →AI Test Generation Prompt
Generates behavioural tests and edge cases, not tautologies.
Related use cases
How people apply it, and what came out.
- →Two-Founder Startup Ships Like a Team of Six
Coding agents plus a hard verification gate replaced three planned hires.
- →Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.
Comparisons & alternatives
Pick between the options.
- →Model-Graded Evals vs Assertion Evals
Assertions are cheap, fast and objective; model grading captures quality you cannot express as a rule.