AI Observability Stack
Traces, cost, evals and quality drift for AI systems in production.
/ quick answer
Know what your AI system actually did, what it cost and whether it got worse. Traces, cost, evals and quality drift for AI systems in production.
- Langfuse (traces, evals, datasets)
- OpenTelemetry (spans across services)
- Sentry (errors and regressions)
- Postgres (run log and metrics)
- Grafana or Metabase (dashboards and alerts)
- Braintrust
- LangSmith
- Helicone
What is the AI Observability Stack stack for?
Know what your AI system actually did, what it cost and whether it got worse.
Which tools are in this stack?
Langfuse (traces, evals, datasets), OpenTelemetry (spans across services), Sentry (errors and regressions), Postgres (run log and metrics), Grafana or Metabase (dashboards and alerts).
Are there alternatives to this stack?
Yes — Braintrust, LangSmith, Helicone.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →Agent Cost Control
Agent cost control is the practice of budgeting tokens, steps and model tiers per task so autonomous systems stay economically viable at scale.
- →Context Engineering
Context engineering is the discipline of deciding exactly what information enters a model's context window, in what order and at what cost.
- →AI Evaluation
AI evaluation is the measurement layer of an AI system: a fixed set of cases, a scoring method and a tracked pass rate you can regress against.
- →AI Monitoring
AI monitoring is production observability for model-driven systems: traces, cost, latency, tool failures and output-quality drift.
Related workflows
Turn this into a repeatable process.
- →Monitor an AI System in Production
See quality, cost and failure drift before your users report it.
- →Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
- →Cut Agent Costs by 60% Without Losing Quality
A measurable cost-reduction pass for any agent already in production.
- →Ship an Autonomous Workflow Safely
Move an automation from human-triggered to autonomous without losing control.
Related tool stacks
The tools that run it in production.
- →Agent Economics Observability Stack
This stack provides tools to monitor, analyze, and optimize the economic performance of AI agents, focusing on token costs, performance, and ROI.
- →AI Security Stack
Least-privilege tooling, approval gates and audit trails for agentic systems.
- →AI Cost Optimization Stack
This stack provides tools and services for monitoring, analyzing, and controlling the operational costs associated with AI agent deployment and LLM usage.
- →AI Compliance Monitoring Stack
This stack provides a set of tools and technologies for continuously monitoring AI systems to ensure ongoing adherence to regulatory requirements like the EU AI Act and data privacy laws.
Related prompts
Reusable prompts for this job.
- →Eval Rubric Prompt
Builds a scoring rubric a grader model can apply consistently.
Related use cases
How people apply it, and what came out.
- →Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.
Comparisons & alternatives
Pick between the options.
- →Model-Graded Evals vs Assertion Evals
Assertions are cheap, fast and objective; model grading captures quality you cannot express as a rule.