Analysis
Deeper analysis surfaces for cost/quality/latency optimization, drift detection, and eval coverage.
Deeper analysis surfaces that build on diagnostics. All commands are deterministic — no LLM calls.
Tradeoffs
Cost, quality, and latency optimization. Finds where you're overpaying, which segments can use cheaper models, and where latency is bottlenecked.
bandito analyze tradeoffs --project my-chatbot
bandito analyze tradeoffs --project my-chatbot --tag prod
bandito analyze tradeoffs --project my-chatbot --days 7
bandito analyze tradeoffs --project my-chatbot --last 100
What it shows
Tradeoffs: my-chatbot
200 traces · 2 models · 3 segments with evaluations
── Model Efficiency ──────────────────────────────
MODEL TRACES QUALITY AVG COST AVG LAT
gpt-4o (openai) 120 85% $0.0052 1.05s
gpt-4o-mini (openai) 80 79% $0.0012 0.42s
gpt-4o-mini scores within 6% of gpt-4o at 77% lower cost.
── Segment Opportunities ─────────────────────────
simple-queries (85 traces)
gpt-4o 91% quality · $0.0048
gpt-4o-mini 89% quality · $0.0011
→ gpt-4o-mini matches quality here at 77% cost savings
complex-queries (65 traces)
gpt-4o 82% quality · $0.0061
gpt-4o-mini 64% quality · $0.0014
→ gpt-4o justified — 18pp quality gap on complex queries
── Cost Drivers ──────────────────────────────────
Top 10% of traces by cost account for 38% of total spend
$0.0089+ per trace vs $0.0034 average
── Quality vs Cost ───────────────────────────────
Cheaper half: 82% avg quality · $0.0018 avg cost
Expensive half: 86% avg quality · $0.0058 avg cost
→ 4pp quality gap for 3.2x the cost
Drift
Detect when quality, cost, or latency shifts significantly. Uses statistical changepoint detection (CUSUM + rolling baseline) to flag regressions.
bandito analyze drift --project my-chatbot
bandito analyze drift --project my-chatbot --days 30
bandito analyze drift --project my-chatbot --all
What it shows
Drift Analysis: my-chatbot
150 traces · 14 days
── Drift Detected ──────────────────────────────
[!] QUALITY ↓
Pass rate dropped from 90% to 76% (15pp change)
Window: 2026-03-25
Traces: 8
[!] LATENCY ↑
Latency increased from 1.50s to 2.22s (48% change)
Window: 2026-03-25
Traces: 8
How it works
- Buckets traces into 1-day windows
- Computes metrics (quality, latency, cost) per window
- Compares first half (baseline) vs second half (recent)
- Flags changes >15% as drift
- Uses CUSUM to detect sudden 3-sigma shifts
Coverage
Find where you're blind. Identifies segments with no or low evaluation coverage, including structural clusters.
bandito analyze coverage --project my-chatbot
bandito analyze coverage --project my-chatbot --all
What it shows
Coverage Analysis: my-chatbot
100 traces · 65 graded
Eval Coverage: [███████████████░░░░] 65%
── Blind Spots ─────────────────────────────────────
model:gpt-4o-mini (30 traces)
0% graded · 30% of traffic
tag:experimental (45 traces)
0% graded · 45% of traffic
── Judge Calibration ────────────────────────────────
Status: Fresh (3 days ago)
TPR: 92% TNR: 88%
15 new grades since calibration
── Recommendation ──────────────────────────────────
Grade 10 traces from blind spots: model:gpt-4o-mini.
Structural clustering
Coverage analysis includes structural clusters — behavioral patterns beyond tags and models:
| Feature | Values |
|---|---|
| Token size | tiny (<100), small (100-1k), medium (1k-10k), large (>10k) |
| Span composition | llm_only, retrieval_then_llm, agent_loop, embedding_heavy |
| Span count | single (1-2), few (3-5), many (6-10), loop (>10) |
| Error status | ok, error |
Cluster key example: medium/retrieval_then_llm/few/ok
This reveals blind spots in behavioral patterns — not just which topics lack evaluation, but which types of traces (simple Q&A vs RAG pipelines vs agent loops) are under-graded.
Options
All analyze commands share these options:
--project NAME Project name
--tag TAG Filter traces by tag
--last N Only use the last N traces
--days N Analyze last N days (default: 14)
--all Analyze all traces (ignore --days)
--json Output as JSON