LLM-as-Judge
Scale your quality evaluation beyond manual grading. Define a rubric, calibrate it against your human grades, then score all traces automatically.
Scale your quality evaluation beyond manual grading. Define a rubric, calibrate it against your human grades, then score all traces automatically.
The flow
Grade 15+ traces → judge calibrate → judge run
Calibrate
bandito judge calibrate --project my-chatbot
bandito judge calibrate --project my-chatbot --tag prod
This is the single entry point. It handles everything:
- Creates a rubric if none exists — drafts criteria from your system prompt using an LLM, lets you edit in
$EDITOR - Seeds examples from your human-graded traces
- Runs calibration — scores your graded traces with the judge and compares to your human grades
- Reports agreement — TPR, TNR, Cohen's kappa, disagreement details
Calibration output
Calibration Results
──────────────────────────────────────────────────────
Traces evaluated: 18
✓ TPR (recall): 94% (target: ≥90%)
✓ TNR (specificity): 91% (target: ≥90%)
Agreement: 93%
Cohen's kappa: 0.85
Judge is well-calibrated.
Ready to score: bandito judge run --project my-chatbot
If TPR or TNR is below 90%, refine your rubric and re-calibrate. The disagreement report shows exactly where the judge gets it wrong.
Run
bandito judge run --project my-chatbot
bandito judge run --project my-chatbot --tag prod --sample 100
Scores ungraded traces. Requires passing calibration.
Scoring 247 traces for 'my-chatbot'...
Judge Run Results
──────────────────────────────────────────────────────
Traces scored: 247
Passed: 211 (85%)
Failed: 36 (15%)
1200 total traces · 45 human-graded (4%)
Low human grading coverage. For better judge accuracy:
1. Grade in TUI: bandito tui --project my-chatbot
2. Re-calibrate: bandito judge calibrate --project my-chatbot
Options
--sample N Randomly sample N traces (avoids recency bias)
--rejudge Re-score traces that already have judge scores
--tag TAG Only score traces with this tag
--dry-run Show the prompt for the first trace without calling the LLM
Calibration gate
judge run checks that:
- A rubric exists
- Calibration passed (TPR and TNR ≥ 90%)
- Calibration is not stale (rubric wasn't edited after last calibration)
- Fewer than 100 new human grades since calibration (nudges you to recalibrate)
If any check fails, it offers to run calibrate first.
Grading coverage
After scoring, the run reports how many traces have human grades. Below 15%, it nudges you to grade more and recalibrate. More human grades = better calibrated judge.
Overview
bandito judge overview --project my-chatbot
bandito judge overview --project my-chatbot --json
Current state of your judge at a glance — calibration health and usage.
Judge Overview: my-chatbot
── Calibration ───────────────────────────
Status: well-calibrated
✓ TPR (recall): 94% (target: ≥90%)
✓ TNR (spec): 92% (target: ≥90%)
Agreement: 93%
Kappa: 0.85
Traces: 50
Calibrated: 2h ago
Disagreements (3):
tr-abc123 fp "Missed hallucination in paragraph 2"
tr-def456 fn "Output was actually correct, judge too strict"
tr-ghi789 fp "Factual error not caught"
── Usage ─────────────────────────────────
Total runs: 5
Total scored: 580
Avg pass rate: 73%
Latest run: 125 traces · 80% (+7% vs avg) · 3h ago
The disagreements section shows exactly which traces the judge gets wrong during calibration — the trace ID, the kind of error (fp = judge too lenient, fn = judge too strict), and the judge's critique. Use this to refine your rubric.
With --json, outputs structured data for programmatic consumption (Claude Code skills, CI pipelines).
Runs
bandito judge runs --project my-chatbot
bandito judge runs --project my-chatbot --last 5
bandito judge runs --project my-chatbot --json
Judge run history with trend context.
Judge Runs: my-chatbot
DATE SCORED PASSED FAILED RATE vs AVG FLAGS
2026-03-25 14:30 125 100 25 80% +7%
2026-03-24 10:15 80 72 8 90% +17% sample=80
2026-03-23 09:00 200 160 40 80% +7% rejudge
2026-03-22 16:45 100 65 35 65% -8%
2026-03-21 11:00 75 38 37 51% -22%
5 runs · 580 total scored · avg pass rate: 73%
The vs AVG column shows each run's pass rate relative to the weighted average across all runs. This tells you the trend at a glance — if recent runs are above average and older ones below, your judge quality is improving.
The average is weighted by traces scored, so a 200-trace run influences the baseline more than a 10-trace sample.
Judge scores in observe traces
When you've run the judge, bandito observe traces shows judge scores alongside human grades:
tr-abc123 3m ago · 4 spans · $0.0032 · 1.2s · graded ✓ · judge ✓
tr-def456 5m ago · 2 spans · $0.0011 · 0.8s · - grade · judge ✗
tr-ghi789 8m ago · 3 spans · $0.0021 · 1.5s · - grade
Use --json for structured output with full evaluation data (human scores, judge scores, critiques).
The rubric
Markdown with YAML frontmatter, stored on the Bandito backend. judge calibrate drafts it for you, but you should edit it:
---
description: Customer support chatbot
---
## What makes a good response
- Answers the user's question accurately
- Professional tone
- Grounded in knowledge base, no hallucination
## What makes a bad response
- Factually incorrect
- Ignores or misunderstands the question
- Makes up information not in the knowledge base
## Examples
### Good
**Input:** How do I reset my password?
**Output:** Go to Settings > Security > Reset Password.
**Critique:** Direct, accurate, actionable.
### Bad
**Input:** What's your refund policy?
**Output:** I'm not sure about that.
**Critique:** Doesn't attempt to answer. Should check knowledge base.
Aim for 2+ good and 2+ bad examples with detailed critiques.
Why this matters
15 human grades become 500 judge scores. This is the leverage point — every downstream feature (analysis, improvement testing, regression detection) benefits from quality data at scale. The grade → calibrate → run loop gets faster each cycle.