Your mini AI Engineer
You grade 15 traces. Bandito turns that into 500 scores, finds the patterns, and shows you exactly what to fix. Your judgment, amplified.
$ bandito judge run --project my-agent
Scoring 1,247 traces against rubric v3...
████████████████████████████████████ 100%
pass rate: 87% · 153 failures
~ context window errors spiking (47)
~ retrieval quality low on long queries (31)
$ bandito analyze tradeoffs --project my-agent
gpt-4o 320 calls · 89% quality · $0.028/trace
gpt-4o-mini 927 calls · 82% quality · $0.002/trace
· 60% of traces could use mini w/ 7% quality loss
· save ~$8.20/week with model routing
$ bandito improve replay --project my-agent
quality: 87% → 88% (+1%)
cost: $0.018 → $0.006/trace (-67%)
✓ Safe to ship.You have traces. Now what?
Trace loggers tell you what happened. They don’t tell you what’s broken, why, or what to do about it.
Your dashboard says
“1,247 traces logged”
You still have to figure out which ones are bad.
Your dashboard says
“avg latency: 2.3s”
You still have to decide if that’s a problem.
Your dashboard says
“model: gpt-4o”
You still have to test if a cheaper model works.
Bandito closes the gap between observing and improving.
Five steps. One workflow.
Each step produces data the next step consumes. Your effort compounds with every cycle.
Observe
Connect to your traces in one command. Local, S3, Postgres, or Langfuse.
Grade
Score a handful by hand. Fast, keyboard-driven TUI. This is the signal Bandito amplifies.
Judge
Your 15 grades become 500 scores. LLM-as-judge learns your rubric, evaluates every trace.
Analyze
Find cost/quality/latency tradeoffs hiding in your data. Deterministic — no LLM needed.
Improve
Replay traces with a new config. See the diff before your users do.
$ bandito observe traces --project my-agent
1,247 traces · 12,847 spans · last 14 days
$ bandito tui
Graded 15 traces · 13 pass · 2 fail
$ bandito judge run --project my-agent
pass rate: 87% · 153 failures
~ context window errors spiking
$ bandito analyze tradeoffs --project my-agent
60% of traces could use mini
save ~$8.20/week
$ bandito improve replay --project my-agent
quality: 87% → 88%
cost: $0.018 → $0.006/trace (-67%)
✓ Safe to ship.First cycle
Slow but educational
You grade traces by hand, write a rubric, calibrate. You learn what “good” means for your app.
Second cycle
One command
Rubric is calibrated. judge run scores every new trace in minutes. Analysis is instant.
Ongoing
Continuous coverage
Judge runs after every deploy. Regressions caught before users notice. 15 minutes of grading → continuous quality coverage.
Plug in, don’t rip out
Bandito reads your traces wherever they already live. No migration, no vendor lock-in.
Trace providers
Langfuse, Braintrust, or the Bandito SDK
Storage
Local JSON, S3, or PostgreSQL
Frameworks
Pydantic AI, OpenAI, Anthropic, LangChain