Datasets

Datasets are curated collections of traces for regression testing, comparison, and offline evaluation.

Create a dataset

bandito dataset create golden-set --project my-chatbot --human-score-eq 1
bandito dataset create problem-cases --project my-chatbot --judge-score-lt 0.5

Creates a dataset from traces matching criteria:

Filter	Description
--human-score-eq 0|1	Filter by human score
--judge-score-eq 0|1	Filter by judge score
--judge-score-lt N	Filter by judge score less than N
--limit N	Max traces to include

List datasets

bandito dataset list --project my-chatbot

Inspect a dataset

bandito dataset inspect golden-set --project my-chatbot

View the traces in a dataset.

Compare datasets

Compare a dataset against current production:

bandito dataset compare golden-set --project my-chatbot

Regression testing

Run diagnostics on a dataset to catch quality regressions:

bandito dataset regression golden-set --project my-chatbot

Export for CI

bandito dataset export golden-set --project my-chatbot --format pytest --output test.py

Export formats: pytest, json, csv.

Run judge on a dataset

Score a dataset before deploying:

bandito judge run --project my-chatbot --dataset golden-set

Add traces from TUI

In the grading TUI (bandito tui --project NAME), press a to add the current trace to a dataset.

Use cases

• Golden sets — curated examples of good outputs for regression testing
• Problem cases — traces that failed quality checks for focused debugging
• A/B testing — compare outputs before and after prompt changes
• CI/CD gates — run assertions and judge on datasets before deploying