Assertions

Run programmatic checks against your traces. Assertions are Python functions — write any check you can express in code.

Write assertions

Create a Python file with functions that take a trace dict and return True (pass) or False (fail):

# my_assertions.py

def response_not_empty(trace):
    """Output should never be empty."""
    return bool(trace.get("output"))

def cost_under_threshold(trace):
    """Each trace should cost less than $0.10."""
    total = sum(s.get("cost", 0) for s in trace.get("spans", []))
    return total < 0.10

def no_error_spans(trace):
    """No spans should have error status."""
    return all(s.get("status") != "error" for s in trace.get("spans", []))

def latency_under_5s(trace):
    """Total trace latency under 5 seconds."""
    from datetime import datetime
    started = trace.get("started_at", "")
    ended = trace.get("ended_at", "")
    if not started or not ended:
        return True
    s = datetime.fromisoformat(started.replace("Z", "+00:00"))
    e = datetime.fromisoformat(ended.replace("Z", "+00:00"))
    return (e - s).total_seconds() < 5.0

Each function's name becomes the assertion identifier in results.

Run from CLI

bandito assert --project my-chatbot --file my_assertions.py
bandito assert --project my-chatbot --file my_assertions.py --tag prod
bandito assert --project my-chatbot --file my_assertions.py --last 50

Output:

  Assertion Results
  ──────────────────────────────────────────────────────

  Traces:     100
  Assertions: 4

  ✓ response_not_empty              100/100 passed
  ✗ cost_under_threshold            92/100 passed · 8 failed
  ✓ no_error_spans                  100/100 passed
  ✓ latency_under_5s                98/100 passed · 2 failed

  10 failed out of 400 checks.

Exit code is 0 if all pass, 1 if any fail — works in CI.

Options

--tag TAG      Filter traces by tag (e.g. prod)
--last N       Only check the last N traces
--store        Save results to Bandito backend

Run from SDK

from bandito import assert_traces

def output_not_empty(trace):
    return bool(trace.get("output"))

def cost_ok(trace):
    total = sum(s.get("cost", 0) for s in trace.get("spans", []))
    return total < 0.10

result = assert_traces(
    "my-chatbot",
    assertions=[output_not_empty, cost_ok],
    last=100,
)

print(result.summary())
print(f"All passed: {result.all_passed}")

# Group by assertion name
for name, counts in result.by_assertion().items():
    print(f"{name}: {counts['passed']} passed, {counts['failed']} failed")

What a trace looks like

Your assertion functions receive a trace dict:

{
    "trace_id": "tr-abc-123",
    "project": "my-chatbot",
    "input": "user's question",
    "output": "llm's response",
    "status": "ok",
    "started_at": "2026-03-25T12:00:00Z",
    "ended_at": "2026-03-25T12:00:02Z",
    "metadata": {"tier": "pro"},
    "tags": ["prod"],
    "spans": [
        {
            "span_id": "sp-1",
            "kind": "retrieval",
            "input": "search query",
            "output": ["doc1", "doc2"],
            "status": "ok",
            "cost": 0.0,
        },
        {
            "span_id": "sp-2",
            "kind": "llm",
            "model": "gpt-4o",
            "provider": "openai",
            "input_tokens": 312,
            "output_tokens": 156,
            "cost": 0.003,
            "status": "ok",
        },
    ],
}

When to use assertions vs judge

	Assertions	Judge
What it checks	Structural, deterministic (cost, latency, format)	Semantic quality (is the response good?)
How	Python functions	LLM + rubric
Cost	Free (no LLM calls)	Paid (LLM calls per trace)
Best for	CI gates, SLA monitoring, regression checks	Quality scoring at scale

Use both. Assertions catch the obvious. Judge catches the subtle.

What's next

See Datasets to create curated test sets from your traces.