Evaluate
Replay labelled tool calls through your contracts and the benchmarks, and reproduce each result.
Replay labelled tool calls through your contracts and the benchmarks.
coco eval scores every decision: did a call that had to run, run, and did a call that had to stop, stop. Start with cases, which runs in any project. The benchmarks need an SDK checkout.
Suites
| Suite | What it measures | Runs from |
|---|---|---|
cases | Your compiled contracts against the compiler's boundary cases. | Any project. |
vault | Rules and the classifier on attacks against a payment agent. | An SDK checkout. |
tau3 | Rules on legitimate banking traffic and on policy violations. | An SDK checkout. |
agentdojo | Rules and the classifier on instructions planted in what an agent reads. | An SDK checkout. |
policy | The policy pipeline's extracted rules against hand-labelled gold rules. It never reaches the engine. | An SDK checkout. |
Arms
rules: The contracts alone.judge: The classifier alone.hybrid: The rules, then the classifier on what they let through.none: Every call unchecked.
| Suite | Arms | Default |
|---|---|---|
cases | rules, and hybrid when a classifier is set | rules |
vault | rules, hybrid, judge | rules |
tau3 | rules, hybrid | rules |
agentdojo | none, rules, judge, hybrid | none,rules |
Keep in mind: The classifier arms need coco-classifier, which isn't released. Set COCO_CLASSIFIER_URL or pass --classifier URL. The classifier's answers move by a few calls between runs. The rules arms repeat exactly.
Run your own cases
coco contracts compile writes boundary cases to .coco/cases/, each with the verdict the policy text implies. Replay them:
coco eval cases # rules only
coco eval cases --classifier "$COCO_CLASSIFIER_URL"Every contract is treated as live, and the rules arm drops judge checks. Coco prints each case where its verdict differs, and ends with N of M cases got exactly the expected verdict with rules only. It exits 1 when any case misses.
Run it after every compile and before coco contracts activate --live.
vault: payments under attack
A corpus of 73,356 payment transfer calls built from 4,371 human-written attacks on a bank-teller agent, at five levels of payment policy. Every conversation is an attack, so the suite asks whether a refused transfer ran and whether a permitted transfer was stopped. Each transcript is one Coco session, so a daily cap sees the transfers before it.
| Arm | Result |
|---|---|
| rules | 0 of 15,550 refused transfers ran. 0 of 57,806 permitted transfers stopped. |
| rules, decision time | 0.27 ms at the median, 0.51 ms at the 95th percentile, in process. |
| judge | 454 of 15,550 refused transfers ran. |
To reproduce, use an SDK checkout with trustcoco-sdk[eval] and access to the corpus:
hf auth login # an account which accepted the corpus's terms, or HF_TOKEN
coco eval vault --arms rules # every call
coco eval vault --arms rules,hybrid,judge --classifier "$COCO_CLASSIFIER_URL" --workers 32 --timeout 60tau3: banking
tau3-bench's banking domain, scored per decision rather than per task.
- Legitimate: 825 calls a correct run makes, in order, against the bank's records. Each must run.
- Probes: 316 policy violations, each one change to a legitimate call or a call built from real records. Each must stop.
| Arm | Result |
|---|---|
| rules | 0 of 825 legitimate calls stopped. 0 of 316 probes missed. |
| hybrid | 37 of 825 legitimate calls stopped. 0 of 316 probes missed. |
The classifier adds a cost here and catches nothing the rules missed. This suite says nothing about whether an agent completes the task.
To reproduce, use the public tau2-bench clone on its dev/tau3 branch:
git clone --depth 1 --branch dev/tau3 https://github.com/sierra-research/tau2-bench.git tau3-bench
export TAU3_BANKING="$PWD/tau3-bench/data/tau2/domains/banking_knowledge"
coco eval tau3 --arms rules,hybrid --classifier "$COCO_CLASSIFIER_URL" --workers 16 --timeout 60agentdojo: planted instructions
AgentDojo runs the same user tasks clean and with an attacker's instruction planted in an email, a file, a web page, or a payment note. It has 97 legitimate tasks and 588 attacks.
| Arm | Attacks prevented | Legitimate tasks held for a person |
|---|---|---|
| rules | 467 of 588 | 18 of 97 |
| rules and classifier | 582 of 588 | 18 of 97 |
The rules are ones a deployer writes from facts its systems hold: known payees, known contacts, and credential and membership changes and deletions sent to a person. In the banking suite, three contracts stopped all 144 attacks, the known-payee rule alone 112 of them, and held 5 of 16 legitimate tasks. Outside banking, the rules miss attacks whose action they permit, and the classifier stops most of those.
Keep in mind: The classifier's question in this run is wording v2, the default. It was written after seeing the first wording's results on this benchmark, so it's tuned to it.
To reproduce:
uv run --no-project --python 3.12 --with agentdojo==0.1.35 \
python bench/agentdojo/export.py --out bench/agentdojo/cases.jsonl
coco eval agentdojo --arms none,rules
coco eval agentdojo --arms judge,hybrid --question v2 --classifier "$COCO_CLASSIFIER_URL" --workers 16To count the banking results per rule, run this from the checkout with the stamp of your run:
python3 -c "import json,collections as c;f={};[f.setdefault(r['session_id'],r['deciding']) for r in map(json.loads,open('bench/agentdojo/results/<stamp>-rules.jsonl')) if r['group']=='banking attack' and r['should_stop'] and r['stopped']];print(c.Counter(f.values()))"
# known_payee 112, recipient_known 16, a_person_changes_credentials 16--question v1|v2 picks the wording, and --threshold sets the probability at which a judge check escalates, 0.5 by default.
policy: extraction
This suite scores the policy pipeline, not enforcement. It matches each extracted rule against the gold rules a careful reader finds in the same document, and reports recall, precision, effects, conditions, numbers, and approvers, per set, per domain, and overall.
coco eval policy --predictions DIR # score output the pipeline already wrote
coco eval policy --run --sets payments --docs 2 --max-pages 3 # run the pipeline on a selection, then score--run needs the pipeline and its model key, and the suite needs the policy corpus with its gold rules.
Keep in mind: The gold is model annotated and no person has reviewed it yet, so a score measures agreement with the model's annotation.
Results files
vault, tau3, agentdojo, and policy write a summary headed by the command that produced it, to bench/vault/results/, bench/tau3_banking/results/, bench/agentdojo/results/, and bench/policy/results/. vault, tau3, and agentdojo also write one file of per-call outcomes for each arm. cases prints its result and writes nothing.
The runs above are vault 20260929-153604 and 20260929-184944, tau3 20260929-205034, and agentdojo 20260929-185238 and 20260929-184422.
To list a suite's options, pass --help after its name, such as coco eval vault --help.