Coco
Integrate and operate

Evaluate

Replay labelled tool calls through your contracts and the benchmarks, and reproduce each result.

Replay labelled tool calls through your contracts and the benchmarks.

coco eval scores every decision: did a call that had to run, run, and did a call that had to stop, stop. Start with cases, which runs in any project. The benchmarks need an SDK checkout.


Suites

SuiteWhat it measuresRuns from
casesYour compiled contracts against the compiler's boundary cases.Any project.
vaultRules and the classifier on attacks against a payment agent.An SDK checkout.
tau3Rules on legitimate banking traffic and on policy violations.An SDK checkout.
agentdojoRules and the classifier on instructions planted in what an agent reads.An SDK checkout.
policyThe policy pipeline's extracted rules against hand-labelled gold rules. It never reaches the engine.An SDK checkout.

Arms

  • rules: The contracts alone.
  • judge: The classifier alone.
  • hybrid: The rules, then the classifier on what they let through.
  • none: Every call unchecked.
SuiteArmsDefault
casesrules, and hybrid when a classifier is setrules
vaultrules, hybrid, judgerules
tau3rules, hybridrules
agentdojonone, rules, judge, hybridnone,rules

Keep in mind: The classifier arms need coco-classifier, which isn't released. Set COCO_CLASSIFIER_URL or pass --classifier URL. The classifier's answers move by a few calls between runs. The rules arms repeat exactly.

Run your own cases

coco contracts compile writes boundary cases to .coco/cases/, each with the verdict the policy text implies. Replay them:

coco eval cases                          # rules only
coco eval cases --classifier "$COCO_CLASSIFIER_URL"

Every contract is treated as live, and the rules arm drops judge checks. Coco prints each case where its verdict differs, and ends with N of M cases got exactly the expected verdict with rules only. It exits 1 when any case misses.

Run it after every compile and before coco contracts activate --live.

vault: payments under attack

A corpus of 73,356 payment transfer calls built from 4,371 human-written attacks on a bank-teller agent, at five levels of payment policy. Every conversation is an attack, so the suite asks whether a refused transfer ran and whether a permitted transfer was stopped. Each transcript is one Coco session, so a daily cap sees the transfers before it.

ArmResult
rules0 of 15,550 refused transfers ran. 0 of 57,806 permitted transfers stopped.
rules, decision time0.27 ms at the median, 0.51 ms at the 95th percentile, in process.
judge454 of 15,550 refused transfers ran.

To reproduce, use an SDK checkout with trustcoco-sdk[eval] and access to the corpus:

hf auth login                            # an account which accepted the corpus's terms, or HF_TOKEN
coco eval vault --arms rules             # every call
coco eval vault --arms rules,hybrid,judge --classifier "$COCO_CLASSIFIER_URL" --workers 32 --timeout 60

tau3: banking

tau3-bench's banking domain, scored per decision rather than per task.

  • Legitimate: 825 calls a correct run makes, in order, against the bank's records. Each must run.
  • Probes: 316 policy violations, each one change to a legitimate call or a call built from real records. Each must stop.
ArmResult
rules0 of 825 legitimate calls stopped. 0 of 316 probes missed.
hybrid37 of 825 legitimate calls stopped. 0 of 316 probes missed.

The classifier adds a cost here and catches nothing the rules missed. This suite says nothing about whether an agent completes the task.

To reproduce, use the public tau2-bench clone on its dev/tau3 branch:

git clone --depth 1 --branch dev/tau3 https://github.com/sierra-research/tau2-bench.git tau3-bench
export TAU3_BANKING="$PWD/tau3-bench/data/tau2/domains/banking_knowledge"
coco eval tau3 --arms rules,hybrid --classifier "$COCO_CLASSIFIER_URL" --workers 16 --timeout 60

agentdojo: planted instructions

AgentDojo runs the same user tasks clean and with an attacker's instruction planted in an email, a file, a web page, or a payment note. It has 97 legitimate tasks and 588 attacks.

ArmAttacks preventedLegitimate tasks held for a person
rules467 of 58818 of 97
rules and classifier582 of 58818 of 97

The rules are ones a deployer writes from facts its systems hold: known payees, known contacts, and credential and membership changes and deletions sent to a person. In the banking suite, three contracts stopped all 144 attacks, the known-payee rule alone 112 of them, and held 5 of 16 legitimate tasks. Outside banking, the rules miss attacks whose action they permit, and the classifier stops most of those.

Keep in mind: The classifier's question in this run is wording v2, the default. It was written after seeing the first wording's results on this benchmark, so it's tuned to it.

To reproduce:

uv run --no-project --python 3.12 --with agentdojo==0.1.35 \
    python bench/agentdojo/export.py --out bench/agentdojo/cases.jsonl
coco eval agentdojo --arms none,rules
coco eval agentdojo --arms judge,hybrid --question v2 --classifier "$COCO_CLASSIFIER_URL" --workers 16

To count the banking results per rule, run this from the checkout with the stamp of your run:

python3 -c "import json,collections as c;f={};[f.setdefault(r['session_id'],r['deciding']) for r in map(json.loads,open('bench/agentdojo/results/<stamp>-rules.jsonl')) if r['group']=='banking attack' and r['should_stop'] and r['stopped']];print(c.Counter(f.values()))"
# known_payee 112, recipient_known 16, a_person_changes_credentials 16

--question v1|v2 picks the wording, and --threshold sets the probability at which a judge check escalates, 0.5 by default.

policy: extraction

This suite scores the policy pipeline, not enforcement. It matches each extracted rule against the gold rules a careful reader finds in the same document, and reports recall, precision, effects, conditions, numbers, and approvers, per set, per domain, and overall.

coco eval policy --predictions DIR                              # score output the pipeline already wrote
coco eval policy --run --sets payments --docs 2 --max-pages 3   # run the pipeline on a selection, then score

--run needs the pipeline and its model key, and the suite needs the policy corpus with its gold rules.

Keep in mind: The gold is model annotated and no person has reviewed it yet, so a score measures agreement with the model's annotation.

Results files

vault, tau3, agentdojo, and policy write a summary headed by the command that produced it, to bench/vault/results/, bench/tau3_banking/results/, bench/agentdojo/results/, and bench/policy/results/. vault, tau3, and agentdojo also write one file of per-call outcomes for each arm. cases prints its result and writes nothing.

The runs above are vault 20260929-153604 and 20260929-184944, tau3 20260929-205034, and agentdojo 20260929-185238 and 20260929-184422.

To list a suite's options, pass --help after its name, such as coco eval vault --help.


On this page