Three-layer eval, the gameable judge, and the regression gate

Through sixteen parts we judged the agent by eyeball, which only checks the answer. An agent can reach the right answer through a wrong, wasteful, or unsafe path. The fix is to grade a run three ways and gate it in CI. Toggle the run, the narration, and the token envelope below; every number is reproduced verbatim from the verified offline run.

The world (one last time): the refund agent. The golden path is search_policy then process_refund(ORD-3300, $180) then finish, three steps, budget three. The motivating case is a run that lands the same correct refund through a wrong path. Eyeballing the final answer would ship it.
1 Three layers grade one run
The trajectory, against the golden path
golden
this run
Graded three orthogonal ways
Outcome PASS
World STATE, not text. Did the ledger end correct? (tau2-style)
ledger == {ORD-3300: 180.0}
Trajectory PASS
Was the PATH good? Set precision/recall, order-aware edit distance, steps vs budget.
Component PASS
Unit checks on the pieces. Every refund call has a positive amount and a real order id.
amount > 0, order_id starts ORD-
2 The LLM judge is gameable; the programmatic guard is not same bad path, both ways

We grade the SAME wrong-path run. A rubric LLM judge scores its narration; the programmatic guard scores its path. Flip the narration and watch only the judge swing.

rubric LLM judge
0.6
terse correct-ish prose
programmatic guard
FAIL
precision 0.75, recall 1.0, edit 2, steps 5/3
unchanged by narration
The judge rewards confident prose; the guard measures the path. Verbose self-narration lifts the judge from 0.6 to 0.9 for the SAME bad path, while the guard holds FAIL. Answer-quality judge biases were cataloged in RAG Part 11; we do not re-list them here.
3 The golden-trajectory regression gate

Replay the four-case suite. Each case asserts outcome AND tool-call correctness AND the operating envelope. Drag the token budget down from 400 to 200 and the good-path case flips RED on cost, even though its path is perfect.

operating envelope ≤ 400 tokens/run
tight 200 loose 400
case (task) steps cost verdict
gate
green cases out of four
cost per success
This is the finale: eyeballing got us here, a three-layer eval plus a golden-trajectory gate is what keeps it here. Outcome is world STATE + POLICY, not the text saying done. Trajectory catches the right-answer-wrong-path run. The security case (the Part 16 poisoned ticket) is just another row, asserting no unauthorized refund and no exfiltration. A security guarantee you do not test is one you do not have.
How we keep it honest: every number is reproduced verbatim from the verified offline run of agent_eval.py (no API key, no network). Good path: precision=1.0 recall=1.0 edit=0 steps=3/3. Wrong path: precision=0.75 recall=1.0 edit=2 steps=5/3. Judge 0.6 -> 0.9; guard FAIL throughout. Gate at ≤400: 3/4 green, cost-per-success $0.00320; tightened to ≤200: 2/4 green, $0.00480. The deterministic judge is the offline stand-in for a hosted rubric judge; only the generate() swap would need edits. Multi-turn and long-horizon trajectory eval is the reader's extension.

outcome   trajectory   judge / caution   green / pass   red / fail