Grade by hand

Eight agent runs, four ways to score the path

The reference trajectory is search → open → convert → answer. Pick a run and watch exact-match, order-aware, set-overlap, and step credit update.

Click a run to load a preset, or build your own with the tool buttons below. Each candidate step is labelled MATCH or EXTRA; any reference step the run never took is labelled MISSING.

Reference

Candidate trajectory, step by step

Build your own: append a step

Exact-match
0.00
Order-aware
0.00
Set-overlap
0.00
Step credit
0.00

Chance-correct the agreement

Mock judge vs. human: toggle a grade, watch kappa move

Each run got a PASS/FAIL rubric grade from a lenient mock judge and from a human. Click any grade to flip it and see the 2×2 table, observed agreement po, chance agreement pe, and Cohen's kappa recompute live.

RunHumanJudgeAgree?
Observed po
0.00
Chance pe
0.00
Cohen’s kappa
0.00