McNemar, live
Drag the discordant cells, watch the p-value
Two models graded on the same eval set. Only the items where they split (A right & B wrong = b, B right & A wrong = c) carry any signal; the rest are concordant and silent.
Set b, c, and the eval-set size n. The concordant count (items both models agree on) fills in automatically as n − b − c.
Headline gap (B − A)
(b−c)/n
chi-square, uncorrected
(b−c)²/(b+c)
chi-square, Yates corrected
(|b−c|−1)²/(b+c)
approx p (chi-square, 1 df)
erfc(√(chi-square/2))
exact p (binomial, two-sided)
2·ΣC(b+c,i) / 2^(b+c)
NOT significant. p = 1.000 ≫ 0.05. A 5-vs-4 split over 9 disagreements is exactly what a fair coin looks like.
How much data would it take?
The set size a real gap needs, at 80% power
Same b, c, n from above. If the gap above were genuinely real, how many items would you need to catch it four times out of five?
Baseline held fixed at pA = 0.82 (Model A's true accuracy); pB = pA + gap. Target: alpha = 0.05 (z = 1.96), 80% power (z = 0.84), same as Part 7's worked example.
Unpaired, per model
5528
two independent evals
Paired, total items
3515
633 discordant pairs needed
Unpaired items needed5528
Your 50-item set is about 70× smaller than the ~3515 items a paired eval would need to reliably catch a +2.0 point gain. Pairing helps (3515 < 5528), but a small eval simply cannot resolve a small gap.