Draw by hand

Which samples pass, and how big is k?

n=5 completions are sampled for one task. Toggle each sample between pass and fail, drag the budget k, and watch the plug-in, the unbiased estimator, and a full brute-force enumeration update together.

Click a sample to flip it between pass and fail. Drag k to change how many of the n=5 you'd draw.

All C(5,2) = 10 size-2 subsets

Plug-in (biased)0.000
Unbiased0.000
Brute-force0.000
✓ matches unbiased

Why plug-in lies

The plug-in is biased low, on average

Fix n=5, k=2, and a true per-sample pass rate p. Slide p and compare the exact truth, 1-(1-p)^k, against the exact expectation of each estimator over c ~ Binomial(5, p), no sampling involved.

True pass@2
0.000
1-(1-p)^2
E[unbiased]
0.000
exact match
E[plug-in]
0.000
low by 0.000