How often is a real gain kept?

Try it: the keep rule illustrative model

How often is a real improvement kept?

An edit truly lifts every task by the true gain. Each task score is measured once, with run-to-run noise. The rule keeps the edit if the mean difference over the dev tasks clears the bar. The defaults are this study's measured values: a per-task standard deviation of about 0.06, 6 dev tasks, a bar of 0.0514.

A simplified model (normal noise, independent tasks); the study's own rule also prices cost and unstable runs.