Nested sampling for frequentist computation: fast estimation of small $p$-values. (arXiv:2105.13923v2 [physics.data-an] UPDATED)
<a href="http://arxiv.org/find/physics/1/au:+Fowlie_A/0/1/0/all/0/1">Andrew Fowlie</a>, <a href="http://arxiv.org/find/physics/1/au:+Hoof_S/0/1/0/all/0/1">Sebastian Hoof</a>, <a href="http://arxiv.org/find/physics/1/au:+Handley_W/0/1/0/all/0/1">Will Handley</a>
We propose a novel method for computing $p$-values based on nested sampling
(NS) applied to the sampling space rather than the parameter space of the
problem, in contrast to its usage in Bayesian computation. The computational
cost of NS scales as $log^2{1/p}$, which compares favorably to the $1/p$
scaling for Monte Carlo (MC) simulations. For significances greater than about
$4sigma$ in both a toy problem and a simplified resonance search, we show that
NS requires orders of magnitude fewer simulations than ordinary MC estimates.
This is particularly relevant for high-energy physics, which adopts a $5sigma$
gold standard for discovery. We conclude with remarks on new connections
between Bayesian and frequentist computation and possibilities for tuning NS
implementations for still better performance in this setting.
We propose a novel method for computing $p$-values based on nested sampling
(NS) applied to the sampling space rather than the parameter space of the
problem, in contrast to its usage in Bayesian computation. The computational
cost of NS scales as $log^2{1/p}$, which compares favorably to the $1/p$
scaling for Monte Carlo (MC) simulations. For significances greater than about
$4sigma$ in both a toy problem and a simplified resonance search, we show that
NS requires orders of magnitude fewer simulations than ordinary MC estimates.
This is particularly relevant for high-energy physics, which adopts a $5sigma$
gold standard for discovery. We conclude with remarks on new connections
between Bayesian and frequentist computation and possibilities for tuning NS
implementations for still better performance in this setting.
http://arxiv.org/icons/sfx.gif
