2012-11 Combination of “Combinations of P-values”

We investigate the impact of an uncertain number of false individual null hypotheses on commonly used p-value combination methods and find that the performance of these methods varies substantially under such uncertainty. These variations yield conflicting results in meta-analysis, motivating the development of a new, reconciling test. We consequently develop a combination of “combinations of p-values” (CCP) test that maintains good power properties across such uncertainty. We base the CCP test on a simple union of rejections decision rule that exploits the similarity between any two p-value combination methods. Monte Carlo simulations show that our test controls size and closely tracks the power of the best individual methods.