Formalized Data Snooping Based on Generalized Error Rates

Abstract

It is common in econometric applications that several hypothesis tests are carried out at the same time. The problem then becomes how to decide whichnhypotheses to reject, accounting for the multitude of tests.nThe classical approach is to control the familywise error rate (FWE), that is, thenprobability of one or more false rejections. But when thennumber of hypotheses under consideration is large, control of the FWE can become too demanding. As a result, the number of false hypotheses rejected may be small or even zero. This suggests replacingncontrol of the FWE by a more liberal measure. To this end,nwe review a number of proposals from the statistical literature.nWe briefly discuss how these procedures apply to the general problem of model selection. A simulation study and two empirical applications illustrate the methods.