Header

UZH-Logo

Maintenance Infos

Avoiding Data Snooping in Multilevel and Mixed Effects Models


Afshartous, David; Wolf, Michael (2005). Avoiding Data Snooping in Multilevel and Mixed Effects Models. Working paper series / Institute for Empirical Research in Economics No. 260, University of Zurich.

Abstract

"Multilevel or mixed effects models are commonly applied to hierarchical data; for example,nsee Goldstein (2003), Raudenbush and Bryk (2002), and Laird and Ware (1982). Although therenexist many outputs from such an analysis, the level-2 residuals, otherwise known as randomneffects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the modelnassumptions at the group level. Current inference on the level-2 residuals, however, typicallyndoes not account for data snooping, that is, for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: (1) Inference on the `absolute' level-2 residuals tondetermine which are significantly different from zero, and (2) Inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London Education Authority data used by Rasbash et al. (2004), the Wafer data used by Pinheiro and Bates (2000), and the NELS data used by Afshartous and de Leeuw (2004)."

Abstract

"Multilevel or mixed effects models are commonly applied to hierarchical data; for example,nsee Goldstein (2003), Raudenbush and Bryk (2002), and Laird and Ware (1982). Although therenexist many outputs from such an analysis, the level-2 residuals, otherwise known as randomneffects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the modelnassumptions at the group level. Current inference on the level-2 residuals, however, typicallyndoes not account for data snooping, that is, for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: (1) Inference on the `absolute' level-2 residuals tondetermine which are significantly different from zero, and (2) Inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London Education Authority data used by Rasbash et al. (2004), the Wafer data used by Pinheiro and Bates (2000), and the NELS data used by Afshartous and de Leeuw (2004)."

Statistics

Downloads

448 downloads since deposited on 29 Nov 2011
4 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:03 Faculty of Economics > Department of Economics
Working Paper Series > Institute for Empirical Research in Economics (former)
Dewey Decimal Classification:330 Economics
Language:English
Date:December 2005
Deposited On:29 Nov 2011 22:31
Last Modified:07 Dec 2017 10:11
Series Name:Working paper series / Institute for Empirical Research in Economics
ISSN:1424-0459
Official URL:http://www.econ.uzh.ch/wp.html

Download

Download PDF  'Avoiding Data Snooping in Multilevel and Mixed Effects Models'.
Preview
Filetype: PDF
Size: 361kB