UZH-Logo

Maintenance Infos

Avoiding "data snooping" in multilevel and mixed effects models


Afshartous, David; Wolf, Michael (2007). Avoiding "data snooping" in multilevel and mixed effects models. Journal of the Royal Statistical Society: Series A, 170(4):1035-1059.

Abstract

Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for "data snooping", i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the "absolute" level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.

Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for "data snooping", i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the "absolute" level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.

Citations

9 citations in Web of Science®
8 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

53 downloads since deposited on 28 Mar 2008
11 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Economics
Dewey Decimal Classification:330 Economics
Language:English
Date:October 2007
Deposited On:28 Mar 2008 09:36
Last Modified:05 Apr 2016 12:22
Publisher:Royal Statistical Society
ISSN:0964-1998
Additional Information:The definitive version is available at www.blackwell-synergy.com
Publisher DOI:10.1111/j.1467-985X.2007.00494.x
Permanent URL: http://doi.org/10.5167/uzh-2300

Download

[img]
Preview
Content: Accepted Version
Filetype: PDF
Size: 355kB
View at publisher

TrendTerms

TrendTerms displays relevant terms of the abstract of this publication and related documents on a map. The terms and their relations were extracted from ZORA using word statistics. Their timelines are taken from ZORA as well. The bubble size of a term is proportional to the number of documents where the term occurs. Red, orange, yellow and green colors are used for terms that occur in the current document; red indicates high interlinkedness of a term with other terms, orange, yellow and green decreasing interlinkedness. Blue is used for terms that have a relation with the terms in this document, but occur in other documents.
You can navigate and zoom the map. Mouse-hovering a term displays its timeline, clicking it yields the associated documents.

Author Collaborations