Header

UZH-Logo

Maintenance Infos

Bayesian mixed-effects inference on classification performance in hierarchical datasets


Brodersen, K H; Mathys, C; Chumbley, J R; Daunizeau, J; Soon Ong, C; Buchmann, J M; Stephan, K E (2012). Bayesian mixed-effects inference on classification performance in hierarchical datasets. Journal of Machine Learning Research, 13:3133-3176.

Abstract

Classification algorithms are frequently used on data with a natural hierarchical structure. For instance, classifiers are often trained and tested on trial-wise measurements, separately for each subject within a group. One important question is how classification outcomes observed in individual subjects can be generalized to the population from which the group was sampled. To address this question, this paper introduces novel statistical models that are guided by three desiderata. First, all models explicitly respect the hierarchical nature of the data, that is, they are mixed-effects models that simultaneously account for within-subjects (fixed-effects) and across-subjects (random-effects) variance components. Second, maximum-likelihood estimation is replaced by full Bayesian inference in order to enable natural regularization of the estimation problem and to afford conclusions in terms of posterior probability statements. Third, inference on classification accuracy is complemented by inference on the balanced accuracy, which avoids inflated accuracy estimates for imbalanced data sets. We introduce hierarchical models that satisfy these criteria and demonstrate their advantages over conventional methods using MCMC implementations for model inversion and model selection on both synthetic and empirical data. We envisage that our approach will improve the sensitivity and validity of statistical inference in future hierarchical classification studies.

Abstract

Classification algorithms are frequently used on data with a natural hierarchical structure. For instance, classifiers are often trained and tested on trial-wise measurements, separately for each subject within a group. One important question is how classification outcomes observed in individual subjects can be generalized to the population from which the group was sampled. To address this question, this paper introduces novel statistical models that are guided by three desiderata. First, all models explicitly respect the hierarchical nature of the data, that is, they are mixed-effects models that simultaneously account for within-subjects (fixed-effects) and across-subjects (random-effects) variance components. Second, maximum-likelihood estimation is replaced by full Bayesian inference in order to enable natural regularization of the estimation problem and to afford conclusions in terms of posterior probability statements. Third, inference on classification accuracy is complemented by inference on the balanced accuracy, which avoids inflated accuracy estimates for imbalanced data sets. We introduce hierarchical models that satisfy these criteria and demonstrate their advantages over conventional methods using MCMC implementations for model inversion and model selection on both synthetic and empirical data. We envisage that our approach will improve the sensitivity and validity of statistical inference in future hierarchical classification studies.

Statistics

Downloads

41 downloads since deposited on 07 Feb 2013
3 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Institute of Biomedical Engineering
Dewey Decimal Classification:170 Ethics
610 Medicine & health
Language:English
Date:2012
Deposited On:07 Feb 2013 08:26
Last Modified:17 Feb 2018 00:51
Publisher:Microtome Publishing
ISSN:1532-4435
OA Status:Green
Official URL:http://jmlr.csail.mit.edu/papers/v13/brodersen12a.html

Download

Download PDF  'Bayesian mixed-effects inference on classification performance in hierarchical datasets'.
Preview
Content: Published Version
Filetype: PDF
Size: 1MB