Header

UZH-Logo

Maintenance Infos

From active towards itive learning: using consideration information to improve labeling correctness


Bernstein, A; Li, J (2009). From active towards itive learning: using consideration information to improve labeling correctness. In: Human Computation Workshop, Paris, France, June 2009 - June 2009.

Abstract

Data mining techniques have become central to many applications. Most of those applications rely on so called supervised learning algorithms, which learn from given examples in the form of data with predefined labels (e.g., classes such as spam, not spam). Labeling, however, is oftentimes expensive, as it typically requires manual work by human experts. Active learning systems reduce the human effort by choosing the most informative instances for labeling. Unfortunately, research in psychology has shown conclusively that human decisions are inaccurate, easily biased by circumstances, and far from the oracle decision making assumed in active learning research. Based on these findings we show experimentally that (human) mistakes in labeling can significantly deteriorate the performance of active learning systems. To solve this problem, we introduce consideration information – a concept from marketing – into an active learning system to bias and improve the human’s labeling performance. Results (with simulated and human labelers) show that consideration information can indeed be used to exert a bias. Furthermore, we find that the choice of appropriate consideration information can be used to positively bias an expert and thereby improving the overall performance of the learning setting.

Abstract

Data mining techniques have become central to many applications. Most of those applications rely on so called supervised learning algorithms, which learn from given examples in the form of data with predefined labels (e.g., classes such as spam, not spam). Labeling, however, is oftentimes expensive, as it typically requires manual work by human experts. Active learning systems reduce the human effort by choosing the most informative instances for labeling. Unfortunately, research in psychology has shown conclusively that human decisions are inaccurate, easily biased by circumstances, and far from the oracle decision making assumed in active learning research. Based on these findings we show experimentally that (human) mistakes in labeling can significantly deteriorate the performance of active learning systems. To solve this problem, we introduce consideration information – a concept from marketing – into an active learning system to bias and improve the human’s labeling performance. Results (with simulated and human labelers) show that consideration information can indeed be used to exert a bias. Furthermore, we find that the choice of appropriate consideration information can be used to positively bias an expert and thereby improving the overall performance of the learning setting.

Statistics

Downloads

94 downloads since deposited on 04 Feb 2010
28 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Language:English
Event End Date:June 2009
Deposited On:04 Feb 2010 11:50
Last Modified:05 Apr 2016 13:39

Download

Preview Icon on Download
Preview
Filetype: PDF
Size: 2MB