Header

UZH-Logo

Maintenance Infos

On the Limits of Minimal Pairs in Contrastive Evaluation


Vamvas, Jannis; Sennrich, Rico (2021). On the Limits of Minimal Pairs in Contrastive Evaluation. In: Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Online and in Punta Cana, Dominican Republic, 11 November 2021, s.n..

Abstract

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English–German MT that implements this recommendation.

Abstract

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English–German MT that implements this recommendation.

Statistics

Downloads

5 downloads since deposited on 17 Sep 2021
5 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Lecture), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:11 November 2021
Deposited On:17 Sep 2021 05:34
Last Modified:17 Sep 2021 05:34
Publisher:s.n.
OA Status:Green
Project Information:
  • : FunderSNSF
  • : Grant IDPP00P1_176727
  • : Project TitleMulti-Task Learning with Multilingual Resources for Better Natural Language Understanding

Download

Green Open Access

Download PDF  'On the Limits of Minimal Pairs in Contrastive Evaluation'.
Preview
Content: Accepted Version
Filetype: PDF
Size: 239kB
Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)