Navigation auf zora.uzh.ch

Search ZORA

ZORA (Zurich Open Repository and Archive)

Maintenance: On Wednesday, February 12th 2025, maintenance work will take place on the MariaDB servers from 9:45pm to approx. 10:45pm. During this period, ZORA and JDB will be unavailable. Thank you for your understanding.

On the Limits of Minimal Pairs in Contrastive Evaluation

Vamvas, Jannis; Sennrich, Rico (2021). On the Limits of Minimal Pairs in Contrastive Evaluation. In: Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Online and in Punta Cana, Dominican Republic, 11 November 2021, ACL Anthology.

Abstract

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English–German MT that implements this recommendation.

Additional indexing

Item Type:Conference or Workshop Item (Lecture), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:11 November 2021
Deposited On:17 Sep 2021 05:34
Last Modified:27 Apr 2022 07:35
Publisher:ACL Anthology
OA Status:Green
Official URL:https://aclanthology.org/2021.blackboxnlp-1.5/
Related URLs:https://github.com/ZurichNLP/distil-lingeval (Research Data)
Project Information:
  • Funder: SNSF
  • Grant ID: PP00P1_176727
  • Project Title: Multi-Task Learning with Multilingual Resources for Better Natural Language Understanding
Download PDF  'On the Limits of Minimal Pairs in Contrastive Evaluation'.
Preview
  • Content: Accepted Version
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)

Metadata Export

Statistics

Citations

Downloads

47 downloads since deposited on 17 Sep 2021
13 downloads since 12 months
Detailed statistics

Authors, Affiliations, Collaborations

Similar Publications