Header

UZH-Logo

Maintenance Infos

The Word Sense Disambiguation Test Suite at WMT18


Rios, Annette; Müller, Mathias; Sennrich, Rico (2018). The Word Sense Disambiguation Test Suite at WMT18. In: Third Conference on Machine Translation (WMT), Brussels, 31 October 2018 - 1 November 2018, 594-602.

Abstract

We present a task to measure an MT system's capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German--English Word Sense Disambiguation (WSD) test set ContraWSD, but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German-English submissions to the WMT'18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81% - 93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.

Abstract

We present a task to measure an MT system's capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German--English Word Sense Disambiguation (WSD) test set ContraWSD, but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German-English submissions to the WMT'18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81% - 93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

0 downloads since deposited on 02 Oct 2018
0 downloads since 12 months

Additional indexing

Item Type:Conference or Workshop Item (Other), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:1 November 2018
Deposited On:02 Oct 2018 13:41
Last Modified:05 Dec 2018 14:49
Publisher:Association for Computational Linguistics
Number:2
OA Status:Green
Publisher DOI:https://doi.org/10.18653/v1/W18-64064
Official URL:http://www.statmt.org/wmt18/pdf/WMT064.pdf
Related URLs:http://www.statmt.org/wmt18/ (Organisation)
Project Information:
  • : FunderSNSF
  • : Grant ID105212_169888
  • : Project TitleRich Context in Neural Machine Translation

Download

Download PDF  'The Word Sense Disambiguation Test Suite at WMT18'.
Preview
Content: Published Version
Filetype: PDF
Size: 144kB
View at publisher