Header

UZH-Logo

Maintenance Infos

Encoders Help You Disambiguate Word Senses in Neural Machine Translation


Tang, Gongbo; Sennrich, Rico; Nivre, Joakim (2019). Encoders Help You Disambiguate Word Senses in Neural Machine Translation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, 3 November 2019 - 7 November 2019, 1429-1435.

Abstract

Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. In contrast to encoders, the effect of decoder is different in models with different architectures. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context.

Abstract

Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. In contrast to encoders, the effect of decoder is different in models with different architectures. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

7 downloads since deposited on 05 Nov 2019
7 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:7 November 2019
Deposited On:05 Nov 2019 14:36
Last Modified:05 Nov 2019 14:37
Publisher:Association for Computational Linguistics
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.18653/v1/D19-1149
Official URL:https://www.aclweb.org/anthology/D19-1149.pdf
Related URLs:https://www.aclweb.org/anthology/D19-1149

Download

Green Open Access

Download PDF  'Encoders Help You Disambiguate Word Senses in Neural Machine Translation'.
Preview
Content: Published Version
Language: English
Filetype: PDF
Size: 334kB
View at publisher
Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)