Header

UZH-Logo

Maintenance Infos

Domain robustness in neural machine translation


Müller, Mathias; Rios, Annette; Sennrich, Rico (2020). Domain robustness in neural machine translation. In: 14th Conference of the Association for Machine Translation in the Americas (AMTA 2020), Virtual, 6 October 2020 - 9 October 2020, 151-164.

Abstract

Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustnes - the generalization of models to unseen test domains - is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the performance of SMT and NMT models on out-of-domain test sets. We find that in unknown domains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are reported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German→English OPUS data, and German→Romansh (a low-resource setting) we find that several methods improve domain robustness. While thosemethods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT

Abstract

Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustnes - the generalization of models to unseen test domains - is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the performance of SMT and NMT models on out-of-domain test sets. We find that in unknown domains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are reported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German→English OPUS data, and German→Romansh (a low-resource setting) we find that several methods improve domain robustness. While thosemethods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT

Statistics

Downloads

10 downloads since deposited on 22 Oct 2020
10 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Speech), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:9 October 2020
Deposited On:22 Oct 2020 17:35
Last Modified:27 Nov 2020 07:34
Publisher:Association for Machine Translation in the Americas
Series Name:Proceedings of the 14th Conference of the Association for Machine Translation in the Americas
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Official URL:https://www.aclweb.org/anthology/2020.amta-research.14

Download

Green Open Access

Download PDF  'Domain robustness in neural machine translation'.
Preview
Content: Published Version
Filetype: PDF
Size: 726kB