Navigation auf zora.uzh.ch

Search ZORA

ZORA (Zurich Open Repository and Archive)

BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation

Jiang, Yuchen Eleanor; Liu, Tianyu; Ma, Shuming; Zhang, Dongdong; Yang, Jian; Huang, Haoyang; Sennrich, Rico; Cotterell, Ryan; Sachan, Mrinmaya; Zhou, Ming (2022). BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, USA, 10 July 2022 - 15 July 2022. Association for Computational Linguistics, 1550-1565.

Abstract

Standard automatic metrics, e.g. BLEU, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric BlonDe to widen the scope of automatic MT evaluation from sentence to document level. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed dataset BWB. The experimental results show that BlonDe possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BlonDe also achieves significantly higher Pearson’s r correlation with human judgments compared to previous metrics.

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Scopus Subject Areas:Physical Sciences > Computer Networks and Communications
Physical Sciences > Hardware and Architecture
Physical Sciences > Information Systems
Physical Sciences > Software
Language:English
Event End Date:15 July 2022
Deposited On:12 Dec 2022 15:18
Last Modified:16 Dec 2022 18:27
Publisher:Association for Computational Linguistics
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Official URL:https://aclanthology.org/2022.naacl-main.111
Download PDF  'BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation'.
Preview
  • Content: Published Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)

Metadata Export

Statistics

Citations

9 citations in Web of Science®
31 citations in Scopus®
Google Scholar™

Downloads

22 downloads since deposited on 12 Dec 2022
8 downloads since 12 months
Detailed statistics

Authors, Affiliations, Collaborations

Similar Publications