Navigation auf zora.uzh.ch

Search ZORA

ZORA (Zurich Open Repository and Archive)

Distributionally Robust Recurrent Decoders with Random Network Distillation

Miceli Barone, Antonio Valerio; Birch, Alexandra; Sennrich, Rico (2022). Distributionally Robust Recurrent Decoders with Random Network Distillation. In: Proceedings of the 7th Workshop on Representation Learning for NLP, Dublin, Ireland, 26 May 2022, Association for Computational Linguistics.

Abstract

Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to “shortcut learning”":" relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Computational Linguistics
Dewey Decimal Classification:000 Computer science, knowledge & systems
410 Linguistics
Language:English
Event End Date:26 May 2022
Deposited On:12 Dec 2022 15:15
Last Modified:18 Dec 2022 09:56
Publisher:Association for Computational Linguistics
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Official URL:https://aclanthology.org/2022.repl4nlp-1.1
Download PDF  'Distributionally Robust Recurrent Decoders with Random Network Distillation'.
Preview
  • Content: Published Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)

Metadata Export

Statistics

Downloads

13 downloads since deposited on 12 Dec 2022
7 downloads since 12 months
Detailed statistics

Authors, Affiliations, Collaborations

Similar Publications