Header

UZH-Logo

Maintenance Infos

Meta-Learning via Hypernetworks


Zhao, Dominic; von Oswald, Johannes; Kobayashi, Seijin; Sacramento, João; Grewe, Benjamin F (2020). Meta-Learning via Hypernetworks. In: 4th Workshop on Meta-Learning at NeurIPS 2020 (MetaLearn 2020), Virtual Conference, 11 December 2020 - 11 December 2020.

Abstract

Recent developments in few-shot learning have shown that during fast adaption, gradient-based meta-learners mostly rely on embedding features of powerful pretrained networks. This leads us to research ways to effectively adapt features and utilize the meta-learner's full potential. Here, we demonstrate the effectiveness of hypernetworks in this context. We propose a soft row-sharing hypernetwork architecture and show that training the hypernetwork with a variant of MAML is tightly linked to meta-learning a curvature matrix used to condition gradients during fast adaptation. We achieve similar results as state-of-art model-agnostic methods in the overparametrized case, while outperforming many MAML variants without using different optimization schemes in the compressive regime. Furthermore, we empirically show that hypernetworks do leverage the inner loop optimization for better adaptation, and analyse how they naturally try to learn the shared curvature of constructed tasks on a toy problem when using our proposed training algorithm.

Abstract

Recent developments in few-shot learning have shown that during fast adaption, gradient-based meta-learners mostly rely on embedding features of powerful pretrained networks. This leads us to research ways to effectively adapt features and utilize the meta-learner's full potential. Here, we demonstrate the effectiveness of hypernetworks in this context. We propose a soft row-sharing hypernetwork architecture and show that training the hypernetwork with a variant of MAML is tightly linked to meta-learning a curvature matrix used to condition gradients during fast adaptation. We achieve similar results as state-of-art model-agnostic methods in the overparametrized case, while outperforming many MAML variants without using different optimization schemes in the compressive regime. Furthermore, we empirically show that hypernetworks do leverage the inner loop optimization for better adaptation, and analyse how they naturally try to learn the shared curvature of constructed tasks on a toy problem when using our proposed training algorithm.

Statistics

Downloads

4 downloads since deposited on 15 Feb 2021
4 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:07 Faculty of Science > Institute of Neuroinformatics
Dewey Decimal Classification:570 Life sciences; biology
Language:English
Event End Date:11 December 2020
Deposited On:15 Feb 2021 14:06
Last Modified:15 Feb 2021 20:30
Publisher:IEEE
OA Status:Green

Download

Green Open Access

Download PDF  'Meta-Learning via Hypernetworks'.
Preview
Content: Published Version
Filetype: PDF
Size: 329kB