Header

UZH-Logo

Maintenance Infos

Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation


Zucchet, Nicolas; Sacramento, João (2022). Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation. Neural Computation, 34(12):2309-2346.

Abstract

This review examines gradient-based techniques to solve bilevel optimization problems. Bilevel optimization extends the loss minimization framework underlying statistical learning to systems that are implicitly defined through a quantity they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers, and even physical systems and allows for greater modeling flexibility compared to the usual explicit definition of such systems. We focus on solving learning problems of this kind through gradient descent, leveraging the toolbox of implicit differentiation and, for the first time applied to this setting, the equilibrium propagation theorem. We present the mathematical foundations behind such methods, introduce the gradient estimation algorithms in detail, and compare the competitive advantages of the different approaches.

Abstract

This review examines gradient-based techniques to solve bilevel optimization problems. Bilevel optimization extends the loss minimization framework underlying statistical learning to systems that are implicitly defined through a quantity they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers, and even physical systems and allows for greater modeling flexibility compared to the usual explicit definition of such systems. We focus on solving learning problems of this kind through gradient descent, leveraging the toolbox of implicit differentiation and, for the first time applied to this setting, the equilibrium propagation theorem. We present the mathematical foundations behind such methods, introduce the gradient estimation algorithms in detail, and compare the competitive advantages of the different approaches.

Statistics

Citations

Dimensions.ai Metrics
2 citations in Web of Science®
3 citations in Scopus®
Google Scholar™

Altmetrics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:07 Faculty of Science > Institute of Neuroinformatics
Dewey Decimal Classification:570 Life sciences; biology
Scopus Subject Areas:Social Sciences & Humanities > Arts and Humanities (miscellaneous)
Life Sciences > Cognitive Neuroscience
Uncontrolled Keywords:Cognitive Neuroscience, Arts and Humanities (miscellaneous)
Language:English
Date:8 November 2022
Deposited On:24 Feb 2023 13:15
Last Modified:28 Apr 2024 01:50
Publisher:MIT Press
ISSN:0899-7667
OA Status:Closed
Publisher DOI:https://doi.org/10.1162/neco_a_01547
PubMed ID:36283053
Full text not available from this repository.