Header

UZH-Logo

Maintenance Infos

Face Restoration via Plug-and-Play 3D Facial Priors


Hu, Xiaobin; Ren, Wenqi; Yang, Jiaolong; Cao, Xiaochun; Wipf, David P; Menze, Bjoern; Tong, Xin; Zha, Hongbin (2022). Face Restoration via Plug-and-Play 3D Facial Priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):8910-8926.

Abstract

State-of-the-art face restoration methods employ deep convolutional neural networks (CNNs) to learn a mapping between degraded and sharp facial patterns by exploring local appearance knowledge. However, most of these methods do not well exploit facial structures and identity information, and only deal with task-specific face restoration (e.g.,face super-resolution or deblurring). In this paper, we propose cross-tasks and cross-models plug-and-play 3D facial priors to explicitly embed the network with the sharp facial structures for general face restoration tasks. Our 3D priors are the first to explore 3D morphable knowledge based on the fusion of parametric descriptions of face attributes (e.g., identity, facial expression, texture, illumination, and face pose). Furthermore, the priors can easily be incorporated into any network and are very efficient in improving the performance and accelerating the convergence speed. Firstly, a 3D face rendering branch is set up to obtain 3D priors of salient facial structures and identity knowledge. Secondly, for better exploiting this hierarchical information (i.e., intensity similarity, 3D facial structure, and identity content), a spatial attention module is designed for image restoration problems. Extensive face restoration experiments including face super-resolution and deblurring demonstrate that the proposed 3D priors achieve superior face restoration results over the state-of-the-art algorithms

Abstract

State-of-the-art face restoration methods employ deep convolutional neural networks (CNNs) to learn a mapping between degraded and sharp facial patterns by exploring local appearance knowledge. However, most of these methods do not well exploit facial structures and identity information, and only deal with task-specific face restoration (e.g.,face super-resolution or deblurring). In this paper, we propose cross-tasks and cross-models plug-and-play 3D facial priors to explicitly embed the network with the sharp facial structures for general face restoration tasks. Our 3D priors are the first to explore 3D morphable knowledge based on the fusion of parametric descriptions of face attributes (e.g., identity, facial expression, texture, illumination, and face pose). Furthermore, the priors can easily be incorporated into any network and are very efficient in improving the performance and accelerating the convergence speed. Firstly, a 3D face rendering branch is set up to obtain 3D priors of salient facial structures and identity knowledge. Secondly, for better exploiting this hierarchical information (i.e., intensity similarity, 3D facial structure, and identity content), a spatial attention module is designed for image restoration problems. Extensive face restoration experiments including face super-resolution and deblurring demonstrate that the proposed 3D priors achieve superior face restoration results over the state-of-the-art algorithms

Statistics

Citations

Dimensions.ai Metrics
7 citations in Web of Science®
7 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

340 downloads since deposited on 02 Feb 2022
124 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:07 Faculty of Science > Department of Quantitative Biomedicine
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Physical Sciences > Software
Physical Sciences > Computer Vision and Pattern Recognition
Physical Sciences > Computational Theory and Mathematics
Physical Sciences > Artificial Intelligence
Physical Sciences > Applied Mathematics
Uncontrolled Keywords:Applied Mathematics, Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Software
Language:English
Date:1 December 2022
Deposited On:02 Feb 2022 16:47
Last Modified:26 Apr 2024 01:40
Publisher:Institute of Electrical and Electronics Engineers
ISSN:0098-5589
Additional Information:© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
OA Status:Green
Publisher DOI:https://doi.org/10.1109/tpami.2021.3123085
  • Content: Accepted Version