Header

UZH-Logo

Maintenance Infos

Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy


Gomariz, Alvaro; Portenier, Tiziano; Helbling, Patrick M; Isringhausen, Stephan; Suessbier, Ute; Nombela-Arrieta, César; Goksel, Orcun (2021). Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy. Nature Machine Intelligence, 3(9):799-811.

Abstract

Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers, and therefore applicable to a very restricted number of experimental settings. We herein propose Marker Sampling and Excite - a neural network approach with a modality sampling strategy and a novel attention module that together enable (i) flexible training with heterogeneous datasets with combinations of markers and (ii) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario where an ensemble of many networks is naïvely trained for each possible marker combination separately. In addition, we demonstrate the feasibility of this framework in high-throughput biological analysis by revising a recent quantitative characterization of bone marrow vasculature in 3D confocal microscopy datasets and further confirm the validity of our approach on an additional, significantly different dataset of microvessels in fetal liver tissues. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.

Abstract

Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers, and therefore applicable to a very restricted number of experimental settings. We herein propose Marker Sampling and Excite - a neural network approach with a modality sampling strategy and a novel attention module that together enable (i) flexible training with heterogeneous datasets with combinations of markers and (ii) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario where an ensemble of many networks is naïvely trained for each possible marker combination separately. In addition, we demonstrate the feasibility of this framework in high-throughput biological analysis by revising a recent quantitative characterization of bone marrow vasculature in 3D confocal microscopy datasets and further confirm the validity of our approach on an additional, significantly different dataset of microvessels in fetal liver tissues. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.

Statistics

Citations

Dimensions.ai Metrics
8 citations in Web of Science®
8 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

7 downloads since deposited on 14 Oct 2021
0 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > University Hospital Zurich > Clinic for Oncology and Hematology
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Physical Sciences > Software
Physical Sciences > Human-Computer Interaction
Physical Sciences > Computer Vision and Pattern Recognition
Physical Sciences > Computer Networks and Communications
Physical Sciences > Artificial Intelligence
Language:English
Date:September 2021
Deposited On:14 Oct 2021 14:03
Last Modified:18 Mar 2024 04:44
Publisher:Nature Publishing Group
ISSN:2522-5839
OA Status:Green
Free access at:PubMed ID. An embargo period may apply.
Publisher DOI:https://doi.org/10.1038/s42256-021-00379-y
PubMed ID:34541455
  • Content: Accepted Version