Header

UZH-Logo

Maintenance Infos

FrameFire: Enabling Efficient Spiking Neural Network Inference for Video Segmentation


Chen, Qinyu; Sun, Congyi; Gao, Chang; Fang, Xinyuan; Luan, Haitao (2023). FrameFire: Enabling Efficient Spiking Neural Network Inference for Video Segmentation. In: 2023 IEEE 5th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Hangzhou, China, 11 June 2023 - 13 June 2023. Institute of Electrical and Electronics Engineers, 1-5.

Abstract

Fast video recognition is essential for real-time scenarios, e.g., autonomous driving. However, applying existing Deep Neural Networks (DNNs) to individual high-resolution images is expensive due to large model sizes. Spiking Neural Networks (SNNs) are developed as a promising alternative to DNNs due to their more realistic brain-inspired computing models. SNNs have sparse neuron firing over time, i.e., spatio-temporal sparsity; thus they are useful to enable energy-efficient computation. However, exploiting the spatio-temporal sparsity of SNNs in hardware leads to unpredictable and unbalanced workloads, degrading energy efficiency. In this work, we, therefore, propose an SNN accelerator called FrameFire for efficient video processing. We introduce a Keyframe-dominated Workload Balance Schedule (KWBS) method. It accelerates the image recognition network with sparse keyframes, then records and analyzes the current workload distribution on hardware to facilitate scheduling workloads in subsequent frames. FrameFire is implemented on a Xilinx XC7Z035 FPGA and verified by video segmentation tasks. The results show that the throughput is improved by 1.7× with the KWBS method. FrameFire achieved 1.04 KFPS throughput and 1.15 mJ/frame recognition energy.

Abstract

Fast video recognition is essential for real-time scenarios, e.g., autonomous driving. However, applying existing Deep Neural Networks (DNNs) to individual high-resolution images is expensive due to large model sizes. Spiking Neural Networks (SNNs) are developed as a promising alternative to DNNs due to their more realistic brain-inspired computing models. SNNs have sparse neuron firing over time, i.e., spatio-temporal sparsity; thus they are useful to enable energy-efficient computation. However, exploiting the spatio-temporal sparsity of SNNs in hardware leads to unpredictable and unbalanced workloads, degrading energy efficiency. In this work, we, therefore, propose an SNN accelerator called FrameFire for efficient video processing. We introduce a Keyframe-dominated Workload Balance Schedule (KWBS) method. It accelerates the image recognition network with sparse keyframes, then records and analyzes the current workload distribution on hardware to facilitate scheduling workloads in subsequent frames. FrameFire is implemented on a Xilinx XC7Z035 FPGA and verified by video segmentation tasks. The results show that the throughput is improved by 1.7× with the KWBS method. FrameFire achieved 1.04 KFPS throughput and 1.15 mJ/frame recognition energy.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

6 downloads since deposited on 31 Jan 2024
6 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:07 Faculty of Science > Institute of Neuroinformatics
Dewey Decimal Classification:570 Life sciences; biology
Scopus Subject Areas:Physical Sciences > Artificial Intelligence
Physical Sciences > Computer Vision and Pattern Recognition
Physical Sciences > Hardware and Architecture
Physical Sciences > Information Systems
Physical Sciences > Electrical and Electronic Engineering
Language:English
Event End Date:13 June 2023
Deposited On:31 Jan 2024 10:14
Last Modified:01 Feb 2024 22:29
Publisher:Institute of Electrical and Electronics Engineers
Series Name:IEEE International Conference on Artificial Intelligence Circuits and Systems
ISSN:2834-9830
ISBN:9798350332674
OA Status:Green
Publisher DOI:https://doi.org/10.1109/aicas57966.2023.10168660
Project Information:
  • : FunderUniversity of Shanghai for Science and Technology
  • : Grant ID
  • : Project Title
  • : FunderNanjing University
  • : Grant ID
  • : Project Title
  • : FunderNational Key Research and Development Program of China
  • : Grant ID
  • : Project Title
  • Content: Submitted Version
  • Language: English