Header

UZH-Logo

Maintenance Infos

DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator


Gao, Chang; Neil, Daniel; Ceolini, Enea; Liu, Shih-Chii; Delbruck, Tobi (2018). DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator. In: 26th ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA' 18), Monterey, 25 February 2018 - 27 February 2018, 21-30.

Abstract

Recurrent Neural Networks (RNNs) are widely used in speech recognition and natural language processing applications because of their capability to process temporal sequences. Because RNNs are fully connected, they require a large number of weight memory accesses, leading to high power consumption. Recent theory has shown that an RNN delta network update approach can reduce memory access and computes with negligible accuracy loss. This paper describes the implementation of this theoretical approach in a hardware accelerator called "DeltaRNN" (DRNN). The DRNN updates the output of a neuron only when the neuron»s activation changes by more than a delta threshold. It was implemented on a Xilinx Zynq-7100 FPGA. FPGA measurement results from a single-layer RNN of 256 Gated Recurrent Unit (GRU) neurons show that the DRNN achieves 1.2 TOp/s effective throughput and 164 GOp/s/W power efficiency. The delta update leads to a 5.7x speedup compared to a conventional RNN update because of the sparsity created by the DN algorithm and the zero-skipping ability of DRNN.

Abstract

Recurrent Neural Networks (RNNs) are widely used in speech recognition and natural language processing applications because of their capability to process temporal sequences. Because RNNs are fully connected, they require a large number of weight memory accesses, leading to high power consumption. Recent theory has shown that an RNN delta network update approach can reduce memory access and computes with negligible accuracy loss. This paper describes the implementation of this theoretical approach in a hardware accelerator called "DeltaRNN" (DRNN). The DRNN updates the output of a neuron only when the neuron»s activation changes by more than a delta threshold. It was implemented on a Xilinx Zynq-7100 FPGA. FPGA measurement results from a single-layer RNN of 256 Gated Recurrent Unit (GRU) neurons show that the DRNN achieves 1.2 TOp/s effective throughput and 164 GOp/s/W power efficiency. The delta update leads to a 5.7x speedup compared to a conventional RNN update because of the sparsity created by the DN algorithm and the zero-skipping ability of DRNN.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

33 downloads since deposited on 12 Mar 2019
32 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Speech), not_refereed, original work
Communities & Collections:07 Faculty of Science > Institute of Neuroinformatics
Dewey Decimal Classification:570 Life sciences; biology
Language:English
Event End Date:27 February 2018
Deposited On:12 Mar 2019 13:28
Last Modified:30 Oct 2019 08:11
Publisher:ACM Digital Library
Number of Pages:10
OA Status:Hybrid
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1145/3174243.3174261
Official URL:https://dl.acm.org/citation.cfm?id=3174261

Download

Hybrid Open Access

Download PDF  'DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator'.
Preview
Content: Published Version
Filetype: PDF
Size: 5MB
View at publisher