Efficient biologically-inspired online learning alternatives to BPTT

Abstract

The error backpropagation through time learning algorithm (BPTT) [1] can be considered as the workhorse to train most of today’s recurrent neural networks (RNNs) and has led to their astounding performance across various application domains. However, BPTT suffers from severe limitations: It requires to (1) store all intermediate activations of the RNN, (2) unroll the RNN for the entire input sequence and (3) halt the regular network operation and send gradient information backwards through time. These three shortcomings are especially problematic in tasks, where the entire input sequence may not be known apriori and the network often has to simultaneously make decisions and learn from this input sequence. In contrast to BPTT, biological systems tune their synaptic connections in different ways. For example, findings from neuroscience indicate the existence of learning signals [2], signals carrying information from the external environment to the neural network, and eligibility traces [3], fading signals of recent neural activity, that influence the learning process. We draw inspiration from those neuroscientific findings and derive from first principles an algorithmic framework, called online spatio-temporal learning (OSTL) [4], which addresses the problems of BPTT mentioned above. In particular, OSTL does not require to propagate information backwards in time and thus enables the network to simultaneously learn and process incoming data. Moreover, for shallow networks, OSTL is equivalent to BPTT and it can be applied to a wide variety of network architectures and unit types, including biologically-inspired neurons as well as traditional long short-term memory (LSTM) or gated recurrent units (GRU). Another key principle of biological systems is to compute synaptic updates solely based on locally available information. However, the learning signal in OSTL vertically traverses through the network architecture. This causes the synaptic updates to be computed involving non-local signals, requires symmetric synaptic connections for the forward and the backward path and that higher network layers need to be updated before lower layers (update-locking problem). Therefore, we extend OSTL by propagating the target information as the learning signal directly to the individual layers of the network architecture and formulate a variant called online spatio-temporal learning with target projection (OSTTP) [5]. This algorithm ultimately combines the online learning capabilities of OSTL while leveraging only locally-available information for updating the synaptic connections, avoiding symmetric connections and alleviating all update-locking problems. We extensively test our learning framework on a variety of challenging spatio-temporal tasks and demonstrate competitive performance compared to the BPTT baseline [4,5,6]. We further showcase the benefits of our algorithms by utilizing in-memory computing neuromorphic hardware (NMHW) [7]. In particular, OSTTP allows to construct a system capable of online learning, while fully leveraging the efficiency of the NMHW. Finally, we demonstrate that the OSTL framework is extensible and can serve as the basis to develop novel biologically-inspired online learning algorithm for RNNs. [1] P. J. Werbos, Proc. IEEE, vol. 78, pp. 1550-1560, 1990 [2] M. Sanhueza et al., Mol. Brain, vol. 6, p. 10, 2013 (Eligibility traces) [3] W. Gerstner et al., Front. Neural Circuits, vol. 12, p. 53, 2018 (learning sign) [4] T. Ortner (Bohnstingl) et al., IEEE TNNLS, vol. 34, pp. 8894-8908, 2022 [5] T. Ortner (Bohnstingl) et al., IEEE AICAS, pp. 11-13, 2023 [6] T. Ortner (Bohnstingl) et al., IEEE AICAS, pp. 13-15, 2022 [7] R. Khaddam-Aljameh et al., IEEE VLSI, pp. 13-19, 2021