Unidirectional and Bidirectional LSTM - Avesta-Ahmadi/Univariate-Time-Series-Forecasting GitHub Wiki

LSTM (Long-Short Term Memory) cells are designed to memorize and capture long-term dependencies of data. Hence, they can be seen as perfect tools for time series forecasting as they can take dependency of each time step on its previous ones. The structure and more detailed information on LSTM networks can be found at https://colah.github.io/posts/2015-08-Understanding-LSTMs/. Following graph represents the schematic of a LSTM cell taken from https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714.

The memory and output from current block will be passed to the next block in the same layer. Likewise, the output from this layer will be passed to the next hidden layer.

In this approach unidirectional LSTM layer or a unidirectional stacked LSTM network (where few layers of LSTM will stack on top of each other) have been used. Input and output data will be generated same as MLP model. Hyper parameters will be adjusted for each particular time series data.

As another approach, Bidirectional LSTMs have been utilized for sequence modelling. In this algorithm, each layer of LSTM has a forward pass and a backward pass which are completely independent of each other. The combination of these two passes will be passed to the next layer. This combination can be addition, multiplication or averaging. Bidirectional layers can be stacked on top of each other to from a stacked bidirectional LSTM network. The following graph demonstrates the structure of bidirectional layer.

The graph is taken from: Cui, Zhiyong, Ruimin Ke, and Yinhai Wang. "Deep bidirectional and unidirectional LSTM recurrent neural network for network-wide traffic speed prediction." arXiv preprint arXiv:1801.02143 (2018).

Also, for more information on bidirectional layers, following references can be helpful:

Schuster, Mike, and Kuldip K. Paliwal. "Bidirectional recurrent neural networks." Signal Processing, IEEE Transactions on 45.11 (1997): 2673-2681.2. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan

Kiperwasser, Eliyahu; Goldberg, Yoav (2016). "Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations". Transactions of the Association for Computational Linguistics. 4: 313–327. doi:10.1162/tacl_a_00101