Long Short-Term Memory (LSTM)_“long short-term memory,” neural comput

                     Long Short-Term Memory (LSTM)

Outline                 

                 Background

                 LSTM Network

                 Extended LSTM

                 LSTM vs. RNN

                 Examples

                 References

Background

Definition

 LSTM Network

For detailed derivation: http://cs224d.stanford.edu/

Extended LSTM---GRU

LSTM vs. RNN

References

1.A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Textbook, Studies in Computational Intelligence, Springer, 2012

2.Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780

3.Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural computation, 12(10), 2451-2471

4.Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012

5.Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2012