Do RNN and LSTM have Long Memory?

Part of Proceedings of the International Conference on Machine Learning 1 pre-proceedings (ICML 2020)

Bibtex »Metadata »Paper »Supplemental »

Bibtek download is not availble in the pre-proceeding


Authors

Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, Guangjian Tian

Abstract

<p>The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, we raise the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a time series perspective. Since the term "long memory" is still not well-defined for a network, we propose a new definition for long memory network. To verify our theory, we make minimal modifications to RNN and LSTM and convert them to long memory networks, and illustrate their superiority in modeling long-term dependence of various datasets.</p>