DUAL TRANSFORMER ENCODERS FOR SESSION-BASED RECOMMENDATION

Authors

DOI:

https://doi.org/10.15625/1813-9663/37/4/15844

Keywords:

Recommender systems, Session-based recommendation, Self-attention, Dual Transformer

Abstract

When long-term user proles are not available, session-based recommendation methods are used to predict the user's next actions from anonymous sessions-based data. Recent advances in session-based recommendation highlight the necessity of modeling not only user sequential behaviors but also the user's main interest in a session, while avoiding the eect of unintended clicks causing interest drift of the user. In this work, we propose a Dual Transformer Encoder Recommendation model (DTER) as a solution to address this requirement. The idea is to combine the following recipes: (1) a Transformer-based model with dual encoders capable of modeling both sequential patterns and the main interest of the user in a session; (2) a new recommendation model that is designed for learning richer session contexts by conditioning on all permutations of the session prex. This approach provides a unied framework for leveraging the ability of the Transformer's self-attention mechanism in modeling session sequences while taking into account the user's main interest in the session. We empirically evaluate the proposed method on two benchmark datasets. The results show that DTER outperforms state-of-the-art session-based recommendation methods on common evaluation metrics.

Author Biographies

Pham Hoang Anh, VinAI

VinAI

Ngo Xuan Bach, Posts & Telecommunications Institute of Technology

Department of Computer Science

Tu Minh Phuong, Posts & Telecommunications Institute of Technology

Department of Computer Science

References

C. Alt, M. Hubner, and L. Hennig, Fine-tuning pre-trained transformer language models to distantly supervised relation extraction," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association

for Computational Linguistics, Jul. 2019, pp. 1388{1398. [Online]. Available: https://www.aclweb.org/anthology/P19-1134

Z. Batmaz, A. Yurekli, A. Bilge, and C. Kaleli, A review on deep learning for recommender systems: challenges and remedies," Articial Intelligence Review, vol. 52, pp. 1-37, 2019.

S. Chen, J. L. Moore, D. Turnbull, and T. Joachims, Playlist prediction via metric embedding," in Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD '12. New York, NY, USA: ACM, 2012, pp. 714-722.

[Online]. Available: http://doi.acm.org/10.1145/2339530.2339643

K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, Oct. 2014,

pp. 1724-1734. [Online]. Available: https://www.aclweb.org/anthology/D14-1179

K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, Electra: Pre-training text encoders as discriminators rather than generators," in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=r1xMH1BtvB

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov., Transformer-XL:

attentive language models beyond a xed-length context," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 2978-2988.

M. Deshpande and G. Karypis, Item-based top-n recommendation algorithms," ACM Trans. Inf. Syst., vol. 22, no. 1, pp. 143{177, Jan. 2004. [Online]. Available: http://doi.acm.org/10.1145/963770.963776

J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding," in Proceedings of NAACL-HLT, 2019, pp. 4171-4186.

J. L. Elman, Finding structure in time," Cognitive Science, vol. 14, no. 2, pp. 179-211, 1990.

F. Figueiredo, B. Ribeiro, J. M. Almeida, and C. Faloutsos, Tribe

ow: Mining & predicting user trajectories," in Proceedings of the 25th International Conference on World Wide Web, ser. WWW '16. Republic and Canton of Geneva, Switzerland: International

World Wide Web Conferences Steering Committee, 2016, pp. 695-706. [Online]. Available: https://doi.org/10.1145/2872427.2883059

D. Goldberg, D. Nichols, B. M. Oki, and D. Terry, Using collaborative ltering to weave an information tapestry," Commun. ACM, vol. 35, no. 12, pp. 61-70, Dec. 1992. [Online]. Available: http://doi.acm.org/10.1145/138859.138867

B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, Session-based recommendations with recurrent neural networks," in Proceedings of the International Conference on Learning Representations, 2016.

B. Hidasi and A. Karatzoglou, Recurrent neural networks with top-k gains for session-based recommendations," in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ser. CIKM '18. New York, NY, USA: ACM, 2018, pp. 843-852.

[Online]. Available: http://doi.acm.org/10.1145/3269206.3271761

W.-C. Kang and J. McAuley, Self-attentive sequential recommendation," in Proceedings of IEEE International Conference on Data Mining (ICDM'18), 2018, pp. 197{206.

Y. Koren, R. Bell, and C. Volinsky, Matrix factorization techniques for recommender systems," Computer, vol. 42, no. 8, pp. 30{37, Aug. 2009. [Online]. Available: http://dx.doi.org/10.1109/MC.2009.263

Dual Transformer Encoders for Session-based Recommendation 15

Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, Handwritten Digit Recognition with a Back-Propagation Network," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. Morgan-Kaufmann, 1990, pp. 396-404.

J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, and J. Ma, Neural attentive session-based recommendation," in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, ser. CIKM '17. New York, NY, USA: ACM, 2017, pp. 1419-1428.

[Online]. Available: http://doi.acm.org/10.1145/3132847.3132926

Q. Liu, Y. Zeng, R. Mokhosi, and H. Zhang, STAMP: short-term attention/memory priority model for session-based recommendation," in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD

'18. New York, NY, USA: ACM, 2018, pp. 1831{1839. [Online]. Available: http://doi.acm.org/10.1145/3219819.3219950

T. M. Phuong, T. C. Thanh, and N. X. Bach, Neural session-aware recommendation," IEEE Access, vol. 7, pp. 86 884{86 896, 2019.

M. Quadrana, A. Karatzoglou, B. Hidasi, and P. Cremonesi, Personalizing session-based recommendations with hierarchical recurrent neural networks," in Proceedings of the Eleventh

ACM Conference on Recommender Systems, ser. RecSys '17. New York, NY, USA: ACM, 2017, pp. 130{137. [Online]. Available: http://doi.acm.org/10.1145/3109859.3109896

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training," URL https://s3-us-west-2. amazonaws.com/openaiassets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018.

S. Rendle, C. Freudenthaler, and L. Schmidt-Thieme, Factorizing personalized markov chains for next-basket recommendation," in Proceedings of the 19th International Conference on World Wide Web, ser. WWW '10. New York, NY, USA: ACM, 2010, pp. 811{820. [Online]. Available: http://doi.acm.org/10.1145/1772690.1772773

G. Shani, R. I. Brafman, and D. Heckerman, "An MDP-based Recommender System"," arXiv e-prints, p. arXiv:1301.0600, "Dec" "2012".

Y. K. Tan, X. Xu, and Y. Liu, Improved recurrent neural networks for session-based recommendations," in Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, ser. DLRS 2016. New York, NY, USA: ACM, 2016, pp. 17-22. [Online]. Available:

http://doi.acm.org/10.1145/2988450.2988452

J. Tang and K. Wang, Personalized top-n sequential recommendation via convolutional sequence embedding," in Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, ser. WSDM '18. New York, NY, USA: ACM, 2018, pp. 565-573.

[Online]. Available: http://doi.acm.org/10.1145/3159652.3159656

T. X. Tuan and T. M. Phuong, 3D convolutional networks for session-based recommendation with content features," in Proceedings of the Eleventh ACM Conference on Recommender Systems, ser. RecSys '17. New York, NY, USA: ACM, 2017, pp. 138-146. [Online]. Available: http://doi.acm.org/10.1145/3109859.3109900

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, "Attention is all you need," in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 60006010.

Pham Hoang Anh, Ngo Xuan Bach, Tu Minh Phuong

E. Voita, R. Sennrich, and I. Titov, "The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 4396-4406.

S. Wang, L. Cao, and Y. Wang, A survey on session-based recommender systems," arXiv:1902.04864, 2019. [Online]. Available: https://arxiv.org/abs/1902.04864

S. Wang, L. Hu, L. Cao, X. Huang, D. Lian, and W. Liu, Attention-based transactional context embedding for next-item recommendation," in Proceedings of AAAI Conference on Articial

Intelligence, 2018.

S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, "Session-based recommendation with

graph neural networks," in Proceedings of AAAI Conference on Articial Intelligence, 2019.

J. Xiao, H. Ye, X. He, H. Zhang, F. Wu, and T.-S. Chua, Attentional factorization machines:

Learning the weight of feature interactions via attention networks," in Proceedings of the 26th

International Joint Conference on Articial Intelligence, ser. IJCAI17. AAAI Press, 2017, p.

Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, XLNet: generalized

autoregressive pretraining for language understanding," arXiv:1906.08237, 2019. [Online].

Available: https://arxiv.org/abs/1906.08237

G. Yap, X. Li, and P. Yu, Eective next-items recommendation via personalized sequential

pattern mining," in Proceedings of the 17th International Conference on Database Systems for

Advanced Applications, 2012, pp. 48{64.

F. Yuan, A. Karatzoglou, I. Arapakis, J. M. Jose, and X. He, A simple convolutional generative

network for next item recommendation," in Proceedings of the Twelfth ACM International

Conference on Web Search and Data Mining, ser. WSDM '19. New York, NY, USA: ACM,

, pp. 582{590.

Downloads

Published

2021-10-12

Issue

Section

SPECIAL ISSUE DEDICATED TO THE MEMORY OF PROFESSOR PHAN DINH DIEU - PART B