Bert Adapter and Contrastive Learning for Continual Classification of Aspect Sentiment Task Sequences
Author affiliations
DOI:
https://doi.org/10.15625/2525-2518/17395Keywords:
continual learning, catastrophic forgetting, knowledge transfer, contrastive learning, aspect sentiment classificationAbstract
Task incremental learning, a setting of continual learning, is an approach to exploit the knowledge from previous tasks for currently new tasks. Task incremental learning aims to solve two big challenges of continual learning: catastrophic forgetting and knowledge transfer or sharing between previous tasks and the current task. This paper improves Task incremental learning by (1) transferring the knowledge (not the training data) learned from previous tasks to a new coming task (contrast of multi-task learning); (2) maintaining or even improving the performance of learned models for previous tasks with avoid forgetting; (3) developing a continual learning model based on results from (1) and (2) to apply for aspect sentiment classification. Specifically, we combine two loss functions based on two contrastive learning modules, which are the Contrastive Knowledge Sharing (CKS) module for encouraging knowledge sharing between old and current tasks and the Contrastive Supervised learning (CSC) module for improving the performance of the current task. The experimental results show that our method could help previously learned tasks to get rid of the catastrophic forgetting phenomenon, outperforming previous studies for aspect sentiment classification.
Downloads
References
Parisi G. I., Kemker R., Part J. L., Kanan C., and Wermter S. - Continual lifelong learning with neural networks: A review, Neural Networks 113 (2019) 54-71. https://doi.org/10.1016/j.neunet.2019.01.012
Van de Ven G. M. and Tolias A. S. - Three scenarios for continual learning, arXiv preprint arXiv:1904.07734 (2019).
Devlin J., Chang M. W., Lee K., and Toutanova K. - BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
Xu H., Liu B., Shu L., and Yu P. S. - BERT post-training for review reading comprehension and aspect-based sentiment analysis, arXiv preprint arXiv:1904.02232 (2019).
Chen Z. and Liu B. - Lifelong machine learning, 2nd Ed., Morgan & Claypool Publishers, San Rafael, 2018.
Ke Z., Liu B., and Huang X. - Continual learning of a mixed sequence of similar and dissimilar tasks, Advances in Neural Information Processing Systems 33 (2020) 18493-18504.
Biesialska M., Biesialska K., and Costa-Jussa M. R. - Continual lifelong learning in natural language processing: A survey, arXiv preprint arXiv:2012.09823 (2020).
Huang Y., Zhang Y., Chen J., Wang X., and Yang D. - Continual learning for text classification with information disentanglement based regularization, arXiv preprint arXiv:2104.05489 (2021).
Ke Z., Xu H., and Liu B. - Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks, arXiv preprint arXiv:2112.03271 (2021).
Houlsby N., Giurgiu A., Jastrzebski S., Morrone B., De Laroussilhe Q., Gesmundo A., Attariyan M., and Gelly S. - Parameter-efficient transfer learning for NLP, Proceedings of the 36th International Conference on Machine Learning (ICML), Vol. 97, Long Beach, 2019, pp. 2790-2799.
Le-Khac P. H., Healy G., and Smeaton A. F. - Contrastive representation learning: A framework and review, IEEE Access 8 (2020) 193907-193934. https://doi.org/10.1109/ACCESS.2020.3031549
Khosla P., Teterwak P., Wang C., Sarna A., Tian Y., Isola P., Maschinot A., Liu C., and Krishnan D. - Supervised contrastive learning, Advances in Neural Information Processing Systems 33 (2020) 18661-18673.
Chen T., Kornblith S., Norouzi M., and Hinton G. - A simple framework for contrastive learning of visual representations, Proceedings of the 37th International Conference on Machine Learning (ICML), Vol. 119, Virtual Event, 2020, pp. 1597-1607.
Hinton G. E., Krizhevsky A., and Wang S. D. - Transforming auto-encoders, in: Honkela T., Duch W., Girolami M., Kaski S. (Eds.), Artificial Neural Networks and Machine Learning--ICANN 2011, Springer, Berlin, Heidelberg, 2011, pp. 44-51.
Sabour S., Frosst N., and Hinton G. E. - Dynamic routing between capsules, Advances in Neural Information Processing Systems 30 (2017) 3856-3866.
Ke Z., Liu B., Xu H., and Shu L. - CLASSIC: Continual and contrastive learning of aspect sentiment classification tasks, arXiv preprint arXiv:2112.02714 (2021).
Jaiswal A., Babu A. R., Zadeh M. Z., Banerjee D., and Makedon F. - A survey on contrastive self-supervised learning, Technologies 9(1) (2020) 2. https://doi.org/10.3390/technologies9010002
Hu M. and Liu B. - Mining and summarizing customer reviews, Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Seattle, 2004, pp. 168-177.
Liu Q., Gao Z., Liu B., and Zhang Y. - Automated rule selection for aspect extraction in opinion mining, Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), Buenos Aires, 2015, pp. 1291-1297.
Ding X., Liu B., and Yu P. S. - A holistic lexicon-based approach to opinion mining, Proceedings of the 2008 International Conference on Web Search and Data Mining (WSDM), Palo Alto, 2008, pp. 231-240.
Tang D., Qin B., and Liu T. - Aspect level sentiment classification with deep memory network, arXiv preprint arXiv:1605.08900 (2016).
Serra J., Suris D., Miron M., and Karatzoglou A. - Overcoming catastrophic forgetting with hard attention to the task, Proceedings of the 35th International Conference on Machine Learning (ICML), Vol. 80, Stockholm, 2018, pp. 4548-4557.
Downloads
Published
How to Cite
Issue
Section
License
![Creative Commons License](http://i.creativecommons.org/l/by-sa/4.0/88x31.png)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Vietnam Journal of Sciences and Technology (VJST) is an open access and peer-reviewed journal. All academic publications could be made free to read and downloaded for everyone. In addition, articles are published under term of the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA) Licence which permits use, distribution and reproduction in any medium, provided the original work is properly cited & ShareAlike terms followed.
Copyright on any research article published in VJST is retained by the respective author(s), without restrictions. Authors grant VAST Journals System a license to publish the article and identify itself as the original publisher. Upon author(s) by giving permission to VJST either via VJST journal portal or other channel to publish their research work in VJST agrees to all the terms and conditions of https://creativecommons.org/licenses/by-sa/4.0/ License and terms & condition set by VJST.
Authors have the responsibility of to secure all necessary copyright permissions for the use of 3rd-party materials in their manuscript.