Bypassing anti-emulation methods for malware detection

Van Loi Cao; Dinh Dai Nguyen

doi:10.15625/1813-9663/20741

Author affiliations

Authors

Van Loi Cao Institute of Information and Communication Technology, Le Quy Don Technical University, 236 Hoang Quoc Viet Street, Cau Giay District, Ha Noi, Viet Nam
Dinh Dai Nguyen Institute of Cryptography Science and Technology, Government Cipher Committee, 24 Ly Thuong Kiet Street, Hoan Kiem District, Ha Noi, Viet Nam

DOI:

https://doi.org/10.15625/1813-9663/20741

Keywords:

Malware analysis, Malware detection, obfuscation, anti-emulation, feature extraction.

Abstract

Malware detection has played a crucial role in many cyberattacks in recent years. Due to the obfuscated nature of malware, the traditional static analysis technique tends to be ineffective. Additionally, modern malware often can identify dynamic analysis environments, posing challenges to dynamic analysis methods. Thus, feature extraction relies on analysis techniques that tend to be less effective in obfuscated malware, resulting in poor performance of subsequent machine learning-based detectors. This study introduces a Bypass Anti-emulation-based Malware Detection framework (BAE-MD) for enhancing the efficiency of obfuscated malware detection. In other words, BAE-MD includes a method that can bypass the anti-emulation mechanism of malware in a controlled dynamic environment. This forces the malware to decrypt and decompress its actual malicious code to memory. By doing so, Yara rules can be applied to memory dump to extract more than $60$ features to feed into detectors. BAE-MD is evaluated on a malware dataset in comparison with others using static and dynamic analysis technique-based feature extraction. The experimental results can confirm that our method outperforms the others. More investigations are also carried out to illustrate the efficiency of BAE-MD. These results suggest that BAE-MD is a promising approach for dealing with the continuous evolution of malware.

Metrics

Metrics Loading ...

References

Jagsir Singh and Jaswinder Singh. A survey on machine learning-based malware detec- tion in executable files. Journal of Systems Architecture, 112:101861, 2021.

Weijie Han, Jingfeng Xue, Yong Wang, Lu Huang, Zixiao Kong, and Limin Mao. Mal- dae: Detecting and explaining malware based on correlation and fusion of static and dynamic characteristics. computers & security, 83:208–233, 2019.

Tuan D. Pham Rajasekhar Chaganti, Vinayakumar Ravi. A multi-view feature fusion approach for effective malware classification using deep learning. Journal of Information Security and Applications, 73, 2023.

Saddaf Rubab Amir Djenna, Ahmed Bouridane and Ibrahim Moussa Marou. Artificial intelligence-based malware detection, analysis, and mitigation. Symmetry, 15(3), 2023.

Senming Yan; Jing Ren; Wei Wang; Limin Sun; Wei Zhang and Quan Yu. A survey of adversarial attack and defense methods for malware classification in cyber security. IEEE Communications Surveys and Tutorials, 25(1), 2023.

Abdullah Asim Yilmaz Semih Serkant Aktug, Merve Ozkan-Okay and Erdal Akin. A comprehensive review of cyber security vulnerabilities, threats, attacks, and solutions. Electronics, 12(6), 2023.

Khalid Khan, Amjad Mehmood, Shafiullah Khan, Muhammad Altaf Khan, Zeeshan Iqbal, and Wali Khan Mashwani. A survey on intrusion detection and prevention in wireless ad-hoc networks. Journal of Systems Architecture, 105:101701, 2020.

Mahdi Abadi Fatemeh Deldar. Deep learning for zero-day malware detection and clas- sification: A survey. ACM Computing Surveys, 56(2):1–37, 2023.

Tao Feng Muhammad Shoaib Akhtar. Malware analysis and detection using machine learning algorithms. Symmetry, 14(11), 2022.

Quang Vinh Dang. Enhancing obfuscated malware detection with machine learning techniques. Communications in Computer and Information Science, 1688, 2022.

Arash Mahboubi Jannatul Ferdous, Rafiqul Islam and Md Zahidul Islam. A review of state-of-the-art malware attack trends and defense mechanisms. IEEE Access, 11, 2023.

Atefeh Mashatan Mohammed M. Alani and Ali Miri. Xmal: A lightweight memory- based explainable obfuscated-malware detector. Computers and Security, 133, 2023.

Muhammad Ikram; Giang Nguyen; Dali Kaafar; Sean Lamont Hassan Jameel Asghar, Benjamin Zi Hao Zhao and Daniel Coscia. Use of cryptography in malware obfuscation. Journal of Computer Virology and Hacking Techniques, 20:135–152, 2024.

Farkhund Iqbal; Mohammed Hussain Rahman Ali, Asmat Ali and Farhan Ullah. Deep learning methods for malware and intrusion detection: A systematic literature review. Security and Communication Networks, 2022.

Muhammad Hanif Durad; Asifullah Khan Umm-e Hani Tayyab, Faiza Babar Khan and Yeon Soo Lee. A survey of the recent trends in deep learning based malware detection. Journal of Cybersecurity and Privacy, 2(4), 2022.

Nguyen Minh Tu, Nguyen Viet Hung, Phan Viet Anh, Cao Van Loi, and Nathan Shone. Detecting malware based on dynamic analysis techniques using deep graph learning. In Future Data and Security Engineering: 7th International Conference, FDSE 2020, Quy Nhon, Vietnam, November 25–27, 2020, Proceedings 7, pages 357–378. Springer, 2020.

Haehyun Cho Minho Kim and Jeong Hyun Yi. Large-scale analysis on anti-analysis techniques in real-world malware. IEEE Access, 10, 2022.

Awadhesh Kumar Singh Amit Sharma, Brij B. Gupta and V.K. Saraswat. Orchestration of apt malware evasive manoeuvers employed for eluding anti-virus and sandbox defense. Comput. Secur., 115, 2022.

Anatoliy Sachenko Pavlo Rehida, George Markowsky and Oleg Savenko. State-based sandbox tool for distributed malware detection with avoid techniques. IEEE Interna- tional Conference on Dependable Systems, Services and Technologies, 2023.

Georgiana Ingrid Stoleru and Dragos Teodor Gavrilut. A practical approach for mal- ware identification based on anti-emulation techniques and feature to image translation. International Conference on Computer Theory and Applications (ICCTA), 2021.

G ̈ok ̧ce Ok Murat Dener and Abdullah Orman. Malware detection using memory analysis data in big data environment. Applied Sciences, 12, 2022.

Lin Yang Weizhong Qiang and Hai Jin. Efficient and robust malware detection based on control flow traces using deep neural networks. Computers and Security, 122, 2022.

Suhuai Luo Kamran Shaukat and Vijay Varadharajan. A novel deep learning-based approach for malware detection. Engineering Applications of Artificial Intelligence, 122, 2023.

Sibi Chakkaravarthy Sethuraman Ph.D Gopinath M. A comprehensive survey on deep learning based malware detection techniques. Computer Science Review, 47, 2023.

Michael Sikorski and Andrew Honig. Practical malware analysis. 2012.

Subarna Basnet Aman Raj Pandey, Tushar Sharma and Sonia Setia. Static analysis approach of malware using machine learning. Cyber Security and Digital Forensics, 896:109–121, 2023.

JyotiPrakashSinghHimanshuKumarSinghandAnandShankerTewari.Staticmalware analysis using machine and deep learning. Proceedings of International Conference on Computing and Communication Networks, 2022.

Viet Hung Nguyen Minh Tu Nguyen and Nathan Shone. Using deep graph learning to improve dynamic analysis-based malware detection in pe files. Journal of Computer Virology and Hacking Techniques, 20:153–172, 2024.

Jan von der Assen; Alberto Huertas Celdr ́an; Adrian Zermin; Raffael Mogicato; G ́erˆome Bovet and Burkhard Stiller. Secbox: A lightweight container-based sandbox for dynamic malware analysis. IEEE/IFIP Network Operations and Management Sym- posium, 2023.

Nguyen Minh Tu, Nguyen Viet Hung, Phan Viet Anh, Cao Van Loi, and Nathan Shone. Detecting malware based on dynamic analysis techniques using deep graph learning. International Conference on Future Data and Security Engineering, 2020.

Abdullah Marish Ali; Fuad A. Ghaleb; Fawaz Jaber Alsolami Faitouri A. Aboaoja, Anazida Zainal and Murad A. Rassam. Dynamic extraction of initial behavior for evasive malware detection. mathematics, 20(2), 2023.

Qiling framework. https://github.com/qilingframework/qiling, 2021.

Unicorn engine. https://github.com/unicorn-engine/unicorn, 2021, 2021.