Open Access Open Access  Restricted Access Subscription Access

Real-Time Smile Detection using Deep Learning

Chi Cuong Nguyen, Giang Son Tran, Thi Phuong Nghiem, Jean-Christophe Burie, Chi Mai Luong


Real-time smile detection from facial images is useful in many real world applications such as automatic photo capturing in mobile phone cameras or interactive distance learning. In this paper, we study different architectures of object detection deep networks for solving real-time smile detection problem. We then propose a combination of a lightweight convolutional neural network architecture (BKNet) with an efficient object detection framework (RetinaNet). The evaluation on the two datasets (GENKI-4K, UCF Selfie) with a mid-range hardware device (GTX TITAN Black) show that our proposed method helps in improving both accuracy and inference time of the original RetinaNet to reach real-time performance. In comparison with the state-of-the-art object detection framework (YOLO), our method has higher inference time, but still reaches real-time performance and obtains higher accuracy of smile detection on both experimented datasets.


Deep Learning, Convolutional Neural Network, Real-Time Smile Detection

Full Text:



F. De la Torre and J. F. Cohn. Facial expression analysis. In Visual analysis of humans, pages 377–409. Springer, 2011.

P. Viola and M. J. Jones. Robust real-time face detection. International journal of computer vision, 57(2):137–154, 2004.

D. Freire, M. C. Santana, and O. D ́eniz-Su ́arez. Smile detection using local binary patterns and support vector machines. In VISAPP (1), pages 398–401, 2009.

O. D ́eniz, M. Castrillon, J. Lorenzo, L. Anton, and G. Bueno. Smile detection for user interfaces. In International Symposium on Visual Computing, pages 602–611. Springer, 2008.

Y. Gao, H. Liu, P. Wu, and C. Wang. A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing, 174:1077–1086, 2016.

D.-Y. Huang, C.-H. Chen, T.-Y. Chen, J.-H. Wu, and C.-C. Ko. Real-time face detection using a moving camera. In 2018 32nd International Conference on Advanced Information Networking and Applications Workshops (WAINA), pages 609–614. IEEE, 2018.

C. C. Nguyen, G. S. Tran, T. P. Nghiem, N. Q. Doan, D. Gratadour, J. C. Burie, and C. M. Luong. Towards real-time smile detection based on faster region convolutional neural network. In Multimedia Analysis and Pattern Recognition (MAPR), 2018 1st International Conference on, pages 1–6. IEEE, 2018.

J. Redmon and A. Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6517–6525, 2017.

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ́ar. Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence, 2018.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

V. S. Dinh, T. B. C. Le, and P. T. Do. Facial smile detection using convolutional neural networks. In Knowledge and Systems Engineering (KSE), 2017 9th International Conference on, pages 136–141. IEEE, 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. Proceedings of International Conference on Learning Representations, 2014.

Y. Shinohara and N. Otsuf. Facial expression recognition using fisher weight maps. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, pages 499–504. IEEE, 2004.

A. Ito, X. Wang, M. Suzuki, and S. Makino. Smile and laughter recognition using speech processing and face recognition from conversation video. In Cyberworlds, 2005. International Conference on, pages 8–pp. IEEE, 2005.

D. G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.

O. D ́eniz, G. Bueno, J. Salido, and F. De la Torre. Face recognition using histograms of oriented gradients. Pattern Recognition Letters, 32(12):1598–1603, 2011.

L. An, S. Yang, and B. Bhanu. Efficient smile detection by extreme learning machine. Neuro- computing, 149:354–363, 2015.

J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, and J. Movellan. Toward practical smile detection. IEEE transactions on pattern analysis and machine intelligence, 31(11):2106–2111, 2009.

K. Zhang, Y. Huang, H. Wu, and L. Wang. Facial smile detection based on deep learning features. In Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on, pages 534–538. IEEE, 2015.

J. Chen, Q. Ou, Z. Chi, and H. Fu. Smile detection in the wild with deep convolutional neural networks. Machine vision and applications, 28(1-2):173–183, 2017.

H. Zhang, X. Wang, J. Zhu, and C.-C. J. Kuo. Accelerating proposal generation network for fast face detection on mobile devices. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 326–330. IEEE, 2018. The MPLab GENKI Database, GENKI-4K Subset.

M. M. Kalayeh, M. Seifu, W. LaLanne, and M. Shah. How to take a good selfie? In Proceedings of the 23rd ACM International Conference on Multimedia, MM ’15, pages 923–926, New York, NY, USA, 2015. ACM.

P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, pages I–I. IEEE, 2001.

DOI: Display counter: Abstract : 384 views. PDF : 216 views.

Journal of Computer Science and Cybernetics ISSN: 1813-9663

Published by Vietnam Academy of Science and Technology