Real-Time Smile Detection using Deep Learning


  • Chi Cuong Nguyen University of Science and Technology of Hanoi
  • Giang Son Tran University of Science and Technology of Hanoi
  • Thi Phuong Nghiem University of Science and Technology of Hanoi
  • Jean-Christophe Burie University of La Rochelle
  • Chi Mai Luong Institute of Information Technology



Deep Learning, Convolutional Neural Network, Real-Time Smile Detection


Real-time smile detection from facial images is useful in many real world applications such as automatic photo capturing in mobile phone cameras or interactive distance learning. In this paper, we study different architectures of object detection deep networks for solving real-time smile detection problem. We then propose a combination of a lightweight convolutional neural network architecture (BKNet) with an efficient object detection framework (RetinaNet). The evaluation on the two datasets (GENKI-4K, UCF Selfie) with a mid-range hardware device (GTX TITAN Black) show that our proposed method helps in improving both accuracy and inference time of the original RetinaNet to reach real-time performance. In comparison with the state-of-the-art object detection framework (YOLO), our method has higher inference time, but still reaches real-time performance and obtains higher accuracy of smile detection on both experimented datasets.


Metrics Loading ...

Author Biographies

Chi Cuong Nguyen, University of Science and Technology of Hanoi

ICT Department

Giang Son Tran, University of Science and Technology of Hanoi

ICT Department

Thi Phuong Nghiem, University of Science and Technology of Hanoi

ICT Department

Jean-Christophe Burie, University of La Rochelle

L3i Laboratory


F. De la Torre and J. F. Cohn. Facial expression analysis. In Visual analysis of humans, pages 377–409. Springer, 2011.

P. Viola and M. J. Jones. Robust real-time face detection. International journal of computer vision, 57(2):137–154, 2004.

D. Freire, M. C. Santana, and O. D ́eniz-Su ́arez. Smile detection using local binary patterns and support vector machines. In VISAPP (1), pages 398–401, 2009.

O. D ́eniz, M. Castrillon, J. Lorenzo, L. Anton, and G. Bueno. Smile detection for user interfaces. In International Symposium on Visual Computing, pages 602–611. Springer, 2008.

Y. Gao, H. Liu, P. Wu, and C. Wang. A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing, 174:1077–1086, 2016.

D.-Y. Huang, C.-H. Chen, T.-Y. Chen, J.-H. Wu, and C.-C. Ko. Real-time face detection using a moving camera. In 2018 32nd International Conference on Advanced Information Networking and Applications Workshops (WAINA), pages 609–614. IEEE, 2018.

C. C. Nguyen, G. S. Tran, T. P. Nghiem, N. Q. Doan, D. Gratadour, J. C. Burie, and C. M. Luong. Towards real-time smile detection based on faster region convolutional neural network. In Multimedia Analysis and Pattern Recognition (MAPR), 2018 1st International Conference on, pages 1–6. IEEE, 2018.

J. Redmon and A. Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6517–6525, 2017.

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ́ar. Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence, 2018.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

V. S. Dinh, T. B. C. Le, and P. T. Do. Facial smile detection using convolutional neural networks. In Knowledge and Systems Engineering (KSE), 2017 9th International Conference on, pages 136–141. IEEE, 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. Proceedings of International Conference on Learning Representations, 2014.

Y. Shinohara and N. Otsuf. Facial expression recognition using fisher weight maps. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, pages 499–504. IEEE, 2004.

A. Ito, X. Wang, M. Suzuki, and S. Makino. Smile and laughter recognition using speech processing and face recognition from conversation video. In Cyberworlds, 2005. International Conference on, pages 8–pp. IEEE, 2005.

D. G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.

O. D ́eniz, G. Bueno, J. Salido, and F. De la Torre. Face recognition using histograms of oriented gradients. Pattern Recognition Letters, 32(12):1598–1603, 2011.

L. An, S. Yang, and B. Bhanu. Efficient smile detection by extreme learning machine. Neuro- computing, 149:354–363, 2015.

J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, and J. Movellan. Toward practical smile detection. IEEE transactions on pattern analysis and machine intelligence, 31(11):2106–2111, 2009.

K. Zhang, Y. Huang, H. Wu, and L. Wang. Facial smile detection based on deep learning features. In Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on, pages 534–538. IEEE, 2015.

J. Chen, Q. Ou, Z. Chi, and H. Fu. Smile detection in the wild with deep convolutional neural networks. Machine vision and applications, 28(1-2):173–183, 2017.

H. Zhang, X. Wang, J. Zhu, and C.-C. J. Kuo. Accelerating proposal generation network for fast face detection on mobile devices. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 326–330. IEEE, 2018. The MPLab GENKI Database, GENKI-4K Subset. The MPLab GENKI Database, GENKI-4K Subset.">

M. M. Kalayeh, M. Seifu, W. LaLanne, and M. Shah. How to take a good selfie? In Proceedings of the 23rd ACM International Conference on Multimedia, MM ’15, pages 923–926, New York, NY, USA, 2015. ACM.

P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, pages I–I. IEEE, 2001.




How to Cite

C. C. Nguyen, G. S. Tran, T. P. Nghiem, J.-C. Burie, and C. M. Luong, “Real-Time Smile Detection using Deep Learning”, JCC, vol. 35, no. 2, p. 135–145, Jun. 2019.



Computer Science