Real-Time Smile Detection using Deep Learning
Author affiliations
DOI:
https://doi.org/10.15625/1813-9663/35/2/13315Keywords:
Deep Learning, Convolutional Neural Network, Real-Time Smile DetectionAbstract
Real-time smile detection from facial images is useful in many real world applications such as automatic photo capturing in mobile phone cameras or interactive distance learning. In this paper, we study different architectures of object detection deep networks for solving real-time smile detection problem. We then propose a combination of a lightweight convolutional neural network architecture (BKNet) with an efficient object detection framework (RetinaNet). The evaluation on the two datasets (GENKI-4K, UCF Selfie) with a mid-range hardware device (GTX TITAN Black) show that our proposed method helps in improving both accuracy and inference time of the original RetinaNet to reach real-time performance. In comparison with the state-of-the-art object detection framework (YOLO), our method has higher inference time, but still reaches real-time performance and obtains higher accuracy of smile detection on both experimented datasets.
Metrics
References
F. De la Torre and J. F. Cohn. Facial expression analysis. In Visual analysis of humans, pages 377–409. Springer, 2011.
P. Viola and M. J. Jones. Robust real-time face detection. International journal of computer vision, 57(2):137–154, 2004.
D. Freire, M. C. Santana, and O. D ́eniz-Su ́arez. Smile detection using local binary patterns and support vector machines. In VISAPP (1), pages 398–401, 2009.
O. D ́eniz, M. Castrillon, J. Lorenzo, L. Anton, and G. Bueno. Smile detection for user interfaces. In International Symposium on Visual Computing, pages 602–611. Springer, 2008.
Y. Gao, H. Liu, P. Wu, and C. Wang. A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing, 174:1077–1086, 2016.
D.-Y. Huang, C.-H. Chen, T.-Y. Chen, J.-H. Wu, and C.-C. Ko. Real-time face detection using a moving camera. In 2018 32nd International Conference on Advanced Information Networking and Applications Workshops (WAINA), pages 609–614. IEEE, 2018.
C. C. Nguyen, G. S. Tran, T. P. Nghiem, N. Q. Doan, D. Gratadour, J. C. Burie, and C. M. Luong. Towards real-time smile detection based on faster region convolutional neural network. In Multimedia Analysis and Pattern Recognition (MAPR), 2018 1st International Conference on, pages 1–6. IEEE, 2018.
J. Redmon and A. Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6517–6525, 2017.
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ́ar. Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence, 2018.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
V. S. Dinh, T. B. C. Le, and P. T. Do. Facial smile detection using convolutional neural networks. In Knowledge and Systems Engineering (KSE), 2017 9th International Conference on, pages 136–141. IEEE, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. Proceedings of International Conference on Learning Representations, 2014.
Y. Shinohara and N. Otsuf. Facial expression recognition using fisher weight maps. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, pages 499–504. IEEE, 2004.
A. Ito, X. Wang, M. Suzuki, and S. Makino. Smile and laughter recognition using speech processing and face recognition from conversation video. In Cyberworlds, 2005. International Conference on, pages 8–pp. IEEE, 2005.
D. G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.
O. D ́eniz, G. Bueno, J. Salido, and F. De la Torre. Face recognition using histograms of oriented gradients. Pattern Recognition Letters, 32(12):1598–1603, 2011.
L. An, S. Yang, and B. Bhanu. Efficient smile detection by extreme learning machine. Neuro- computing, 149:354–363, 2015.
J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, and J. Movellan. Toward practical smile detection. IEEE transactions on pattern analysis and machine intelligence, 31(11):2106–2111, 2009.
K. Zhang, Y. Huang, H. Wu, and L. Wang. Facial smile detection based on deep learning features. In Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on, pages 534–538. IEEE, 2015.
J. Chen, Q. Ou, Z. Chi, and H. Fu. Smile detection in the wild with deep convolutional neural networks. Machine vision and applications, 28(1-2):173–183, 2017.
H. Zhang, X. Wang, J. Zhu, and C.-C. J. Kuo. Accelerating proposal generation network for fast face detection on mobile devices. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 326–330. IEEE, 2018.
http://mplab.ucsd.edu. The MPLab GENKI Database, GENKI-4K Subset.
http://mplab.ucsd.edu. The MPLab GENKI Database, GENKI-4K Subset.">
M. M. Kalayeh, M. Seifu, W. LaLanne, and M. Shah. How to take a good selfie? In Proceedings of the 23rd ACM International Conference on Multimedia, MM ’15, pages 923–926, New York, NY, USA, 2015. ACM.
P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, pages I–I. IEEE, 2001.
Downloads
Published
How to Cite
Issue
Section
License
1. We hereby assign copyright of our article (the Work) in all forms of media, whether now known or hereafter developed, to the Journal of Computer Science and Cybernetics. We understand that the Journal of Computer Science and Cybernetics will act on my/our behalf to publish, reproduce, distribute and transmit the Work.2. This assignment of copyright to the Journal of Computer Science and Cybernetics is done so on the understanding that permission from the Journal of Computer Science and Cybernetics is not required for me/us to reproduce, republish or distribute copies of the Work in whole or in part. We will ensure that all such copies carry a notice of copyright ownership and reference to the original journal publication.
3. We warrant that the Work is our results and has not been published before in its current or a substantially similar form and is not under consideration for another publication, does not contain any unlawful statements and does not infringe any existing copyright.
4. We also warrant that We have obtained the necessary permission from the copyright holder/s to reproduce in the article any materials including tables, diagrams or photographs not owned by me/us.
