Emotion recognition with facial landmark heatmaps

Abstract

Facial expression recognition is a very challenging problem and has attracted more and more researchers’ attention. In this paper, considering that facial expression recognition is closely related to the features of key facial regions, we propose a facial expression recognition network that explicitly utilizes the landmark heatmap information to precisely capture the most discriminative features. In addition to directly adding the information of facial fiducial points in the form of landmark heatmaps, we also propose an end-to-end network structure--heatmap aiding emotion network (HAE-Net) by embedding the landmark detection module based on stack-based hourglass network into the facial expression recognition network. Experiments on CK+, RAF and AffectNet databases show that our method achieves better results compared with the state-of-the-art methods, which demonstrates that adding additional landmark information, as well as joint training of landmark detection and expression recognition, are beneficial to improve recognition performance.