Facial Landmark Localization Using an Ensemble of Regression Trees on PC Game Controlling

Abstract

Traditional PC gaming system has remained unmodified over the years. While GPU’s and high graphics machines have taken the gaming industry to the next level, there is a need to physically improve the gamers’ experience. Facial landmarks are the localization of certain points on the face. This paper proposes a novel model to control the computer games. It identifies the face and facial landmarks of the player sitting in front of the PC. To do this an ensemble regression trees are used with cascaded regression model. It performs various computations on the extracted landmarks of the input face to estimate the action to be performed. Then, it gives input to the game accordingly. Likewise, many of the keyboard and mouse controls can be mapped with facial movements. This paper gives an overview of head, eyes, and mouth mapping with the game controls. It eliminates the use of a keyboard or any such hardware device, thus creating a realistic gaming experience. Results show that with a continuous frame by frame input of a single face, the proposed model takes 209 milli-sec in poor lighting conditions and 12 milli-sec in good lighting conditions to identify and perform computations on the facial movements. Moreover, the observed gameplay of the final model is very smooth and robust.