Fusion of Color/Infrared Video for Human Detection

Abstract

In this chapter, we approach the task of human silhouette extraction from color and thermal image sequences using automatic image registration. Image registration between color and thermal images is a challenging problem due to the difficulties associated with finding correspondence. However, moving people in a static scene provide cues to address this problem. We first propose a hierarchical scheme to automatically find the correspondence between the preliminary human silhouettes extracted from synchronous color and thermal image sequences for image registration. Next, we discuss strategies for probabilistically combining cues from registered color and thermal images for improved human silhouette detection. It is shown that the proposed approach achieves good results for image registration and human silhouette extraction. Experimental results also show a comparison of various sensor fusion strategies and demonstrate the improvement in performance over non-fused cases for human silhouette extraction. The initial step of most of the gait recognition approaches is human silhouette extraction {[}72, 82, 107, 126, 137, 166]. Many gait recognition approaches use electro-optical (EO) sensors such as CCD cameras. However, it is very likely that some part of the human body or clothing has colors similar to the background. In this case, human silhouette extraction usually fails on this part. Moreover, the existence of shadows is a problem for EO sensors {[}121]. In addition, EO sensors do not work under low lighting conditions such as night or indoor environment without lighting. The top rows in Fig. 6.1 show human silhouette extraction results from two color images. To avoid the disadvantages of using EO sensors, infrared (ER) sensors are used for object detection {[}122, 148]. We investigate the possibility of using an IR sensor for gait analysis {[}14]. Unlike a commonly used video camera that operates in the visible band of the spectrum and records reflected light, a long wave (8-12 mu m) IR sensor records electromagnetic radiations emitted by objects in a scene as a thermal image whose pixel values represent temperature. In a thermal image that consists of humans in a scene, human silhouettes can be generally extracted from the background regardless of lighting conditions and colors of the human clothing and skin, and backgrounds because the temperatures of the human body and background are different in most situations {[}6]. Although the human silhouette extraction results from IR sensors are generally better than that from EO sensors, human silhouette extraction is unreliable when some part of the human body or clothing has the temperature similar to the background temperature. In addition, human body casts obvious projection on smooth surfaces such as a smooth floor. The last two rows in Fig. 6.1 show human silhouette extraction results from a thermal image. In Fig. 6.1, notice that the unreliably extracted body parts from one sensor might be reliably extracted from the other sensor. This provides an opportunity for improving the human detection performance by the fusion of EO and IR sensors {[}62].