SGFusion: Camera-LiDAR Semantic and Geometric Fusion for 3D Object Detection

Abstract

Camera and lidar are considered as important sensors to achieve higher-level autonomous driving. And the complementary information provided by these sensors offer more opportunities for improving performance. However, it is difficulty to fuse them because of different representation of both. In this work, we propose a novel fusion framework SGFusion, which fuse the image and point clouds at semantic and geometric level. The SGFusion framework is divided in sequence into two stages: semantic fusion stage and geometric fusion stage. First, the point clouds are painted with the object-level semantic information obtained from an 2D object detector on the semantic fusion stage. At the same time, the output candidates of 2D object detector will be saved. Then, the painted point clouds fed to one LiDAR-based detector for obtaining high-quality 3D detection candidates. Finally, on the geometric fusion stage, these 3D detection candidates and the 2D detection candidates just saved are combined, and the more accurate detection results can be obtained by using their geometric consistence. The experimental results on KITTI detection benchmark show that our SGFusion achieve up to 3.64% AP improvement on car class compared to three different baselines. Furthermore, our method outperforms prior state-of-the-art works on 3D detection of car class on KITTI testing benchmark.