yinjunbo/CenterPoint-Fusion

The proposed approach enhances the CenterPoint baseline with a multimodal fusion mechanism. First, inspired by PointPainting, an off-the-shelf Mask-RCNN model trained from nuImages is employed to generate 2D object mask information based on the camera images. Furthermore, the Cylinder3D is also adopted to produce the 3D semantic information of t...

The proposed approach enhances the CenterPoint baseline with a multimodal fusion mechanism. First, inspired by PointPainting, an off-the-shelf Mask-RCNN model trained from nuImages is employed to generate 2D object mask information based on the camera images. Furthermore, the Cylinder3D is also adopted to produce the 3D semantic information of the input LiDAR point cloud. Then, an improved version of CenterPoint takes the painted points(with 2D instance segmentation and 3D semantic segmentation) as inputs for accurate object detection. Specifically, we replace the RPN module in CenterPoint with modified Spatial-Semantic Feature Aggregation(SSFA) to well address multi-class detection. A simple pseudo labeling technique is also integrated in a semi-supervised learning manner. In addition, the Test Time Augmentation(TTA) strategy including multiple flip and rotation operations is applied during the inference time. Finally, the detections generated from multiple voxel resolutions (0.05m to 0.125m) are assembled with 3D Weighted Bounding Box Fusion(WBF) technique to produce the final results.