3D object detection is crucial for many real-world applications, attracting many researchers’ attention. Beyond 2D object detection, 3D object detection usually needs to extract appearance, depth, position and orientation information from the LiDAR and camera sensors. However, due to more degrees- of-freedom and vertices, existing detection methods that directly transform from 2D to 3D still face several challenges, such as exploding increase of anchors’ number and inefficient or hard- to-optimize objective. To this end, we present a fast segmentation method for 3D point clouds to reduce anchors, which can largely decrease computing cost. Moreover, taking advantages of 3D GIoU and L1 losses, we propose a fused loss to facilitate the optimization of 3D object detection. A series of experiments show the proposed method has alleviated the issues above effectively