Data association is a crucial part for tracking-by-detection framework. Although many works about constructing the matching cost between trajectories and detections have been proposed in the community, few researchers pay attention to how to improve the efficiency of bipartite graph matching in realtime multi-object tracking. In this paper, we start with the optimal solution of integer linear programming, explore the best application of bipartite graph matching in tracking task and evaluate the rationality of cost matrix simultaneously. Frist, we analyze the defects of bipartite graph matching process in some multi-object tracking methods, and establish a criteria of similarity measure between trajectories and detections. Then we design two weight matrices for multi-object tracking by applying our criteria. Besides, a novel tracking process is proposed to handle visual-information-free scenario. Our method improves the accuracy of the graph-matching-based approach at very fast running speed (3000+ FPS). Comprehensive experiments performed on MOT benchmarks demonstrate that the proposed approach achieves the state-of-the-art performance in methods without visual information. Moreover, the efficient matching process can also be assembled on approaches with appearance information to replace cascade matching.