In the past few years, Siamese networks have achieved outstanding improvements in visual object tracking. However, visual distractors with similar semantics can be easily misclassified as the target by Siamese networks and may consequently result in the drift problem. Besides, the Hanning window penalty, which is generally used to suppress distractors, could fail in many challengeable scenes. Notably, most failures violate the assumption of motion continuity. Thus, in this work, we explore motion information to mitigate the drift problem in visual tracking. First, we introduce a simple linear Kalman filter to predict the bounding box of the target in the current frame, which acts as a reference for decisions. Second, an IoU-Guided penalty is assembled in the post-processing to suppress distractors effectively. It’s worth mentioning that our method is almost cost-free. We conduct numerous experimental validations and analyses of our approach on several challenging sequences and datasets. Our tracker runs at approximately 40 fps and performs well on those sequences which include the Background Clutter attribute. Finally, by simultaneously integrating the IoU-Guided penalty and the Hanning window penalty with a strong baseline tracker TransT, our method achieves favorable gains by 69.1→71.5, 65.7→66.7, 64.9→65.9 success on OTB-100, LaSOT, NFS.