Improving the quality of image-guided radiation therapy requires the tracking of respiratory motion in ultrasound sequences. However, the low signal-to-noise ratio and the artifacts in ultrasound images make it difficult to track targets accurately and robustly. In this study, we propose a novel deep learning model, called a Cascaded One-shot Deformable Convolutional Neural Network (COSD-CNN), to track landmarks in real time in long ultrasound sequences. Specifically, we design a cascaded Siamese network structure to improve the tracking performance of CNN-based methods. We propose a one-shot deformable convolution module to enhance the robustness of the COSD-CNN to appearance variation in a meta-learning manner. Moreover, we design a simple and efficient unsupervised strategy to facilitate the network's training with a limited number of medical images, in which many corner points are selected from raw ultrasound images to learn network features with high generalizability. The proposed COSD-CNN has been extensively evaluated on the public Challenge on Liver UltraSound Tracking (CLUST) 2D dataset and on our own ultrasound image dataset from the First Affiliated Hospital of Sun Yat-sen University (FSYSU). Experiment results show that the proposed model can track a target through an ultrasound sequence with high accuracy and robustness. Our method achieves new state-of-the-art performance on the CLUST 2D benchmark set, indicating its strong potential for application in clinical practice.