There are great demands for automatically regu-lating inappropriate appearance of shocking firearm images insocial media or identifying firearm types in forensics. Image re-trieval techniques have great potential to solve these problems. Tofacilitate research in this area, we introduce Firearm 14k, a largedataset consisting of over 14,000 images in 167 categories. It canbe used for both fine-grained recognition and retrieval of firearmimages. Recent advances in image retrieval are mainly drivenby fine-tuning state-of-the-art convolutional neural networks forretrieval task. The conventional single margin contrastive loss,known for its simplicity and good performance, has been widelyused. We find that it performs poorly on the Firearm 14k datasetdue to: (1) Loss contributed by positive and negative imagepairs is unbalanced during training process. (2) A huge domaingap exists between this dataset and ImageNet. We propose todeal with the unbalanced loss by employing a double margincontrastive loss. We tackle the domain gap issue with a two-stage training strategy, where we first fine-tune the network forclassification, and then fine-tune it for retrieval. Experimentalresults show that our approach outperforms the conventionalsingle margin approach by a large margin (up to 88.5% relativeimprovement) and even surpasses the strong triplet-loss-basedapproach.