Infrared small target detection has made many breakthroughs in early warning, guidance and battlefield intelligence. However, infrared small target occupies less pixels and lacks color and texture features, which makes infrared small target detection a challenging subject. To achieve the infrared small target detection, an efficient end-to-end network ISTDet is proposed in this paper. ISTDet mainly consists of two modules, including image filtering module and infrared small target detection module. The image filtering module is proposed to obtain the confidence map, aiming to enhance the response of infrared small targets and suppress the response of background. The infrared small target detection module takes the infrared image activated by the confidence map as input, aiming to speculate the category and position of the infrared small targets. Multi-task loss function is used to train the ISTDet in an end-to-end way. Finally, we do comparative experiments on five infrared small target sequences to demonstrate the detection performance of ISTDet. The results show ISTDet has better performance for infrared small target detection compared with other detectors.