In this paper, we mainly studied how to calculate the energy values of stars received by ground optical systems. Then according to the characteristics of the radiation spectrum of stars and the radiation spectrum of sky background light, we analyzed and selected the observation spectrum of the image-forming system. We also studied the influence of optical system design parameters, such as field of view, focal length and aperture, on detection capability. At the same time, we analyzed and calculate the limitation of the detector's dark current, readout noise and other noise factors on the ability of receiving stars. The significance of these studies is that we provide an effective theoretical basis for designing and improving ground-based photoelectric detection system for star observation during the daytime. In addition, we used digital image processing technology to process the existing observation images and improve the quality of the image. We provide several algorithms for extracting small targets in strong background. We use threshold segmentation, morphological filtering and other methods to improve the signal-to-noise ratio of the image and then improve the detection ability of the system again. According to the simulation, the target extraction accuracy can reach 1/10 pixels when the target imaging size is 4 pixels and the signal-to-noise ratio is less than 5. Improving the detection ability of photoelectric detection system, detecting more available stars and obtaining their relative position information are the important basis for star map matching and the estimation of targets' position and gesture. © 2019 SPIE.