Image captioning has attracted more and more attention in remote sensing filed since it provides more specific information than traditional tasks, such as classification. Though image captioning has gained some developments in recent years, it is difficult to describe the image in one simple sentence. To relieve the limitation, a novel captioning task is proposed and a novel framework is proposed to solve the novel task. The proposed framework uses semantic embedding to measure the image representation and the sentence representation. The captioning performance is improved by a proposed sentence representation (collective representation). Experimental results and human evaluations on three captioning data sets in remote sensing field demonstrate that the proposed framework can lead to advancement in image captioning results. © 2019 IEEE.