Public sentiment is regarded as an important measure for event detection, information security, policy making etc. Analyzing public sentiments relies more and more on large amount of multimodal contents, in contrast to the traditional text-based and image-based sentiment analysis. However, most previous works directly extract feature from image as the additional information for text modality and then merge these features for multimodal sentiment analysis. More detailed semantic information in image, like image caption which contains useful semantic components for sentiment analysis, has been ignored. In this paper, we propose a Hierarchical Semantic Attentional Network based on image caption, HSAN, for multimodal sentiment analysis. It has a hierarchical structure that reflects the hierarchical structure of tweet and uses image caption to extract visual semantic feature as the additional information for text in multimodal sentiment analysis task. We also introduce the attention with context mechanism, which learns to consider the context information for encoding. The experiments on two public available datasets show the effectiveness of our model.