Refocused Attention: Long Short-Term Rewards Guided Video Captioning

doi:10.1007/s11063-019-10030-y

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Refocused Attention: Long Short-Term Rewards Guided Video Captioning
	Dong, Jiarong 1,2; Gao, Ke 1; Chen, Xiaokai 1,2; Cao, Juan 1
刊名	NEURAL PROCESSING LETTERS
	2020-10-01
卷号	52 期号:2 页码:935-948
关键词	Video captioning Hierarchical attention Reinforcement learning Reward
ISSN号	1370-4621
DOI	10.1007/s11063-019-10030-y
英文摘要	The adaptive cooperation of visual model and language model is essential for video captioning. However, due to the lack of proper guidance for each time step in end-to-end training, the over-dependence of language model often results in the invalidation of attention-based visual model, which is called 'Attention Defocus' problem in this paper. Based on an important observation that the recognition precision of entity word can reflect the effectiveness of the visual model, we propose a novel strategy called refocused attention to optimize the training and cooperating of visual model and language model, using ingenious guidance at appropriate time step. The strategy consists of a short-term-reward guided local entity recognition and a long-term-reward guided global relation understanding, neither requires any external training data. Moreover, a framework with hierarchical visual representations and hierarchical attention is established to fully exploit the potential strength of the proposed learning strategy. Extensive experiments demonstrate that the ingenious guidance strategy together with the optimized structure outperform state-of-the-art video captioning methods with relative improvements 7.7% in BLEU-4 and 5.0% in CIDEr-D on MSVD dataset, even without multi-modal features.
WOS研究方向	Computer Science
语种	英语
出版者	SPRINGER
WOS记录号	WOS:000574296700001
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/15637]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Ke
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Dong, Jiarong,Gao, Ke,Chen, Xiaokai,et al. Refocused Attention: Long Short-Term Rewards Guided Video Captioning[J]. NEURAL PROCESSING LETTERS,2020,52(2):935-948.
APA	Dong, Jiarong,Gao, Ke,Chen, Xiaokai,&Cao, Juan.(2020).Refocused Attention: Long Short-Term Rewards Guided Video Captioning.NEURAL PROCESSING LETTERS,52(2),935-948.
MLA	Dong, Jiarong,et al."Refocused Attention: Long Short-Term Rewards Guided Video Captioning".NEURAL PROCESSING LETTERS 52.2(2020):935-948.