Learning Video Moment Retrieval Without a Single Annotated Video

doi:10.1109/TCSVT.2021.3075470

CORC > 自动化研究所 > 中国科学院自动化研究所 > 模式识别国家重点实验室 > 多媒体计算与图形学团队

	Learning Video Moment Retrieval Without a Single Annotated Video
	Gao, Junyu 2,3; Xu, Changsheng 1,2,3
刊名	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
	2022-03-01
卷号	32 期号:3 页码:1646-1657
关键词	Visualization Task analysis Generators Training Graph neural networks Semantics Detectors Video moment retrieval graph neural network unpaired learning
ISSN号	1051-8215
DOI	10.1109/TCSVT.2021.3075470
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要	Video moment retrieval has progressed significantly over the past few years, aiming to search the moment that is most relevant to a given natural language query. Most existing methods are trained in a fully-supervised or a weakly-supervised manner, which requires a time-consuming and expensive manually labeling process. In this work, we propose an alternative approach to achieving video moment retrieval that requires no textual annotations of videos and instead leverages the existing visual concept detectors and a pre-trained image-sentence embedding space. Specifically, we design a video-conditioned sentence generator to produce a suitable sentence representation by utilizing the mined visual concepts in videos. We then design a GNN-based relation-aware moment localizer to reasonably select a portion of video clips under the guidance of the generated sentence. Finally, the pre-trained image-sentence embedding space is adopted to evaluate the matching scores between the generated sentence and moment representations with the knowledge transferred from the image domain. By maximizing these scores, the sentence generator and moment localizer can enhance and complement each other to achieve the moment retrieval task. Experimental results on the Charades-STA and ActivityNet Captions datasets demonstrate the effectiveness of our proposed method.
资助项目	National Key Research and Development Plan of China[2020AAA0106200] ; National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[61721004] ; National Natural Science Foundation of China[62072286] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[61832002] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[62002355] ; National Natural Science Foundation of China[U1836220] ; National Natural Science Foundation of China[U1705262] ; Key Research Program of Frontier Sciences of Chinese Academy of Sciences (CAS)[QYZDJSSWJSC039] ; Beijing Natural Science Foundation[L201001]
WOS关键词	NETWORK
WOS研究方向	Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000766700400059
资助机构	National Key Research and Development Plan of China ; National Natural Science Foundation of China ; Key Research Program of Frontier Sciences of Chinese Academy of Sciences (CAS) ; Beijing Natural Science Foundation
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/48126]
专题	自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者	Xu, Changsheng
作者单位	1.PengCheng Lab, Shenzhen 518066, Peoples R China 2.Chinese Acad Sci, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Gao, Junyu,Xu, Changsheng. Learning Video Moment Retrieval Without a Single Annotated Video[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2022,32(3):1646-1657.
APA	Gao, Junyu,&Xu, Changsheng.(2022).Learning Video Moment Retrieval Without a Single Annotated Video.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,32(3),1646-1657.
MLA	Gao, Junyu,et al."Learning Video Moment Retrieval Without a Single Annotated Video".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 32.3(2022):1646-1657.