Debiased Video-Text Retrieval via Soft Positive Sample Calibration

doi:10.1109/TCSVT.2023.3248873

CORC > 自动化研究所 > 中国科学院自动化研究所 > 多模态人工智能系统全国重点实验室

	Debiased Video-Text Retrieval via Soft Positive Sample Calibration
	Zhang, Huaiwen 1,4,5; Yang, Yang 1,4,5; Qi, Fan 6; Qian, Shengsheng 2,3; Xu, Changsheng 2,3
刊名	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
	2023-09-01
卷号	33 期号:9 页码:5257-5270
关键词	debias soft positive samples
ISSN号	1051-8215
DOI	10.1109/TCSVT.2023.3248873
通讯作者	Qian, Shengsheng(shengsheng.qian@nlpr.ia.ac.cn)
英文摘要	With the emergence of enormous videos on various video apps, semantic video-text retrieval has become a critical task for improving the user experience. The primary paradigm for video-text retrieval learns the semantic video-text representations in a common space by pulling the positive samples close to the query and pushing the negative samples away. However, in practice, the video-text datasets contain only the annotations of positive samples. The negative samples are randomly drawn from the entire dataset. There may exist soft positive samples, which are sampled as negatives but share the same semantics as positive samples. Indiscriminately enforcing the model to push all the negative samples away from the query leads to inaccurate supervision and then misleads the video-text feature representation learning. In this paper, we introduce debiased video-text retrieval objectives that calibrate the punishment of soft positive samples. In particular, we propose a novel uncertainty measure framework to estimate the credibility of negative samples for each instance. Then, the reliability of negative samples is used to find the soft positive samples and rescale their contribution within video-text retrieval losses, including triplet loss and contrastive loss. Experimental results on five widely used datasets demonstrate that our debiased video-text retrieval objectives achieve significant performance improvements and establish a new state-of-the-art.
资助项目	National Natural Science Foundation of China[62206137] ; National Natural Science Foundation of China[62206200] ; National Natural Science Foundation of China[62276257] ; National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[62066033] ; National Natural Science Foundation of Inner Mongolia[2022MS06025] ; Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region[NJYT23105] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2020GG0046] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2021GG0158] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2020PT0002] ; National Natural Science Foundation of China[62206137] ; National Natural Science Foundation of China[62206200] ; National Natural Science Foundation of China[62276257] ; National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[62066033] ; National Natural Science Foundation of Inner Mongolia[2022MS06025] ; Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region[NJYT23105] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2020GG0046] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2021GG0158] ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region[2020PT0002]
WOS研究方向	Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:001063316800060
资助机构	National Natural Science Foundation of China ; National Natural Science Foundation of Inner Mongolia ; Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region ; National Natural Science Foundation of China ; National Natural Science Foundation of Inner Mongolia ; Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region ; Applied Technology Research and Development Program of Inner Mongolia Autonomous Region
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/53152]
专题	多模态人工智能系统全国重点实验室
通讯作者	Qian, Shengsheng
作者单位	1.Inner Mongolia Key Lab Mongolian Informat Proc Tec, Hohhot 010021, Peoples R China 2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 4.Inner Mongolia Univ, Coll Comp Sci, Hohhot 010021, Peoples R China 5.Natl & Local Joint Engn Res Ctr Intelligent Inform, Hohhot 010021, Peoples R China 6.Tianjin Univ Technol, Sch Comp Sci & Engn, Tianjin 300384, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Huaiwen,Yang, Yang,Qi, Fan,et al. Debiased Video-Text Retrieval via Soft Positive Sample Calibration[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2023,33(9):5257-5270.
APA	Zhang, Huaiwen,Yang, Yang,Qi, Fan,Qian, Shengsheng,&Xu, Changsheng.(2023).Debiased Video-Text Retrieval via Soft Positive Sample Calibration.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,33(9),5257-5270.
MLA	Zhang, Huaiwen,et al."Debiased Video-Text Retrieval via Soft Positive Sample Calibration".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 33.9(2023):5257-5270.