Emotion Knowledge Driven Video Highlight Detection

doi:10.1109/TMM.2020.3035285

CORC > 自动化研究所 > 中国科学院自动化研究所 > 模式识别国家重点实验室 > 多媒体计算与图形学团队

	Emotion Knowledge Driven Video Highlight Detection
	Qi, Fan 1,4; Yang, Xiaoshan 1,2,3; Xu, Changsheng 1,2,3
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2021
卷号	23 页码:3999-4013
关键词	Visualization Training data Predictive models Training Semantics Emotion recognition Computational modeling Deep ranking knowledge graph video highlight detection
ISSN号	1520-9210
DOI	10.1109/TMM.2020.3035285
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要	This paper addresses video highlight detection which aims to select a small subset of frames according to user's major or special interest. The performances of conventional methods highly depend on large-scale manually labeled training data which are time-consuming and labor-intensive to collect. To deal with this problem, we trace back to the original problem definition and find that whether a user is interested in a specific video segment heavily depends on human's subjective emotions. Leveraging this insight, we introduce an emotion knowledge driven video detection framework for modeling human's general emotion and inferencing highlight strength. Firstly, we obtain the concept-level representation of the video clip with a front-end network. The concepts are used as nodes to build an emotion-related knowledge graph, and their relationships in the graph are modeled via external public knowledge graphs. Then we adopt Siamese GCNs to model the dependencies between nodes in the graph and propagate messages along the edges. Finally, we compute the emotion-aware representation of the video clip based on the GCN layers and further use it to predict the highlight score. Our framework, including the front-end network, graph convolution layers and the highlight mapping network, can be trained in an end-to-end manner with the constraint of a ranking loss. Experiments on two benchmark datasets show that our proposed method performs favorably against the state-of-the-art methods.
资助项目	National Key Research and Development Program of China[2018AAA0100604] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[61702511] ; National Natural Science Foundation of China[61751211] ; National Natural Science Foundation of China[61620106003] ; National Natural Science Foundation of China[61532009] ; National Natural Science Foundation of China[U1836220] ; National Natural Science Foundation of China[U1705262] ; National Natural Science Foundation of China[61872424] ; Key Research Program of Frontier Sciences of CAS[QYZDJSSWJSC039] ; Research Program of National Laboratory of Pattern Recognition[Z-2018007]
WOS关键词	RETRIEVAL
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000720519900007
资助机构	National Key Research and Development Program of China ; National Natural Science Foundation of China ; Key Research Program of Frontier Sciences of CAS ; Research Program of National Laboratory of Pattern Recognition
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/46455]
专题	自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者	Xu, Changsheng
作者单位	1.Peng Cheng Lab, Shenzhen 518055, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 4.HeFei Univ Technol, Comp Sci, Hefei 230009, Peoples R China
推荐引用方式 GB/T 7714	Qi, Fan,Yang, Xiaoshan,Xu, Changsheng. Emotion Knowledge Driven Video Highlight Detection[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2021,23:3999-4013.
APA	Qi, Fan,Yang, Xiaoshan,&Xu, Changsheng.(2021).Emotion Knowledge Driven Video Highlight Detection.IEEE TRANSACTIONS ON MULTIMEDIA,23,3999-4013.
MLA	Qi, Fan,et al."Emotion Knowledge Driven Video Highlight Detection".IEEE TRANSACTIONS ON MULTIMEDIA 23(2021):3999-4013.