Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection

doi:10.1109/TCSVT.2018.2799968

	Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
	Liu, Jiaying 1; Li, Yanghao 1; Song, Sijie 1; Xing, Junliang2 ; Lan, Cuiling 3; Zeng, Wenjun 3
刊名	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
	2019-09-01
卷号	29 期号:9 页码:2667-2682
关键词	Action detection recurrent neural network multi-modality joint classification-regression
ISSN号	1051-8215
DOI	10.1109/TCSVT.2018.2799968
通讯作者	Liu, Jiaying(liujiaying@pku.edu.cn)
英文摘要	Online action detection is a brand new challenge and plays a critical role in visual surveillance analytics. It goes one step further than a conventional action recognition task, which recognizes human actions from well-segmented clips. Online action detection is desired to identify the action type and localize action positions on the fly from the untrimmed stream data. In this paper, we propose a multi-modality multi-task recurrent neural network, which incorporates both RGB and Skeleton networks. We design different temporal modeling networks to capture specific characteristics from various modalities. Then, a deep long short-term memory subnetwork is utilized effectively to capture the complex long-range temporal dynamics, naturally avoiding the conventional sliding window design and thus ensuring high computational efficiency. Constrained by a multi-task objective function in the training phase, this network achieves superior detection performance and is capable of automatically localizing the start and end points of actions more accurately. Furthermore, embedding subtask of regression provides the ability to forecast the action prior to its occurrence. We evaluate the proposed method and several other methods in action detection and forecasting on the online action detection data set and gaming action data set datasets. Experimental results demonstrate that our model achieves the state-of-the-art performance on both tasks.
资助项目	NVIDIA Corporation
WOS关键词	ACTION RECOGNITION ; ENSEMBLE ; MOTION
WOS研究方向	Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000489738900012
资助机构	NVIDIA Corporation
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/26635]
专题	中国科学院自动化研究所
通讯作者	Liu, Jiaying
作者单位	1.Peking Univ, Inst Comp Sci & Technol, Beijing 100080, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100080, Peoples R China 3.Microsoft Res Asia, Beijing 100080, Peoples R China
推荐引用方式 GB/T 7714	Liu, Jiaying,Li, Yanghao,Song, Sijie,et al. Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(9):2667-2682.
APA	Liu, Jiaying,Li, Yanghao,Song, Sijie,Xing, Junliang,Lan, Cuiling,&Zeng, Wenjun.(2019).Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(9),2667-2682.
MLA	Liu, Jiaying,et al."Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.9(2019):2667-2682.