Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM

doi:10.1007/s11042-023-16130-1

CORC > 自动化研究所 > 中国科学院自动化研究所 > 复杂系统认知与决策实验室

	Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM
	Lu, Zhi 4; Qin, Shiyin 2,3; Lv, Pin 4; Sun, Liguo 4; Tang, Bo 1
刊名	MULTIMEDIA TOOLS AND APPLICATIONS
	2023-07-14
页码	38
关键词	Continuous detection Gesture recognition Long short-term memory 3D densely connected convolutional networks Connectionist temporal classification Canonical correlation analysis
ISSN号	1380-7501
DOI	10.1007/s11042-023-16130-1
通讯作者	Lu, Zhi(zhi.lu@ia.ac.cn)
英文摘要	With the continuous development of the deep learning theory, novel gesture recognition approaches have been constantly emerging, and the performance has also been continuously improved. However, most research methods focus on the recognition of isolated gestures, and the detection and recognition of continuous gestures are rarely studied. To this end, aiming at the real-time detection and classification of dynamic gestures in untrimmed sequences, a well-designed end-to-end architecture based on the variants of 3D DenseNet and unidirectional LSTM was hereby proposed as an effective tool to extract the discriminative spatio-temporal features of untrimmed hand gesture sequences. Then, connectionist temporal classification was combined to train the network on a publicly available dataset, and some effective capacities could be transferred to enhance the learning ability of the proposed network by training large gesture samples. In this way, the class-conditional probability of an incoming sequence belonging to a given gesture class was predicted and then compared with a predefined threshold to automatically determine the start and end of gestures. In addition, to enhance the classification accuracy of segmented gestures, a bidirectional LSTM network was utilized to model the temporal information, with both the past frames and the future ones taken into account. Finally, a continuous gesture dataset collected indoors for specific application was introduced to validate the proposed method. On this challenge dataset, the 3D DenseNet-LSTM model achieves real-time early detection and classification tasks on unsegmented gesture sequences, and the 3D DenseNet-BiLSTM not only achieves an accuracy of 92.06% on segmented gestures, but also a classification accuracy of 89.8% and 99.7% on nvGesture and SKIG public datasets, respectively. The experimental results demonstrate the performance advantages of the detection and classification as well as the real-time response speed.
资助项目	National Natural Science Foundation of China[61731001] ; Natural Science Foundation of Zhejiang Province[LY21E050017]
WOS关键词	FUSION ; ROBUST
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	SPRINGER
WOS记录号	WOS:001030536100003
资助机构	National Natural Science Foundation of China ; Natural Science Foundation of Zhejiang Province
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/53726]
专题	复杂系统认知与决策实验室
通讯作者	Lu, Zhi
作者单位	1.China Jiliang Univ, Coll Metrol & Measurement Engn, Hangzhou 310018, Zhejiang, Peoples R China 2.Beihang Univ, Sch Automat Sci & Elect Engn, 37 Xueyuan Rd, Beijing 100191, Peoples R China 3.Dongguan Univ Technol, Sch Elect Engn & Intelligentizat, Dongguan 523808, Peoples R China 4.Chinese Acad Sci, Inst Automat, 95 Zhongguancun East Rd, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Lu, Zhi,Qin, Shiyin,Lv, Pin,et al. Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2023:38.
APA	Lu, Zhi,Qin, Shiyin,Lv, Pin,Sun, Liguo,&Tang, Bo.(2023).Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM.MULTIMEDIA TOOLS AND APPLICATIONS,38.
MLA	Lu, Zhi,et al."Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM".MULTIMEDIA TOOLS AND APPLICATIONS (2023):38.