Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

doi:10.1088/1741-2552/acbfdf

CORC > 自动化研究所 > 中国科学院自动化研究所 > 类脑智能研究中心 > 神经计算及脑机交互

	Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion
	Zhang,Yukun 1,2; Qiu,Shuang 1,2; He,Huiguang 1,2
刊名	Journal of Neural Engineering
	2023-03-13
卷号	20 期号:2
关键词	brain-computer interface motor imagery multimodal EEG-fNIRS center loss
ISSN号	1741-2560
DOI	10.1088/1741-2552/acbfdf
通讯作者	Qiu,Shuang() ; He,Huiguang()
英文摘要	Abstract Objective. A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. Multimodal MI-BCI that uses multiple neural signals contains rich common and complementary information and is promising for enhancing the decoding accuracy of MI-BCI. However, the heterogeneity of different modalities makes the multimodal decoding task difficult. How to effectively utilize multimodal information remains to be further studied. Approach. In this study, a multimodal MI decoding neural network was proposed. Spatial feature alignment losses were designed to enhance the feature representations extracted from the heterogeneous data and guide the fusion of features from different modalities. An attention-based modality fusion module was built to align and fuse the features in the temporal dimension. To evaluate the proposed decoding method, a five-class MI electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) dataset were constructed. Main results and significance. The comparison experimental results showed that the proposed decoding method achieved higher decoding accuracy than the compared methods on both the self-collected dataset and a public dataset. The ablation results verified the effectiveness of each part of the proposed method. Feature distribution visualization results showed that the proposed losses enhance the feature representation of EEG and fNIRS modalities. The proposed method based on EEG and fNIRS modalities has significant potential for improving decoding performance of MI tasks.
语种	英语
出版者	IOP Publishing
WOS记录号	IOP:JNE_20_2_026009
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/51824]
专题	类脑智能研究中心_神经计算及脑机交互
通讯作者	Qiu,Shuang; He,Huiguang
作者单位	1.Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, People’s Republic of China 2.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, People’s Republic of China
推荐引用方式 GB/T 7714	Zhang,Yukun,Qiu,Shuang,He,Huiguang. Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion[J]. Journal of Neural Engineering,2023,20(2).
APA	Zhang,Yukun,Qiu,Shuang,&He,Huiguang.(2023).Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion.Journal of Neural Engineering,20(2).
MLA	Zhang,Yukun,et al."Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion".Journal of Neural Engineering 20.2(2023).