Hierarchical Attention Networks for Fact-based Visual Question Answering

doi:10.1007/s11042-023-16151-w

CORC > 自动化研究所 > 中国科学院自动化研究所 > 多模态人工智能系统全国重点实验室

	Hierarchical Attention Networks for Fact-based Visual Question Answering
	Yao, Haibo 2; Luo, Yongkang 1; Zhang, Zhi 2; Yang, Jianhang 2; Cai, Chengtao 2
刊名	MULTIMEDIA TOOLS AND APPLICATIONS
	2023-07-22
页码	18
关键词	Fact-based Visual Question Answering Hierarchical attention networks Self-attention Multiple attention interaction Positional encoding
ISSN号	1380-7501
DOI	10.1007/s11042-023-16151-w
通讯作者	Zhang, Zhi(zhangzhi1981@hrbeu.edu.cn)
英文摘要	Fact-based Visual Question Answering (FVQA) aims to answer questions with images and facts. It requires a fine-grained and simultaneous understanding visual content, textual questions, and factual knowledge. We propose a novel Hierarchical Attention Network (HANet) for FVQA to address the limitations of existing methods. Most existing FVQA methods only consider external facts as a library of answers, which weakens the role of the external facts, and ignore information from images, questions, and external knowledge. Additionally, they only utilize appearance features of images and disregard position information, which results in a model failing to answer many complex questions, due to the absence of important information in images. Our proposed model considers FVQA as a triple modal interaction task and exploits self-attention and multiple attention interaction to make full use of information from all three modalities. In specific, we introduce three attention modules: Self-Attention Layer, Triple-modal Attention Layer, and Bi-Attention Layer to sufficiently extract useful information from images, questions, facts. Furthermore, we also introduce positional encoding into image embedding acquisition to further improve performance of the model. Our proposed method achieves state-of-the-art performance on the FVQA dataset, with top-3 accuracy of 85.98% and top-1 accuracy of 71.68%.
资助项目	National Key R amp;D Program of China[2019YFE0105400]
WOS关键词	RECOMMENDATION SYSTEM ; GRAPH
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	SPRINGER
WOS记录号	WOS:001034665300003
资助机构	National Key R amp;D Program of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/53897]
专题	多模态人工智能系统全国重点实验室
通讯作者	Zhang, Zhi
作者单位	1.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China 2.Harbin Engn Univ, Coll Intelligent Syst Sci & Engn, Harbin 150001, Peoples R China
推荐引用方式 GB/T 7714	Yao, Haibo,Luo, Yongkang,Zhang, Zhi,et al. Hierarchical Attention Networks for Fact-based Visual Question Answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2023:18.
APA	Yao, Haibo,Luo, Yongkang,Zhang, Zhi,Yang, Jianhang,&Cai, Chengtao.(2023).Hierarchical Attention Networks for Fact-based Visual Question Answering.MULTIMEDIA TOOLS AND APPLICATIONS,18.
MLA	Yao, Haibo,et al."Hierarchical Attention Networks for Fact-based Visual Question Answering".MULTIMEDIA TOOLS AND APPLICATIONS (2023):18.