Object-difference drived graph convolutional networks for visual question answering

doi:10.1007/s11042-020-08790-0

CORC > 自动化研究所 > 中国科学院自动化研究所 > 数字内容技术与服务研究中心 > 远程智能医疗

	Object-difference drived graph convolutional networks for visual question answering
	Zhu, Xi 3,5; Mao, Zhendong 4; Chen, Zhineng 6; Li, Yangyang 2; Wang, Zhaohui 3,5; Wang, Bin 1
刊名	MULTIMEDIA TOOLS AND APPLICATIONS
	2020-03-20
页码	19
关键词	Visual question answering Graph convolutional networks Object-difference
ISSN号	1380-7501
DOI	10.1007/s11042-020-08790-0
通讯作者	Mao, Zhendong(maozhendong2008@gmail.com)
英文摘要	Visual Question Answering(VQA), an important task to evaluate the cross-modal understanding capability of an Artificial Intelligence model, has been a hot research topic in both computer vision and natural language processing communities. Recently, graph-based models have received growing interest in VQA, for its potential of modeling the relationships between objects as well as its formidable interpretability. Nonetheless, those solutions mainly define the similarity between objects as their semantical relationships, while largely ignoring the critical point that the difference between objects can provide more information for establishing the relationship between nodes in the graph. To achieve this, we propose an object-difference based graph learner, which learns question-adaptive semantic relations by calculating inter-object difference under the guidance of questions. With the learned relationships, the input image can be represented as an object graph encoded with structural dependencies between objects. In addition, existing graph-based models leverage the pre-extracted object boxes by the object detection model as node features for convenience, but they are suffering from the redundancy problem. To reduce the redundant objects, we introduce a soft-attention mechanism to magnify the question-related objects. Moreover, we incorporate our object-difference based graph learner into the soft-attention based Graph Convolutional Networks to capture question-specific objects and their interactions for answer prediction. Our experimental results on the VQA 2.0 dataset demonstrate that our model gives significantly better performance than baseline methods.
资助项目	National Key Research and Development Program of China[2016QY03D0505] ; National Natural Science Foundation of China[U19A2057]
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	SPRINGER
WOS记录号	WOS:000521018700001
资助机构	National Key Research and Development Program of China ; National Natural Science Foundation of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/38696]
专题	数字内容技术与服务研究中心_远程智能医疗
通讯作者	Mao, Zhendong
作者单位	1.Xiaomi Inc, Xiaomi AI Lab, Beijing, Peoples R China 2.China Acad Elect & Informat Technol, Beijing, Peoples R China 3.Chinese Acad Sci, Inst Informat Engn, Beijing, Peoples R China 4.Univ Sci & Technol China, Hefei, Peoples R China 5.Univ Chinese Acad Sci, Sch Cyber Secur, Beijing, Peoples R China 6.Chinese Acad Sci, Inst Automat, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Zhu, Xi,Mao, Zhendong,Chen, Zhineng,et al. Object-difference drived graph convolutional networks for visual question answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2020:19.
APA	Zhu, Xi,Mao, Zhendong,Chen, Zhineng,Li, Yangyang,Wang, Zhaohui,&Wang, Bin.(2020).Object-difference drived graph convolutional networks for visual question answering.MULTIMEDIA TOOLS AND APPLICATIONS,19.
MLA	Zhu, Xi,et al."Object-difference drived graph convolutional networks for visual question answering".MULTIMEDIA TOOLS AND APPLICATIONS (2020):19.