Knowledge-Embedded Mutual Guidance for Visual Reasoning

doi:10.1109/TCYB.2023.3310892

CORC > 自动化研究所 > 中国科学院自动化研究所 > 多模态人工智能系统全国重点实验室

	Knowledge-Embedded Mutual Guidance for Visual Reasoning
	Zheng, Wenbo 3,7; Yan, Lan 2,6; Chen, Long 1,4; Li, Qiang 5; Wang, Fei-Yue 5
刊名	IEEE TRANSACTIONS ON CYBERNETICS
	2023-09-20
页码	13
关键词	Attention model joint learning knowledge embedding visual reasoning
ISSN号	2168-2267
DOI	10.1109/TCYB.2023.3310892
通讯作者	Zheng, Wenbo(zwb2022@whut.edu.cn)
英文摘要	Visual reasoning between visual images and natural language is a long-standing challenge in computer vision. Most of the methods aim to look for answers to questions only on the basis of the analysis of the offered questions and images. Other approaches treat knowledge graphs as flattened tables to search for the answer. However, there are two major problems with these works: 1) the model disregards the fact that the world we surrounding us interlinks our hearing and speaking of natural language and 2) the model largely ignores the structure of the KG. To overcome these challenging deficiencies, a model should jointly consider two modalities of vision and language, as well as the rich structural and logical information embedded in knowledge graphs. To this end, we propose a general joint representation learning framework for visual reasoning, namely, knowledge-embedded mutual guidance. It realizes mutual guidance not only between visual data and natural language descriptions but also between knowledge graphs and reasoning models. In addition, it exploits the knowledge derived from the reasoning model to boost knowledge graphs when applying the visual relation detection task. The experimental results demonstrate that the proposed approach performs dramatically better than state-of-the-art methods on two benchmarks for visual reasoning.
资助项目	Natural Science Foundation of China[62303361] ; Natural Science Foundation of China[62302161] ; Natural Science Foundation of China[U1811463] ; Hainan Provincial Natural Science Foundation of China[623QN266] ; Fundamental Research Funds for the Central Universities[WUT: 233110002] ; National Key Research and Development Program of China[2018AAA0101502]
WOS研究方向	Automation & Control Systems ; Computer Science
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:001071913500001
资助机构	Natural Science Foundation of China ; Hainan Provincial Natural Science Foundation of China ; Fundamental Research Funds for the Central Universities ; National Key Research and Development Program of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/53140]
专题	多模态人工智能系统全国重点实验室
通讯作者	Zheng, Wenbo
作者单位	1.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100864, Peoples R China 2.Natl Supercomp Ctr Changsha, Changsha 410082, Hunan, Peoples R China 3.Wuhan Univ Technol, Sch Comp Sci & Artificial Intelligence, Wuhan 430070, Peoples R China 4.Waytous Inc, Beijing 100864, Peoples R China 5.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China 6.Hunan Univ, Coll Comp Sci & Engn, Changsha 410082, Peoples R China 7.Wuhan Univ Technol, Sanya Sci & Educ Innovat Pk, Sanya 572000, Peoples R China
推荐引用方式 GB/T 7714	Zheng, Wenbo,Yan, Lan,Chen, Long,et al. Knowledge-Embedded Mutual Guidance for Visual Reasoning[J]. IEEE TRANSACTIONS ON CYBERNETICS,2023:13.
APA	Zheng, Wenbo,Yan, Lan,Chen, Long,Li, Qiang,&Wang, Fei-Yue.(2023).Knowledge-Embedded Mutual Guidance for Visual Reasoning.IEEE TRANSACTIONS ON CYBERNETICS,13.
MLA	Zheng, Wenbo,et al."Knowledge-Embedded Mutual Guidance for Visual Reasoning".IEEE TRANSACTIONS ON CYBERNETICS (2023):13.