Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution

doi:10.1007/s11263-022-01699-1

	Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution
	Zhou, Man 5,6; Yan, Keyu 5,6; Pan, Jinshan 7; Ren, Wenqi 2; Xie, Qi 3; Cao, Xiangyong 1,4
刊名	INTERNATIONAL JOURNAL OF COMPUTER VISION
	2022-10-15
关键词	Guided image super-resolution Deep unfolding network Persistent memory mechanism Pan-sharpening Depth image super-resolution MR image super-resolution
ISSN号	0920-5691
DOI	10.1007/s11263-022-01699-1
通讯作者	Cao, Xiangyong(caoxiangyong@mail.xjtu.edu.cn)
英文摘要	Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-based methods mainly take the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image, simply ignoring many non-local common characteristics between them. To alleviate this issue, we firstly propose a maximum a posteriori (MAP) estimation model for GISR with two types of priors on the HR target image, i.e., local implicit prior and global implicit prior. The local implicit prior aims to model the complex relationship between the HR target image and the HR guidance image from a local perspective, and the global implicit prior considers the non-local auto-regression property between the two images from a global perspective. Secondly, we design a novel alternating optimization algorithm to solve this model for GISR. The algorithm is in a concise framework that facilitates to be replicated into commonly used deep network structures. Thirdly, to reduce the information loss across iterative stages, the persistent memory mechanism is introduced to augment the information representation by exploiting the Long short-term memory unit (LSTM) in the image and feature spaces. In this way, a deep network with certain interpretation and high representation ability is built. Extensive experimental results validate the superiority of our method on a variety of GISR tasks, including Pan-sharpening, depth image super-resolution, and MR image super-resolution. Code will be released at https://github.com/manman1995/pansharpening.
资助项目	National Key Research and Development Project of China[2021ZD0110700] ; National Natural Science Foundation of China[62272375] ; National Natural Science Foundation of China[61906151] ; National Natural Science Foundation of China[62050194] ; National Natural Science Foundation of China[62037001] ; Innovative Research Group of the National Natural Science Foundation of China[61721002] ; Innovation Research Team of Ministry of Education[IRT_17R86] ; Project of China Knowledge Centre for Engineering Science and Technology ; Project of XJTU Undergraduate Teaching Reform[20JX04Y]
WOS关键词	MULTI-CONTRAST SUPERRESOLUTION ; SPARSE REPRESENTATION ; FUSION ; INTERPOLATION ; ENHANCEMENT ; RECOVERY ; MRI
WOS研究方向	Computer Science
语种	英语
出版者	SPRINGER
WOS记录号	WOS:000869199700001
资助机构	National Key Research and Development Project of China ; National Natural Science Foundation of China ; Innovative Research Group of the National Natural Science Foundation of China ; Innovation Research Team of Ministry of Education ; Project of China Knowledge Centre for Engineering Science and Technology ; Project of XJTU Undergraduate Teaching Reform
内容类型	期刊论文
源URL	[http://ir.hfcas.ac.cn:8080/handle/334002/129317]
专题	中国科学院合肥物质科学研究院
通讯作者	Cao, Xiangyong
作者单位	1.Xi An Jiao Tong Univ, Sch Comp Sci & Technol, Xian, Peoples R China 2.Chinese Acad Sci, Inst Informat Engn, Beijing, Peoples R China 3.Xi An Jiao Tong Univ, Sch Math & Stat, Xian, Peoples R China 4.Xi An Jiao Tong Univ, Minist Educ, Key Lab Intelligent Networks & Network Secur, Xian, Peoples R China 5.Chinese Acad Sci, Hefei Inst Phys Sci, Hefei, Peoples R China 6.Univ Sci & Technol China, Hefei, Peoples R China 7.Nanjing Univ Sci & Technol, Nanjing, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Man,Yan, Keyu,Pan, Jinshan,et al. Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2022.
APA	Zhou, Man,Yan, Keyu,Pan, Jinshan,Ren, Wenqi,Xie, Qi,&Cao, Xiangyong.(2022).Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution.INTERNATIONAL JOURNAL OF COMPUTER VISION.
MLA	Zhou, Man,et al."Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution".INTERNATIONAL JOURNAL OF COMPUTER VISION (2022).