Stochastic Conjugate Gradient Algorithm With Variance Reduction

doi:10.1109/TNNLS.2018.2868835

	Stochastic Conjugate Gradient Algorithm With Variance Reduction
	Jin, Xiao-Bo 1; Zhang, Xu-Yao2 ; Huang, Kaizhu 3; Geng, Guang-Gang 4
刊名	IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
	2019-05-01
卷号	30 期号:5 页码:1360-1369
关键词	Computational efficiency covariance reduction empirical risk minimization (ERM) linear convergence stochastic conjugate gradient (CG)
ISSN号	2162-237X
DOI	10.1109/TNNLS.2018.2868835
通讯作者	Geng, Guang-Gang(gengguanggang@cnnic.cn)
英文摘要	Conjugate gradient (CG) methods are a class of important methods for solving linear equations and nonlinear optimization problems. In this paper, we propose a new stochastic CG algorithm with variance reduction(1) and we prove its linear convergence with the Fletcher and Reeves method for strongly convex and smooth functions. We experimentally demonstrate that the CG with variance reduction algorithm converges faster than its counterparts for four learning models, which may be convex, nonconvex or nonsmooth. In addition, its area under the curve performance on six large-scale data sets is comparable to that of the LIBLINEAR solver for the L2-regularized L2-loss but with a significant improvement in computational efficiency.
资助项目	Fundamental Research Funds for the Henan Provincial Colleges and Universities in Henan University of Technology[2016RCJH06] ; National Natural Science Foundation of China[61103138] ; National Natural Science Foundation of China[61602154] ; National Natural Science Foundation of China[61375039] ; National Natural Science Foundation of China[61473236] ; National Natural Science Foundation of China[U1804326] ; National Key Research & Development Program[2016YFD0400104-5] ; National Basic Research Program of China[2012CB316301]
WOS关键词	GLOBAL CONVERGENCE
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000466192100007
资助机构	Fundamental Research Funds for the Henan Provincial Colleges and Universities in Henan University of Technology ; National Natural Science Foundation of China ; National Key Research & Development Program ; National Basic Research Program of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/24223]
专题	中国科学院自动化研究所
通讯作者	Geng, Guang-Gang
作者单位	1.Henan Univ Technol, Dept Informat Sci & Engn, Zhengzhou 450001, Henan, Peoples R China 2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 3.Xian Jiaotong Liverpool Univ, Dept Elect & Elect Engn, Suzhou 215123, Peoples R China 4.Chinese Acad Sci, Comp Network Informat Ctr, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Jin, Xiao-Bo,Zhang, Xu-Yao,Huang, Kaizhu,et al. Stochastic Conjugate Gradient Algorithm With Variance Reduction[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2019,30(5):1360-1369.
APA	Jin, Xiao-Bo,Zhang, Xu-Yao,Huang, Kaizhu,&Geng, Guang-Gang.(2019).Stochastic Conjugate Gradient Algorithm With Variance Reduction.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,30(5),1360-1369.
MLA	Jin, Xiao-Bo,et al."Stochastic Conjugate Gradient Algorithm With Variance Reduction".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 30.5(2019):1360-1369.