Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends

doi:10.1109/ACCESS.2019.2949611

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends
	Cui, Guoxin 1,2; Guo, Jiafeng 1; Fan, Yixing 1; Lan, Yanyan 1; Cheng, Xueqi 1
刊名	IEEE ACCESS
	2019
卷号	7 页码:156848-156859
关键词	Training Market research Acceleration Convergence Servers Stochastic processes Machine learning Parameter trend asynchronous SGD accelerate training
ISSN号	2169-3536
DOI	10.1109/ACCESS.2019.2949611
英文摘要	Stochastic gradient descent(SGD) is the fundamental sequential method in training large scale machine learning models. To accelerate the training process, researchers proposed to use the asynchronous stochastic gradient descent (A-SGD) method in model learning. However, due to the stale information when updating parameters, A-SGD converges more slowly than SGD in the same iteration number. Moreover, A-SGD often converges to a high loss value and results in lower model accuracy. In this paper, we propose a novel algorithm called Trend-Smooth which can be adapted to the asynchronous parallel environment to overcome the above problems. Specifically, Trend-Smooth makes use of the parameter trend during the training process to shrink the learning rate of some dimensions where the gradients directions are opposite to the trends of parameters. Experiments on MNIST and CIFAR-10 datasets confirm that Trend-Smooth can accelerate the convergence speed in asynchronous training process. The test accuracy that Trend-Smooth achieves is shown to be higher than other asynchronous parallel baseline methods, and is very close to the SGD method. Moreover, Trend-Smooth can also be combined with other adaptive learning rate methods(like Momentum, RMSProp and Adam) in the asynchronous parallel environment to promote their performance.
资助项目	National Natural Science Foundation of China (NSFC)[61425016] ; National Natural Science Foundation of China (NSFC)[61722211] ; National Natural Science Foundation of China (NSFC)[61773362] ; National Natural Science Foundation of China (NSFC)[61872338] ; Youth Innovation Promotion Association CAS[20144310] ; Youth Innovation Promotion Association CAS[2016102] ; National Key Research and Development Program of China[2016QY02D0405] ; Foundation and Frontier Research Key Program of Chongqing Science and Technology Commission[cstc2017jcyjBX0059]
WOS研究方向	Computer Science ; Engineering ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000497165400120
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/14954]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Cui, Guoxin; Guo, Jiafeng
作者单位	1.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Cui, Guoxin,Guo, Jiafeng,Fan, Yixing,et al. Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends[J]. IEEE ACCESS,2019,7:156848-156859.
APA	Cui, Guoxin,Guo, Jiafeng,Fan, Yixing,Lan, Yanyan,&Cheng, Xueqi.(2019).Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends.IEEE ACCESS,7,156848-156859.
MLA	Cui, Guoxin,et al."Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends".IEEE ACCESS 7(2019):156848-156859.