Improved learning algorithms for mixture of experts in multiclass   classification

doi:10.1016/S0893-6080(99)00043-X

CORC > 北京大学 > 信息科学技术学院

	Improved learning algorithms for mixture of experts in multiclass classification
	Chen, K ; Xu, L ; Chi, H
刊名	neural networks
	1999
关键词	mixture of experts multiclass classification multinomial density generalized Bernoulli density Expectation-Maximization (EM) algorithm Newton-Raphson method iterative reweighted least squares (IRLS) algorithm BFGS algorithm DEPENDENT SPEAKER IDENTIFICATION EM ALGORITHM RECOGNITION HMMS
DOI	10.1016/S0893-6080(99)00043-X
英文摘要	Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.L, Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181-214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton-Raphson method to the inner loop of the EM algorithm in the case of multiclass classification, where the exact Hessian matrix is adopted. To tackle the expensive computation of the Hessian matrix and its inverse, we propose an approximation to the Newton-Raphson algorithm based on a so-called generalized Bernoulli density. The Newton-Raphson algorithm and its approximation have been applied to synthetic data, benchmark, and real-world multiclass classification tasks, For comparison, the IRLS algorithm and a quasi-Newton algorithm called BFGS have also been applied to the same tasks. Simulation results have shown that the use of the proposed learning algorithms avoids the instability problem and makes the ME architecture produce good performance in multiclass classification. In particular, our approximation algorithm leads to fast learning. In addition, the limitation of our approximation algorithm is also empirically investigated in this paper. (C) 1999 Published by Elsevier Science Ltd. All rights reserved.; http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000083333500003&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=8e1609b174ce4e31116a60747a720701 ; Computer Science, Artificial Intelligence; SCI(E); EI; 61; ARTICLE; 9; 1229-1252; 12
语种	英语
内容类型	期刊论文
源URL	[http://ir.pku.edu.cn/handle/20.500.11897/214166]
专题	信息科学技术学院
推荐引用方式 GB/T 7714	Chen, K,Xu, L,Chi, H. Improved learning algorithms for mixture of experts in multiclass classification[J]. neural networks,1999.
APA	Chen, K,Xu, L,&Chi, H.(1999).Improved learning algorithms for mixture of experts in multiclass classification.neural networks.
MLA	Chen, K,et al."Improved learning algorithms for mixture of experts in multiclass classification".neural networks (1999).