Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility

doi:10.1109/JAS.2023.123624

CORC > 自动化研究所 > 中国科学院自动化研究所 > 学术期刊 > IEEE/CAA Journal of Automatica Sinica

	Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility
	Syed Ali Asad Rizvi; Amanda J. Pertzborn; Zongli Lin
刊名	IEEE/CAA Journal of Automatica Sinica
	2023
卷号	10 期号:8 页码:1704-1715
关键词	HVAC control optimal tracking Q-learning reinforcement learning (RL)
ISSN号	2329-9266
DOI	10.1109/JAS.2023.123624
英文摘要	We present the development of a bias compensating reinforcement learning (RL) algorithm that optimizes thermal comfort (by minimizing tracking error) and control utilization (by penalizing setpoint deviations) in a multi-zone heating, ventilation, and air-conditioning (HVAC) lab facility subject to unmeasurable disturbances and unknown dynamics. It is shown that the presence of unmeasurable disturbance results in an inconsistent learning equation in traditional RL controllers leading to parameter estimation bias (even with integral action support), and in the extreme case, the divergence of the learning algorithm. We demonstrate this issue by applying the popular Q-learning algorithm to linear quadratic regulation (LQR) of a multi-zone HVAC environment and showing that, even with integral support, the algorithm exhibits bias issue during the learning phase when the HVAC disturbance is unmeasurable due to unknown heat gains, occupancy variations, light sources, and outside weather changes. To address this difficulty, we present a bias compensating learning equation that learns a lumped bias term as a result of disturbances (and possibly other sources) in conjunction with the optimal control parameters. Experimental results show that the proposed scheme not only recovers the bias-free optimal control parameters but it does so without explicitly learning the dynamic model or estimating the disturbances, demonstrating the effectiveness of the algorithm in addressing the above challenges.
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/52333]
专题	自动化研究所_学术期刊_IEEE/CAA Journal of Automatica Sinica
推荐引用方式 GB/T 7714	Syed Ali Asad Rizvi,Amanda J. Pertzborn,Zongli Lin. Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility[J]. IEEE/CAA Journal of Automatica Sinica,2023,10(8):1704-1715.
APA	Syed Ali Asad Rizvi,Amanda J. Pertzborn,&Zongli Lin.(2023).Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility.IEEE/CAA Journal of Automatica Sinica,10(8),1704-1715.
MLA	Syed Ali Asad Rizvi,et al."Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility".IEEE/CAA Journal of Automatica Sinica 10.8(2023):1704-1715.