Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework

doi:10.1109/TG.2022.3207068

CORC > 自动化研究所 > 中国科学院自动化研究所 > 复杂系统认知与决策实验室

	Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework
	Liu, Boyin 1,2; Pu, Zhiqiang 1,2; Zhang, Tianle 1,2; Wang, Huimu 3; Yi, Jianqiang 1,2; Mi, Jiachen 4
刊名	IEEE TRANSACTIONS ON GAMES
	2023-12-01
卷号	15 期号:4 页码:648-657
关键词	Deformable convolution football analysis pitch control reinforcement learning
ISSN号	2475-1502
DOI	10.1109/TG.2022.3207068
通讯作者	Pu, Zhiqiang(zhiqiang.pu@ia.ac.cn)
英文摘要	Applying deep reinforcement learning to football games has recently received extensive attention. However, this remains challenging due to the excessively high complexity of the football environment, such as high-dynamical game states, sparse rewards, and multiple roles with different capabilities. Existing works aim to address these problems without considering abundant domain knowledge of football. In this article, a football knowledge-embedded learning framework is proposed. Specifically, the pitch control concept is innovatively introduced to design a knowledge-embedded state representation. As a result, a novel pitch control model is designed that quantitatively provides space influence values of a single player, the whole team, and the ball. Different from existing models, this model additionally considers each player's various capabilities, including flexibility, explosive force, and stamina. Furthermore, the deformable convolution network is adopted for state representation extracting, which is used to process the geometric transformation of the players' positions and spatial influence values generated by the pitch control model. Then, based on this comprehensive state representation, a proximal policy optimization-based reinforcement learning scheme is adopted to generate the final policy. Finally, extensive simulations, including learning against a fixed opponent and learning from self-play, clearly show the effectiveness and adaptability of our proposed framework.
资助项目	National Key Research, Development Program of China
WOS关键词	GAME
WOS研究方向	Computer Science
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:001128375200008
资助机构	National Key Research, Development Program of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/54889]
专题	复杂系统认知与决策实验室
通讯作者	Pu, Zhiqiang
作者单位	1.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China 3.JD COM, Beijing 100176, Peoples R China 4.Beijing Technol & Business Univ, Sch Artificial Intelligence, Beijing 102401, Peoples R China
推荐引用方式 GB/T 7714	Liu, Boyin,Pu, Zhiqiang,Zhang, Tianle,et al. Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework[J]. IEEE TRANSACTIONS ON GAMES,2023,15(4):648-657.
APA	Liu, Boyin,Pu, Zhiqiang,Zhang, Tianle,Wang, Huimu,Yi, Jianqiang,&Mi, Jiachen.(2023).Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework.IEEE TRANSACTIONS ON GAMES,15(4),648-657.
MLA	Liu, Boyin,et al."Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework".IEEE TRANSACTIONS ON GAMES 15.4(2023):648-657.