George Washington University: Hanhan Zhou | PAC: auxiliary value factor decomposition with counterfactual prediction in Multi-Agent Reinforcement Learning

【 title 】PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning

【 The author team 】Hanhan Zhou, Tian Lan, Vaneet Aggarwal

【 Date of publication 】2022.6.22

【 Thesis link 】https://arxiv.org/pdf/2206.11420.pdf

【 Recommended reasons 】 With the development of value function decomposition method , Multi agent reinforcement learning （MARL） Great progress has been made . Due to monotonicity , It allows the optimization of joint action value functions by maximizing the utility of each agent of factorization . This article shows that , Partially observable MARL In question ,agent Ordering its own actions may impose concurrency constraints on representable function classes （ Across different states ）, This leads to significant estimation errors in the training process . Therefore, some suggestions are put forward PAC, A new framework for generating auxiliary information through counterfactual prediction of the best joint action selection , It provides explicit help for Factorization of value function . An information coding method based on variational reasoning is proposed , Used to collect and encode counterfactual predictions from the estimated baseline . To achieve decentralized execution , From the maximum entropy MARL Each agent strategy of factorization is derived from the framework . In multi-agent predators - Bait and a group of starcraft II The proposed approach is evaluated on a micro management task PAC. Research shows that on all benchmarks ,PAC The results are better than the most advanced value based and Policy Based Multi-Agent Reinforcement learning algorithms .