Double dqn notes

double-dqn yes dqn The more famous variety , One of the main problems he solved was before DQN In the formula , We use the network to estimate Q Value ratio Q The actual value of the value is much higher , That is, the famous over estimation problem .
First, let's talk about what is overestimation , Those who have read Zhang Sijun's great God will know
https://zhuanlan.zhihu.com/p/109498587
Q The most fidelity value , That is, his definition , It should be the expectation of all possible states following the direct step triggered by this action , But we can't say that we can't solve this problem until all the states have been iterated Q value .
So in Q-Learning We chose the next stage state Maximum Q Value to update Q surface , This is why estimates have been made .
In other words, this overestimation is actually Q-Learning Bring it ,double-dqn What you do is that it uses target Come and ask for Q value , But the action of this choice cannot be Q-target to , He is given by the network that is learning in real time Q The most valuable action , use Q-Target give Q value , The advantage of this is that after iterative learning ,Q-target and Q- It is estimated that the selected action is not the same action , Then the problem of overestimation is reduced in a certain probability .