2、TD+Learning

Discounted Return

Sarsa

TD Algorithm , Used to learn the action value function QΠ

Sarsa：Tabular Version

Sarsa’s Name

Table status Sarsa Applicable to less States and actions , As the state and action increase , It is difficult to learn when the table is enlarged

Sarsa：Neural Network Version

Q-Learning

TD Algorithm , Learn the optimal action Algorithm

Sarsa And Q-Learning

Derive TD Target

Q-Learning(tabular version)

Q-Learning(DQN Version)

Multi-Setp TD Target

• Using One Reward
  
• Using Multiple Rewards
  
  
  
  
  

Value playback （Revisiting DQN and TD Learning）

• Shortcoming 1：Waste of Experience

• Shortcoming2：Correlated Updates
  
• Experience playback

• History

Prioritized Experience Replay

On the left is a common scene of Mario , On the right is boos Off scene , Relative to the left , The right side is more rare , Therefore, we should increase the weight of the scene on the right ,TD error The bigger it is , Then the more important the scene is


The learning rate of random gradient descent should be adjusted according to the importance of sampling



Of a sample TD The bigger it is , Then the greater the sampling weight , The lower the learning rate

Overestimation problem

Bootstrapping： Bootstrap problem , Pull your shoes and lift yourself up
Similar to the method of stepping on the right foot with the left foot , It doesn't exist in reality , There exist in reinforcement learning

Problem of Overestimation

• Reason 1:Maximization
  
  
  
• Reason 2:Bootstrapping
  
• Why does overestimation happen
  

• Why overestimation is a shortcoming
  
  
  
  
• Solutions
  

Target Network

TD Learning with Target Network

Update Target Network
Comparisons

Target Network Although a little better , But we still cannot get rid of the problem of overestimation

Double DQN

• Naive Update
  

• Using Target Network
  

• Double DQN
  

• Why does Double DQN work better
  

Dueling Network

Advantage Function（ Dominance function ）

• Value Functions
  

• Optimal Value Functions
  
  Properties of Advantage Function
  
  

Dueling Network

Revisiting DQN

Approximating Advantage Function

Approximating State-Value Function

Dueling Network:Formulation



Blue plus red and then subtract the maximum value of red to get purple finally Dueling Network Output

Problem of Non-identifiability