Paper notes: universal value function approvers

PMLR 2015

1 Introduce

         This article paper Put forward UVFA（universal value function approximators), That's according to a state（ Other value function Some parts ） and goal（ Other value function The part that doesn't have ） To estimate the expected return

          Study UVFA The challenge is , Generally speaking agent Only a small part will be seen (s,g) Combine , It's impossible to traverse all the state-goal Yes . If we use supervised learning to train , It is also likely that the data volume is insufficient and the fitting is not good , Become a difficult regression problem .

        here UVFA A method similar to matrix decomposition is used , Think of the data as a sparse matrix , Each line is an observed state s, Each column is an observed target g. Then the matrix is decomposed into states embedding Φ(s) And the target embedding φ(g).

        ——> So we can learn from state To Φ(s);goal To φ(g) The nonlinearity of mapping

2 The model part

two-stream architecture You can learn well state and goal The common structure between  

• In many cases ,goal Can be defined as state In the form of /state The combination of ,. thus Φ and φ There should be something to share feature.
  • This paper is in MLP Φ and φ in , The parameters of the previous layers are shared , therefore state and goal The common feature Can be learned
  • ——>partially symmetric architecture
• In some cases ,UVFA It may be symmetrical
  • For example, calculation state s and goal g The distance between UVFA
  • At this point we can make Φ=φ,h Is a symmetric operator （ Like dot product ）
  • ——>symmetric architecture

2.1 Supervised learning UVFA

2.1.1 End to end learning

Through a suitable loss function（ such as MSE )+ Gradient descent realization

2.1.2 two-stage Study

• stage1： take V*(g) Put it in a matrix , Row representation state, Column means goal. Perform matrix decomposition , obtain and 【 chart 1 The right half of the third picture 】
• stage2： take and As ground-truth, Study Φs and φg 【 chart 1 The left half of the third picture 】

 

 

2.2 Reinforcement learning UVFA

Intensive learning , There is no ground-truth V*(g) 了 , We have to find out in some ways Q-value

The article uses a kind of Horde  The way of architecture can produce the corresponding Q-value, That article paper Didn't look , But use bootstriping（TD） Words , The result is similar 【TD Will be slightly unstable 】

 【 Be careful. ： Specifically, this goal How did you get it , The article still doesn't say 】

【 To the first 10 Step ,Q-value After the calculation , It has nothing to do with reinforcement learning , The next few steps are matrix decomposition + Two embedding network Of training】