[five minute paper] reinforcement learning based on parameterized action space

• Thesis title ：Reinforcement Learning with Parameterized Actions

The problem solved ？

background

Parameterized action space means that a discrete action has a vectorized parameter . At every decision step , An agent needs to decide which action to perform , And which parameter does this action take to execute .

The method adopted ？

Put forward Q-PAMDP Algorithm , Learn discrete actions and continuous actions alternately . So for in state $s$ The probability of selecting a parameterized action can be expressed as $\pi (a,x|s)$. The choice of discrete actions can be expressed as ${\pi }^d(a|s)$, The choice of action parameters can be expressed as ${\pi }^a(x|s)$, The probability of the whole strategy can be expressed as ：

$$\pi (a,x|s)={\pi }^d(a|s){\pi }^a(x|s)$$

Select the strategy parameters of discrete actions with $w$ Express , Then for ${\pi }_w^d(a|s)$, Parameterized action strategies are represented by a set of parameters $\theta$, Defined as ${\pi }_{\theta }^a(x|s)$, This parameterized set can be expressed as $\theta =[{\theta }_{a_1},\cdots ,{\theta }_{a_k}]$.

Want to optimize parameters , The first way is to optimize directly $\theta$ and $w$ Two parameters ：

$$J(\theta ,\omega )={\mathbb{E}}_{s_0\sim D}\left[V^{{\pi }_{\Theta }}\left(s_0\right)\right]$$

The second way is to update the two alternately , Fix $\theta$ Can optimize $w$ Parameters ：

$$W(\theta )=\arg \underset{\omega }{\max }J(\theta ,\omega )={\omega }_{\theta }^{\ast }$$

Later fixed $w$ Optimize $\theta$ Parameters ：

$$\begin{array}{rl}{J}_{\omega }(\theta )& =J(\theta ,\omega )\\ H(\theta )& =J(\theta ,W(\theta ))\end{array}$$

The pseudo code of the algorithm is ：

The author also provides a theoretical analysis to prove , If you need it later, make it up .

The result is ？

Published information ？ The author information ？

Reference link

Related papers