Exercises in Chapter II of intensive learning

1. Markov properties （Markov Property)： The next state of a state is only related to the current state , It has nothing to do with the past

2. Markov chain （Markov Chain): Stochastic processes with Markov properties and existing in discrete exponential sets and state spaces

3. State transition matrix ： Each line describes the probability of reaching all other nodes from one node

4. Markov reward process （Markov Reward) A reward function is added to the Markov chain

5. horizon: Defines the same episode Or the length of the whole track , Determined by a finite number of steps

6. return : Discount the reward , Then get the corresponding income

7. Ballman Equation:( Behrman's equation ）：

8. Monte Carlo Algorithm( The monte carlo method ）： Calculate the value of the value function

9. Iterative Algorithm( Dynamic programming method ）： By iterating over the corresponding
   Bellman Equation, Finally, make it converge , When the last updated state does not change much from the previous state ： Update stopped

10. Q function ：（active-value function)

11. Behrman equation in matrix form is difficult to solve

12. Calculate the Behrman equation ：
    Monte Carlo ： When you get MRP, Let it put the boat in , Let him drift with the tide , Then a trajectory is generated , After generating a track , Get a reward , And then put it's discouneted The reward is calculated directly , Calculate it and accumulate , When accumulated to a certain number of tracks , Divide directly by this trajectory , Get its value
    Dynamic programming ：
    Combination of the two

13. Markov reward process （MRP） And Markov decision process （MDP） difference ：
    MDP： More Decision, That's one more action, There is also one more state transition condition
    There is a transformation relationship between the two ：MDP+policy=MRP

14. Looking for the best policy Method ：
    Get the best value function – For this Q Function maximization – Get the best function
    – Directly in this Q Take one from the function and let this Action Maximum value – Take out its best directly policy

Method ：
Exhaustive method （ Generally do not use ）
policy iteration: Optimize policy- Take out the value function – calculate Q function - Maximize
value iteration: Keep iterating Bellman Optimality Equation