Markov Decision Process

$\large S$ $\large A$ $\large R$ and a new state.

$\large S_t$ $\large R_t$ $\large A_t$ $\large t$ .

Agent Environment Diagram

Markov Property

Markov Property $\large S_{t+1}$ $\large S_t$ $\large A_t$ , and not on past states or actions.

\begin{matrix} P {R_{t + 1} = r, S_{t + 1} = s^{'} | S_{0}, A_{0}, R_{1}, . . ., S_{t - 1}, A_{t - 1}, R_{t}, S_{t}, A_{t}} \\ = P {R_{t + 1} = r, S_{t + 1} = s^{'} | S_{t}, A_{t}} \end{matrix}

policy $\pi_t(a | s)$ $\large t$ $\large a$ $\large s$ .

Episodic tasks are ones that have finite steps and a terminal state.

Continuing tasks are ones that do not end or have a terminal state,

$R_t$ return $\large G_t$ is the expression we aim to maximize at any step.

G_{t} = R_{t + 1} + R_{t + 2} + R_{t + 3} + . . . = \sum_{k = 0}^{\infty} R_{t + k + 1}

G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + . . . = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1}

value function $\large s$ $\large \pi$ $\large s$ $\large \pi$ hereafter:

\begin{matrix} v_{π} (s) = E_{π} [G_{t} | S_{t} = s] \\ = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s] \end{matrix}

value function $\large s$ $\large \pi$ $\large s$ $\large a$ $\large \pi$ thereafter:

\begin{matrix} q_{π} (s, a) = E_{π} [G_{t} | S_{t} = s, A_{t} = a] \\ = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s, A_{t} = a] \end{matrix}

\begin{matrix} \begin{aligned} v_{π} (s) & = E_{π} [G_{t} | S_{t} = s] \\ = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s] \\ = E_{π} [R_{t + 1} + γ G_{t + 1} | S_{t} = s] \\ = \sum_{a} π (a | s) \sum_{s^{'}, r} p (s^{'}, r | s, a) [r + γ v_{π} (s^{'})] \end{aligned} \end{matrix}