Researched
Dynamic Programming (Bellman)
Richard Bellman's method of the 1950s solves a long chain of decisions by working out the best choice for each smaller problem first; it is the mathematical core of reinforcement learning.
Open in the interactive tree →Bellman developed dynamic programming at the RAND Corporation in the early 1950s and set it out in his 1957 book Dynamic Programming. Its key idea, now called the Bellman equation, writes the value of a decision as the reward now plus the value of the best decision afterwards. In 1957 he also introduced Markov decision processes, choices made under chance, and Ronald Howard added policy iteration in 1960. Reinforcement-learning programs, from TD-Gammon onwards, learn approximations of exactly this value function.
Prerequisites
- Lagrangian & Hamiltonian Mechanics1788-1833Bellman extended the 19th-century Hamilton-Jacobi theory.
- Markov Chains1906Markov decision processes add choices to Markov chains.
Unlocks
- Reinforcement Learning (TD-Gammon)1992Temporal-difference learning approximates the Bellman value function.