Dissecting “Reinforcement Studying” by Richard S. Sutton with customized Python implementations, Episode V
In our earlier publish, we wrapped up the introductory collection on elementary reinforcement studying (RL) strategies by exploring Temporal-Distinction (TD) studying. TD strategies merge the strengths of Dynamic Programming (DP) and Monte Carlo (MC) strategies, leveraging their greatest options to type a few of the most necessary RL algorithms, equivalent to Q-learning.
Constructing on that basis, this publish delves into n-step TD studying, a flexible strategy launched in Chapter 7 of Sutton’s guide [1]. This methodology bridges the hole between classical TD and MC strategies. Like TD, n-step strategies use bootstrapping (leveraging prior estimates), however in addition they incorporate the subsequent n rewards, providing a novel mix of short-term and long-term studying. In a future publish, we’ll generalize this idea even additional with eligibility traces.
We’ll comply with a structured strategy, beginning with the prediction downside earlier than transferring to management. Alongside the best way, we’ll:
- Introduce n-step Sarsa,
- Prolong it to off-policy studying,
- Discover the n-step tree backup algorithm, and
- Current a unifying perspective with n-step Q(σ).
As at all times, you could find all accompanying code on GitHub. Let’s dive in!
