Reinforcement Learning

Reinforcement learning (RL) is a problem formulation for learning to make decisions by trial and error. An agent interacts with an environment, and no one tells it which action is right. The agent acts, sees what happens, and receives a reward, and over time it learns to choose the actions that collect the most reward.

These notes build the core ideas step by step, from multi-armed bandits to policy gradients. Along the way you can pull bandit arms, step through grid worlds, and check your understanding with a short quiz at the end of every chapter.

Start with chapter 1 →Try the Grid World demo

Learning path

The eight chapters follow Reinforcement Learning: An Introduction by Sutton and Barto (S&B), and each card shows the book chapter it covers. Each chapter builds on the previous one, so reading them in order works best.

Foundations

What an RL problem is, and how to write one down.

1. 1
   Chapter 1: Introduction & Multi-armed BanditsInteractiveS&B Ch. 2

   How RL differs from supervised learning, and the exploration–exploitation trade-off in its simplest setting, a slot machine with k arms.

   • Action values
   • ε-greedy
   • Optimistic initial values
   • UCB
2. 2
   Chapter 2: Markov Decision ProcessesInteractiveS&B Ch. 3

   Formalizes sequential decisions with states, actions, rewards and returns, then defines policies, value functions and the Bellman equations.

   • Returns & discounting
   • Value functions
   • Bellman equations
   • Optimality

Tabular methods

Every state gets its own entry in a table.

1. 3
   Chapter 3: Dynamic ProgrammingInteractiveS&B Ch. 4

   Computes optimal policies when the environment's dynamics are fully known.

   • Policy evaluation
   • Policy iteration
   • Value iteration
   • GPI
2. 4
   Chapter 4: Monte Carlo MethodsS&B Ch. 5

   Learns values from complete episodes of experience, without a model of the environment.

   • First-visit MC
   • Exploring starts
   • ε-soft policies
   • Importance sampling
3. 5
   Chapter 5: Temporal-Difference LearningS&B Ch. 6

   Updates after every step by bootstrapping from its own estimates, combining ideas from Monte Carlo and DP.

   • TD(0)
   • Sarsa
   • Q-learning
   • Expected Sarsa
   • Double Q-learning
4. 6
   Chapter 6: Planning and LearningS&B Ch. 8

   Learns a model of the environment and plans with it, mixing simulated and real experience.

   • Dyna-Q
   • Prioritized sweeping
   • Trajectory sampling
   • MCTS

Approximate methods

Generalize to states the agent has never seen.

1. 7
   Chapter 7: Function ApproximationS&B Ch. 9–11

   Replaces the table with a parameterized function, so learning scales to large and continuous state spaces.

   • Tile coding
   • Semi-gradient TD
   • Semi-gradient Sarsa
   • Average reward

   • 7.1 On-policy prediction
   • 7.2 On-policy control
   • 7.3 Off-policy methods
2. 8
   Chapter 8: Policy Gradient MethodsS&B Ch. 13

   Learns a parameterized policy directly, instead of deriving it from action values.

   • Policy gradient theorem
   • REINFORCE
   • Baselines
   • Actor-critic

Beyond the fundamentals

• Next trackDeep Reinforcement Learning

  From DQN onward, with neural networks as function approximators. A reading list for now; notes in progress.

• DemoExploring a Grid World

  Drive a robot vacuum to the dust yourself, then see how long a random walk takes to find it.

• DemoGoRight

  A hallway where the reward depends on timing. Compare a random walk with the optimal agent.

Before you start

You don't need prior RL experience, but these help:

• Basic probability, covering random variables, probability distributions and expected values. Value functions are expectations, so this matters from chapter 2 onward.
• Familiarity with gradients, since from chapter 7 on, methods learn by gradient descent on a vector of weights.

You don't need to write any code, every interactive example runs in your browser.

How to use these notes

Each chapter opens with a short list of what it covers and ends with a recap quiz and a pointer to the next chapter. Chapters marked Interactive also have examples you can play with. The notes use the book's notation, so you can read them alongside Sutton & Barto or on their own as a refresher.

About these notes

These notes are meant as an additional resource for students, not a replacement for the courses or the book. They summarize the material from the courses I took:

• the Reinforcement Learning Specialization on Coursera, taught by Adam White and Martha White;
• CMPUT 655: Reinforcement Learning I at the University of Alberta, taught by Professors Michael Bowling and Simone Parisi;
• CMPUT 628: Deep Reinforcement Learning at the University of Alberta, taught by Professor Marlos C. Machado;
• CMPUT 365: Introduction to Reinforcement Learning at the University of Alberta, for which I have served as a teaching assistant under Professor Machado for 3 terms.

References

General references for RL as a whole.

Books and surveys

• Sutton, R. S., & Barto, A. G. (2018)
  Reinforcement Learning: An Introduction (2nd Edition). The main reference for these notes.
• Szepesvári, C. (2010)
  Algorithms for Reinforcement Learning. A short, more mathematical treatment of the same core algorithms.
• Zhao, S. (2025)
  Mathematical Foundations of Reinforcement Learning. A gentle, math-first book with a free English lecture series.

Courses and lecture series

• University of Alberta - Reinforcement Learning MOOC
  Coursera: Reinforcement Learning Specialization by Professors Adam and Martha White.
• DeepMind x UCL - Introduction to Reinforcement Learning (2015)
  YouTube Lecture Series by Professor David Silver.
• DeepMind x UCL - Reinforcement Learning Lecture Series (2021)
  YouTube Lecture Series by Hado van Hasselt, Diana Borsa and Matteo Hessel.
• Stanford - CS 234: Reinforcement Learning (2024)
  YouTube Lecture Series by Professor Emma Brunskill.
• University of Alberta - CMPUT 653: Theoretical Foundations of Reinforcement Learning (2022)
  Lecture notes and videos by Professor Csaba Szepesvári.

Deep RL

• Foundations of Deep RL (2021)
  YouTube Video Series by Professor Pieter Abbeel.
• UC Berkeley - CS 285: Deep Reinforcement Learning (2023)
  YouTube Lecture Series by Professor Sergey Levine.
• OpenAI - Spinning Up in Deep RL
  Online guide with an introduction to the key ideas, algorithm write-ups and reference code.