Reinforcement Learning of Large Language Models

Reinforcement Learning of Large Language Models, Spring 2025

Lecture slides

Chapter 0: Prologue.
Chapter 1: Deep Reinforcement learning.
Chapter 2: Large Language Models.
Chapter 3: Reinforcement Learning of Large Language Models.

Lecture videos

Chapter 0: Prologue.
Chapter 1.1: MDP foundations, imitation learning, and value iteration.
Chapter 1.2: Deep policy evaluation.
Chapter 1.3: Deep policy gradient methods (A3C).
Chapter 1.4: Deep policy gradient methods (PPO, GRPO).
Chapter 1.5: AlphaGo, test-time compute, and expert iteration.
Chapter 2.1: NLP foundations, language modeling, RNNs.
Chapter 2.2: Transformers I (BERT, GPT-1).
Chapter 2.3: Transformers II (modern transformers updates and sampling methods).
Chapter 2.4: In-context learning and instruction fine-tuning.
Chapter 3.1: Reinforcement learning from human feedback (PPO, DPO).
Chapter 3.2: Reinforcement learning with verifiable rewards (RLVR).

Homework exercises

Code.
Homework exercises.

Course Information

Instructor

Ernest K. Ryu
Assistant Professor of Mathematics, UCLA,

Photo of Ernest Ryu

Prerequisites

Students are expected to have basic familiarity with deep learning at the level of image classification. No prior experience with reinforcement learning (RL) or large language models (LLMs) is assumed. For the deep RL lectures, students should be familiar with conditional expectations and the tower property (law of total expectation).