Calendar

Lecture Schedule

Tuesday, Aug 29

Building Blocks of Modern LLMs

Slides

Readings: Jurasky and Martin, Chapter 10, Illustrated Transformer

Thursday, Aug 31

Transformer Architecture, LLM Pretraining Methods

Slides 1 Slides 2

Readings: Dai and Le, Radford et al., Lewis et al.

Tuesday, Sep 5

LLM Pretraining Data HW1 OUT

Slides

Readings: ClueWeb22, Documenting C4

Thursday, Sep 7

In-Context Learning, Prompt Engineering, and Alignment

Slides

Readings: GPT-2 paper, Prompt Engineering

Tuesday, Sep 12

PROJECT OUT Interpretability

Slides

Readings: Edge Probing, Loss Landscape

Thursday, Sep 14

Generation-based Automatic Evaluation Methods

Slides

Readings: GEM Benchmark, Evaluation Challenges

Tuesday, Sep 19

Human Evaluation Methods

Slides

Readings: Evaluation of Text Generation: A Survey

Thursday, Sep 21

Parameter-Efficient Tuning Methods

Slides

Readings: Prompt Tuning, PETM blog post

Tuesday, Sep 26

Generation before LLMs, Discuss HW1 HW1 DUE PROJECT PROPOSALS DUE

Slides

Readings: Building NLG Systems, chapter 3, Two Decades of Statistical Language Modeleing

Thursday, Sep 28

In-class Project Proposals HW2 OUT

Proposals

Tuesday, Oct 3

Industry Talk: Together.ai; Chatbots and AI Agents

Slides

Readings: Diplomacy agents, Generative Agents: Interactive Simulacra of Human Behavior

Thursday, Oct 5

LLM for Search Engines

Slides

Readings: ANCE, SentenceT5

Tuesday, Oct 10

MIDTERM

Readings: None

Thursday, Oct 12

LLMs for Music Generation (Guest Lecture from Chris Donahue)

Readings: AudioLM, Oore+ 18

Tuesday, Oct 17

Fall break (no class)

Readings: Enjoy Fall Break

Thursday, Oct 19

Fall break (no class)

Readings: Enjoy Fall Break

Tuesday, Oct 24

Anonymous Feedback for the Class. LLM for Search Engines Continued HW2 DUE

Slides

Readings: Scaling Laws, Chinchilla

Thursday, Oct 26

Paradigmns of Visual Representation Learning (Guest Lecture from Xinlei Chen)

Slides

Readings: SimCLR, MAE

Tuesday, Oct 31

Midpoint Project Presentations PROJECT MIDPOINT DUE

Project Midpoint Slides

Thursday, Nov 2

Scaling Up: Scaling Laws

Slides

Readings: Adam, 8-bit Adam

Tuesday, Nov 7

Democracy day (no class)

Readings: None

Thursday, Nov 9

Industry Talk: Anthropic; Scaling Up: Optimizer Basis

Slides

Readings: GPipe, ZeRO

Tuesday, Nov 14

HW3 OUT Industry Talk: LLaMA2; Scaling Up: Parallel Optimization

Slides

Readings: ZeRO Offload, LLaMA2

Thursday, Nov 16

Retrieval Augmentation

Slides

Readings: RAG, REALM

Tuesday, Nov 21

Efficient Serving (Guest Lecture from Beidi Chen)

Slides

Readings: H2O, StreamLB

Thursday, Nov 23

Happy Thanksgiving (no class)

Readings: Enjoy break

Tuesday, Nov 28

LLMs for Code and Tool Use (Guest Lecture from Daniel Fried)

Readings: TODO

Thursday, Nov 30

HW3 DUE; Industry Talk: Cohere; Legal Considerations (Guest Lecture from A Feder Cooper)

Readings: AI Supply Chain

Tuesday, Dec 5

Ethical Considerations and Model Bias (Guest Lecture from Maarten Sap)

Readings: TODO

Thursday, Dec 7

Final Project Presentations

Readings: TODO