Oxford LLMs Materials

This collection brings together lecture slides, workshop materials, and practical guides from our past iterations of the “Oxford LLMs” summer school for social sciences. The content spans foundational concepts to advanced applications—from transformer architectures and model interpretability to fine-tuning techniques and LLM agent systems. Whether you’re exploring core NLP principles or specialized implementations like retrieval-augmented generation, these materials provide both theoretical grounding and hands-on guidance. All resources are openly available for researchers, practitioners, and students working with LLMs in their research!

2024

Building on the 2023 program, the materials in 2024 feature both the fundamentals of LLMs, applied tutorials, and recent advancements. Particularly, the program delves deep into foundational concepts in model architecture and evaluation. Subsequent lectures explore agent-based systems and advanced fine-tuning techniques. The practical components address specialized applications including retrieval-augmented generation for social science research, implementation of observability tools, and workflow development for LLM agents. Technical deep dives cover the Gemini architecture and practical considerations for self-hosting models, providing end-to-end perspectives on contemporary LLM applications. Grigory Sapunov, Tatiana Shavrina, and Ilya Boytsov developed the lecture materials, with seminar contributions from Atita Arora, John Githuly, Christian Silva, and Ciera Fowler. See the full list of materials below:

Lectures

• Introduction to LLMs by Grigory Sapunov.

• LLM Evaluation by Tatiana Shavrina.

• LLM Agents by Tatiana Shavrina.

• LLM Agents (continued) by Grigory Sapunov.

• Fine-Tuning and Alignment by Ilya Boytsov.

Seminars

• Navigating RAG for Social Science by Atita Arora.

• LLM Observability and Evaluation by John Githuly.

• Tutorial: Creating LLama-Researcher using LlamaIndex workflows by John Githuly.

• Gemeni Model Deep Dive by Christian Silva.

• ORI’s Guide to Self Hosting LLMs by Ciera Fowler.

2023

The 2023 program covers fundamental and applied aspects of transformer-based language models, tracing their evolution from the 2017 seminal “Attention is all you need” paper to contemporary systems like Chat-GPT. Lectures created by Elena Voita examine critical challenges including model bias, interpretability methods, and alignment techniques. Practical seminars designed by Ilya Boytsov provide hands-on experience with transformer models, from basic fine-tuning approaches to advanced methods like parameter-efficient adaptation. Additional sessions explore specialized applications including prompt engineering, classic NLP pipelines, and reinforcement learning for model detoxification. See the full list of materials below:

Lectures

The following lecture materials were and created by Elena Voita.

• The Evolutionary Journey of NLP from rule-based systems to modern Transformers-based models, which are the core technology underpinning LLMs.
• Bias in LLMs and a (bit of) Interpretability.
• LLMs and Alignment.

Seminars

The following workshop materials were designed and implemented by Ilya Boytsov.

• Google Collab environment setup, general intro
• Introduction to Huggingface transformers library
• Topic modelling with Transformers using BERTopic library
• A guide how to fine-tune pretrained model for a classification task
• Parameter Efficient Fine Tuning (PEFT)
• Transformers interpretability, Attention visualisation, and saliency methods (e.g. Integrated gradients)
• Model analys with classic NLP using Spacy
• Prompts and instructions with Llama 2
• Detoxifying summarisation model with Reinforcement Learning from Human Feedback (RLHF)