Module 4 — Generative AI, LLMs & RLHF
Same theory-first structure as Modules 1–3: for every technique below, what it is, why it exists — the specific problem it was invented to solve — what else you could use instead, and a real financial example. Module 3 gave you the Transformer's self-attention mechanism. Module 4 points that mechanism at generation, then walks the exact pipeline that turns a raw text-completion engine into something that behaves like a helpful assistant.
Recap — what RR Finance already has
Module 4 is not a fresh start. Here's exactly what it inherits before adding language generation and alignment.
A validated tabular dataset, a logistic-regression baseline, and a documented data-poisoning defence.
Deep learning fundamentals — an ANN, a CNN, and the first inference-time attack (adversarial examples) with a measured defence.
Text representations, contextual embeddings (a small hand-built Transformer encoder), and two security lessons — prompt injection and PII redaction, including a toy mock of what LLM instruction-hijacking might look like.
From "predict the next word" to "a helpful assistant"
A model trained on nothing but next-word prediction, at scale, implicitly learns grammar, facts, and reasoning patterns — but it does NOT automatically learn to be a helpful, instruction-following assistant. That gap is what most of this module's lessons close.
The surprising part
A raw next-token predictor is just as likely to continue a question with another question as with an answer — that's a statistically plausible continuation of internet text too.
The pipeline this module walks
Raw model → SFT (teach instruction-following) → Reward Model (learn what "good" means) → RL step (optimise toward it) → DPO (a simpler alternative).
Why the security lessons follow naturally
An LLM that follows instructions in its input is exactly the property that can be hijacked — closing the loop Module 3's toy prompt-injection demo opened.
The GPT Decoder: Causal Self-Attention
The one architectural change that turns a representation-learner into a text generator.
A GPT-style decoder is a Transformer that predicts the next token from everything before it, using a causal mask that prevents any position from attending to future tokens — the key difference from Module 3's bidirectional BERT-style encoder.
✓ Why the causal mask matters
- Without it, the model could "see the answer" during training by copying its own future input — trivial training loss, useless at real generation where future tokens don't exist yet
✗ What this doesn't give you yet
- A model that completes text plausibly is not the same as a model that follows instructions helpfully — that gap is what Lessons 3–6 close
P(x₁..x_T) = Π P(xₜ | x<ₜ)
Sub-word Tokenization (BPE)
How a model represents words it has never seen before.
Byte-Pair Encoding (BPE) builds a vocabulary of frequently-occurring sub-word chunks from a training corpus — common words become single tokens, rare or unseen words fall back to smaller pieces, so the model never hits a hard "unknown word" wall.
| Option | Trade-off |
|---|---|
| Character-level (Module 4's Lesson 1 tokenizer) | Simple, no "unknown word" problem, but produces long sequences for long words |
| Whole-word | Short sequences, but any unseen word becomes an unknown token entirely |
| Sub-word BPE | The practical middle ground — common words stay whole, rare/novel words decompose gracefully |
Supervised Fine-Tuning (SFT)
The first step that teaches a raw model to follow instructions at all.
SFT is continued training on a curated dataset of high-quality example interactions — architecturally identical to pretraining (same cross-entropy next-token loss), just on a smaller, higher-quality, task-shaped dataset.
✓ What SFT gives you
- A model that has seen many examples of "instruction → helpful response" and mimics that pattern
✗ What SFT alone doesn't give you
- A general sense of which of several plausible responses is better — that requires the reward model (next lesson)
- Too many SFT steps on a narrow dataset risks "catastrophic forgetting" of general capability
Reward Modelling
Turning "humans prefer this response" into a trainable scoring function.
A reward model is a classifier that outputs one scalar score for a piece of text, trained so `score(preferred) > score(rejected)` on human-labelled preference pairs — the same sequence-classification architecture as Module 3's sentiment model, repurposed.
| Option | Trade-off |
|---|---|
| Absolute scoring by humans | Inconsistent across raters and over time |
| Pairwise preference (Bradley-Terry loss) | More consistent human judgments; only needs relative ranking, not an absolute scale |
L = −log σ(r(chosen) − r(rejected))
The RL Step: PPO
How a reward score actually updates the model's weights.
Given a trained reward model, the RL step adjusts the language model to produce higher-reward outputs more often. At its core: policy gradient / REINFORCE — `loss = -advantage × log π(action)`. PPO adds clipping and a KL penalty on top, purely for training stability at scale.
| Option | Trade-off |
|---|---|
| REINFORCE / policy gradient (the core mechanism) | Simple, directly demonstrable — but noisy and less stable at scale |
| PPO | REINFORCE plus clipping and a KL penalty — the stability engineering that makes it viable at production scale |
ρₜ = π_new(a|s)/π_old(a|s)
DPO: Direct Preference Optimisation
Collapsing three training stages into one.
DPO shows that the reward model and the RL step (Lessons 4–5) can be mathematically collapsed into one direct loss function, computed straight from (prompt, chosen, rejected) triples against a frozen reference copy of the model — no separate reward model, no RL loop.
| RL pipeline (SFT → Reward Model → PPO) | DPO | |
|---|---|---|
| Moving parts | Three separate stages | One stage, on top of SFT |
| Compute cost | Higher | Lower |
| When still preferred | Need the reward model as a reusable, inspectable artifact | Just need the fine-tuned model, fewer infrastructure pieces |
Parameter-Efficient Fine-Tuning: LoRA & QLoRA
Adapting a huge model by training a tiny fraction of it.
LoRA freezes an entire pretrained model and injects small, trainable low-rank matrices alongside its existing weights — often training under 1% of the original parameter count. QLoRA adds 4-bit quantisation of the frozen weights on top, cutting GPU memory needs further.
✓ Why this matters practically
- Maintaining several lightweight fine-tuned variants of one base model (e.g. retail lending, commercial lending, customer support) becomes practical instead of storing several full copies
✗ The capacity trade-off
- Higher rank (`r`) gives the adapter more expressive capacity at the cost of more trainable parameters — the same trade-off as choosing hidden-layer width in Module 2's ANN lesson
LLM Jailbreaks
Same harmful intent, different words — and a naive filter can't tell.
A jailbreak is a prompt crafted to make an instruction-following model ignore its safety training — typically by reframing a harmful request as roleplay, fiction, or a claimed "developer mode," rather than asking directly.
Red-Teaming With Garak
Moving from manually inventing jailbreaks to a systematic scanning tool.
Garak ("Generative AI Red-teaming & Assessment Kit") is a real, open-source, actively maintained LLM vulnerability scanner with dozens of probe categories — jailbreaks, prompt injection, data leakage, toxic generation, and more — each paired with automated detectors that judge whether an attack succeeded.
✓ What the notebook demonstrates
- A real garak scan, run fully offline against a null test target — genuine plumbing: a real probe, a real detector, a real pass/fail report
✗ What it doesn't prove on its own
- A clean scan against one probe category (or an offline test target) says nothing about the others — a real security review runs many probe categories against the actual deployed system
Module 4 hand-off
The notebook trains a from-scratch GPT, a real SFT model, a real reward model, a real DPO model, a real LoRA adapter, and produces both a manual jailbreak demonstration and a real garak scan — saved as artifacts and a metrics summary for the next module.