RR Skillverse Module 3 of 12 Theory Handbook Finance + AI + Security

Module 3 — NLP + Financial Text AI

Same theory-first structure as Modules 1 and 2: for every technique below, what it is, why it exists — the specific problem it was invented to solve — what else you could use instead, and a real financial example. Modules 1 and 2 worked on numbers and pixels; Module 3 adds a third kind of input entirely — free-flowing text — and the techniques here exist because text doesn't arrive as a table or a fixed-size grid.

⬇ Download the Module 3 notebook ← Back to Module 2 All modules →

Recap — what RR Finance already has

Module 3 is not a fresh start. Here's exactly what it inherits before adding text as a new capability.

MODULE 1

A validated 800-row loan-risk dataset, a logistic-regression baseline (ROC-AUC ≈ 0.70), an Isolation Forest anomaly screen, and a documented data-poisoning defence — all saved as reusable artifacts.

MODULE 2

An ANN benchmarked honestly against the Module 1 baseline, a cheque-digit CNN, and the first inference-time attack (FGSM/PGD adversarial examples) with a measured adversarial-training defence.

What's new in Module 3

Everything so far assumed numeric input — a row of ratios, or a grid of pixels. Text is neither. This module's first four lessons exist purely to answer "how do we even turn text into numbers a model can use?" before anything else becomes possible.

Why text needs its own techniques

A sentence has no fixed length the way a 28×28 image always has 784 pixels, and the word "loan" is not a number. How you turn text into numbers is most of what this module teaches — and it brings two new risks with it that pixels and tabular ratios never did.

A librarian, not a conversation partner

Bag-of-Words/TF-IDF catalogue which words appear — blind to order and meaning. Contextual embeddings track what each word means given everything around it.

A dictionary vs. context

Word2Vec gives "bank" one fixed vector. BERT-style contextual embeddings give the same word a different vector depending on the sentence — a fundamentally different idea.

Two new attack surfaces

Prompt injection — text trying to hijack instructions, not just describe facts — and PII exposure, both explored as this module's security lessons.

CONCEPT 1

Bag-of-Words & TF-IDF

The simplest possible way to turn text into numbers — and still a strong baseline today.

Overview

🧮 Math & Algorithm

What it is

Bag-of-Words represents a document as raw word counts, ignoring order entirely. TF-IDF (Term Frequency–Inverse Document Frequency) refines this by down-weighting words that appear in almost every document and up-weighting words distinctive to a specific one.

Why it exists — the history

1960s–1972

Term-frequency weighting in information retrieval dates to 1960s research; Karen Spärck Jones' 1972 paper introducing inverse document frequency is the standard citation for the "IDF" half of TF-IDF. This predates deep learning by decades and remains, even today, a fast and surprisingly strong first thing to try.

Alternatives — and why start here

Option	Trade-off
TF-IDF	Fast, interpretable, no training corpus needed beyond your own documents — the right first baseline before reaching for embeddings
Word2Vec (next lesson)	Captures word meaning/similarity, needs a large training corpus
BERT embeddings (Lesson 3)	Captures context-dependent meaning, needs the most compute and data

✓ Use it when

You need a fast, interpretable baseline classifier over text
Word order and deep context matter less than topic/keyword signal

✗ Reach for something else when

Word order and meaning genuinely matter — "the bank approved the loan" and "the loan approved the bank" are identical to a bag-of-words model

REAL-WORLD FINANCIAL EXAMPLE

RR Finance's Module 3 notebook builds its news-sentiment baseline on exactly this — a fast, fully local model that reaches production-viable accuracy before anyone reaches for a heavier transformer model.

⬇ Run this lesson in the notebook

TF-IDF(t,d) = tf(t,d) × log(N / dfₜ)

tf(t,d) — how many times term t appears in document d (raw or normalised count).

N / dfₜ — total documents divided by the number of documents containing t; the log dampens this ratio. A term in every document (dfₜ=N) gets log(1)=0 weight — driven out entirely. A rare term gets a large multiplier.

Net effect: common connective words ("the," "and") score near zero everywhere; words distinctive to one document score highest in that document — the "distinctiveness" signal Bag-of-Words alone can't produce.

CONCEPT 2

Word2Vec

Giving words meaning by teaching a model to guess them from their neighbours.

Overview

🧮 Math & Algorithm

What it is

Word2Vec learns a dense vector per word by training a shallow neural network to predict a word from its surrounding context (or vice versa) across a large corpus — words used in similar contexts end up with similar vectors, learned entirely without labels.

Why it exists — the history

2013

Introduced by Tomas Mikolov and colleagues at Google, Word2Vec was a major practical breakthrough — far faster to train than the neural language models that preceded it, and its vectors captured surprisingly linear relationships (the famous "king − man + woman ≈ queen" demonstration), which convinced the field that dense embeddings genuinely encode meaning, not just co-occurrence statistics.

Alternatives — and why Word2Vec's idea still matters

Option	Trade-off
Word2Vec	Fast, well-understood, the historical breakthrough that made "dense word embeddings" the default — but gives every word exactly ONE vector, regardless of context
GloVe	A similar-era alternative using global co-occurrence statistics rather than a prediction task; comparable quality
fastText	Extends Word2Vec with subword information, handling misspellings and rare words better
Contextual embeddings (Lesson 3)	Solves Word2Vec's core limitation directly — the same word gets different vectors in different contexts

A small corpus teaches the mechanism, not the quality

The notebook trains Word2Vec on a ~10-sentence demo corpus — intentionally too small to produce reliable embeddings, but large enough to show exactly how the training mechanism works. Real Word2Vec needs a corpus of millions of words; in production you'd almost always load pretrained vectors rather than train from scratch on a small in-house dataset.

REAL-WORLD FINANCIAL EXAMPLE

A sentiment or NER model that has learned "profits" and "earnings" occupy nearby regions of vector space generalises to headlines using either word — without needing every synonym labelled by hand.

⬇ Run this lesson in the notebook

Skip-gram objective: maximise Σ log P(context | centre)
P(w_O|w_I) = softmax(v'_O · v_I)

Skip-gram trains a shallow network to predict surrounding context words from a centre word — the actual prediction task is thrown away after training; what's kept is the learned weight matrix, one row per word, which becomes that word's embedding vector.

Words used in similar contexts get pushed to similar vectors purely as a side-effect of both being good at predicting similar neighbours — no human ever labels "profits" and "earnings" as related; the training objective does it implicitly.

Cosine similarity (v₁·v₂ / ‖v₁‖‖v₂‖) between two word vectors — 1 means same direction (similar meaning), 0 means unrelated, −1 means opposite — is the standard way to query the learned space.

CONCEPT 3

Contextual Embeddings: The Idea Behind BERT

The same word, a different vector, depending on what surrounds it.

Overview

🧮 Math & Algorithm

What it is

A Transformer's self-attention mechanism computes each word's representation while looking at every other word in the sentence — so "bank" in "loan from the bank" and "bank" in "river bank" produce genuinely different output vectors, something Word2Vec structurally cannot do.

Why it exists — the history

2017

The Transformer architecture (Vaswani et al., "Attention Is All You Need") replaced the recurrent networks (RNNs/LSTMs) that previously dominated NLP — those processed text one word at a time and struggled to connect words far apart in a sentence.

2018

BERT (Devlin et al., Google) showed that pre-training a large Transformer on masked-word prediction across huge amounts of text, then fine-tuning it for a specific task, dramatically outperformed models trained from scratch per task — a shift the whole field followed, and the direct architectural ancestor of the LLMs in Module 4.

✓ What the notebook shows

A tiny, fully local, from-scratch transformer encoder proving the mechanism: the same token produces different vectors in different sentences
Real pretrained BERT loaded from Hugging Face (when internet access allows) for genuinely meaningful contextual embeddings

✗ What the toy encoder can't show

Untrained weights mean the toy version's exact numbers aren't meaningful — only the mechanism is real. Meaningful values require the pretrained model, or real training on real data.

REAL-WORLD FINANCIAL EXAMPLE

Financial text is full of context-dependent words — "credit" (a positive balance, or a risk category), "position" (a job, or a trading exposure), "return" (giving something back, or a financial gain). A model that only sees one fixed vector per word cannot tell these apart; a contextual model can.

⬇ Run this lesson in the notebook

Attention(Q,K,V) = softmax(QKᵀ/√d) V

Q, K, V (query, key, value) are three learned linear projections of every token's embedding. QKᵀ computes, for every pair of tokens, how relevant one is to the other.

Dividing by √d keeps the dot products from growing too large as dimensionality d increases (which would push softmax into a near-one-hot, hard-to-train regime).

Softmax turns those relevance scores into weights summing to 1; multiplying by V produces, for each token, a weighted blend of every other token's value vector — literally "how much should this word's output depend on that word," computed fresh for every sentence, which is why the same token "bank" gets a different vector depending on its neighbours.

CONCEPT 4

Named Entity Recognition (NER)

Pulling structured facts out of prose.

What it is

NER automatically identifies and labels spans of text as specific entity types — people, organisations, money amounts, dates — turning unstructured prose into structured, extractable facts.

Why it exists — the history

1990s

NER as a formal task traces to the Message Understanding Conferences (MUC), originally motivated by extracting facts from news wire text for government and intelligence use.

2015

spaCy (Explosion AI) became popular specifically for being fast and production-ready, in deliberate contrast to academic NLP toolkits of the era that prioritised research flexibility over speed — exactly why it's the right first tool for a production-facing lesson like this one.

REAL-WORLD FINANCIAL EXAMPLE

A loan officer's free-text case note like "Approved ₹50,000 for Rajesh Kumar on March 15" contains structured data (amount, person, date) trapped in prose — NER extracts it automatically, without a human re-typing it into a form.

General-purpose models have gaps

A pretrained NER model recognises common entity types well but has no idea "Gold Loan" or "Personal Loan" are meaningful financial-product entities — that gap is exactly what the next lesson closes.

⬇ Run this lesson in the notebook

CONCEPT 5

Fine-Tuning NER for Domain-Specific Entities

Teaching a general-purpose model your organisation's own vocabulary.

What it is

Adding a new entity label (e.g. LOAN_PRODUCT) to an existing pretrained NER pipeline, then training just that component on a small set of labelled examples — far cheaper than training a full NER model from nothing.

Why it exists

The general principle: transfer learning for NLP

Just as Module 2's CNN transfer learning reused a pretrained image backbone and retrained only the final layer, fine-tuning a pretrained NER pipeline reuses everything the base model already knows about general language structure, adding only the narrow, organisation-specific vocabulary on top.

Alternatives — light fine-tuning vs. full BERT fine-tuning

Option	Trade-off
Fine-tune spaCy's existing NER component	Works with a handful of labelled examples, fast to train, lower capacity ceiling
Fine-tune a BERT-based token classifier	Higher capacity, generally better with hundreds+ of labelled examples, needs a GPU and meaningfully more time

Validated on held-out sentences, not just the training examples

The notebook tests the fine-tuned model on sentences it never saw during training — the only honest way to confirm it actually generalised the concept of "loan product," rather than memorising the exact training sentences.

REAL-WORLD FINANCIAL EXAMPLE

Automatically tagging which loan product a case note discusses turns thousands of free-text notes into structured, queryable data — without a human reading every single one.

⬇ Run this lesson in the notebook

CONCEPT 6

Sentiment Analysis: TF-IDF Baseline to FinBERT

Classifying financial text as positive, negative or neutral — and why domain matters.

What it is

Sentiment analysis classifies text by emotional/evaluative tone. The notebook builds a fast TF-IDF + Logistic Regression baseline (Module 1's classification pattern, applied to text), then shows FinBERT — BERT further pretrained specifically on financial text — as the production upgrade.

Why FinBERT exists — the history

2019

Dogu Araci introduced FinBERT by further pretraining BERT on a large corpus of financial text (analyst reports, earnings calls, financial news) before fine-tuning it for sentiment — domain-adapted pretraining, not just a bigger generic model. This mattered because general-purpose sentiment models (often trained on movie reviews or tweets) perform poorly on financial text, where the same words carry different connotations: "aggressive growth" is positive, "aggressive accounting" is a red flag.

✓ When the TF-IDF baseline is enough

Text is fairly unambiguous, or you need something fast and fully local with no external dependency

✗ When you need FinBERT instead

Real financial text is genuinely ambiguous — the notebook's baseline scores near-perfect accuracy on synthetic templates, then is shown honestly for what it is: a demo that doesn't yet face the ambiguity real headlines contain

REAL-WORLD FINANCIAL EXAMPLE

"The company cut costs aggressively to protect margins" is genuinely ambiguous — financial discipline, or a distress signal? A domain-adapted model like FinBERT is built specifically to resolve that kind of context a generic bag-of-words approach cannot.

⬇ Run this lesson in the notebook

CONCEPT 7

Event Study: Sentiment vs. Next-Day Returns

Testing whether a signal has real predictive value before trusting it.

Overview

🧮 Math & Algorithm

What it is

An event study compares outcomes (like next-day stock returns) around a defined event (like a strongly positive or negative news day) to test whether the event actually relates to what happens next — rather than assuming it does because a story sounds plausible.

Why it exists — the history

1969

The event-study framework in finance traces to Fama, Fisher, Jensen and Roll's paper studying stock splits — the general technique (measure abnormal returns around a defined event) has since been applied to earnings announcements, regulatory changes, and — as done here — news sentiment.

Correlation, not proof of causation

The notebook's synthetic market is built with a known, modest sentiment effect on purpose, so recovering a real (if noisy) correlation confirms the methodology works. A real event study must also control for confounds — market-wide moves, sector trends, earnings-date proximity — that this simplified demo intentionally sets aside, flagged rather than hidden.

REAL-WORLD FINANCIAL EXAMPLE

Before any trading or risk system uses news sentiment as an input, this is the standard first check: does sentiment actually correlate with subsequent price movement in this specific market, or is that just an appealing story?

⬇ Run this lesson in the notebook

r = Σ(xᵢ−x̄)(yᵢ−ȳ) / √[Σ(xᵢ−x̄)² Σ(yᵢ−ȳ)²]

Pearson correlation r — measures the strength and direction of a linear relationship between sentiment score and next-day return; ranges from −1 (perfectly inverse) to +1 (perfectly aligned), 0 meaning no linear relationship.

The numerator is the co-movement of the two series around their means; the denominator normalises by each series' own spread, so r is unitless and comparable across any two variables.

Correlation says nothing about causation — a real event study also needs to control for confounds (market-wide moves, sector trends) before treating this number as a tradeable signal, exactly the caveat flagged in this lesson.

CONCEPT 8

Aspect-Based Sentiment on Earnings Calls

One sentence can be simultaneously positive and negative — about different things.

What it is

Aspect-Based Sentiment Analysis (ABSA) assigns sentiment per topic within a document, rather than one blended label for the whole thing — "revenue grew strongly but costs also rose" is positive about revenue and negative about costs at the same time.

Why it exists — the history

Mid-2000s

ABSA emerged from product-review mining research — a customer rating a restaurant's "food" highly but its "service" poorly in one review — and was adapted to earnings-call analysis as NLP matured enough to handle longer, more technical financial text.

A real, undisguised limitation

The notebook's simple keyword-window approach genuinely fails on one of its own three test aspects — sentiment meant for one aspect ("beating guidance") bleeds into a nearby aspect's window ("costs"), producing a misleadingly neutral result where the true sentiment is negative. This is shown directly, not hidden, because it's exactly the limitation that motivates trained aspect-based sentiment models in production.

REAL-WORLD FINANCIAL EXAMPLE

An analyst listening to an earnings call needs to know sentiment per aspect — revenue, costs, guidance — not one blended number that hides which specific area is actually struggling.

⬇ Run this lesson in the notebook

CONCEPT 9 · CYBERSECURITY

Prompt Injection & Evasion Attacks

Text is the first data type in this course anyone can write directly.

What it is

Evasion attacks manipulate a text classifier's output by exploiting how it represents text (e.g. stuffing buzzwords to shift a TF-IDF vector) without changing the underlying facts. Prompt injection specifically targets systems that follow instructions embedded in text, hijacking their behaviour via text crafted to look like a command.

Why it exists — the history

Mid-2000s

Adversarial manipulation of text classifiers has been studied since the spam-filter arms race — attackers have always tried to fool automated text systems.

2022 onward

Prompt injection specifically is a much more recent term, emerging directly alongside instruction-following LLMs, when researchers and practitioners noticed that text embedded in a document — not just a user's direct chat message — could hijack a language model's behaviour if the system didn't clearly separate "instructions" from "data."

Attack	Targets	Demonstrated how
Evasion (keyword stuffing)	Any text classifier, including plain TF-IDF/LogReg	Real, working attack in the notebook — buzzwords mechanically shift the model's decision
Prompt injection proper	Instruction-following systems (LLMs)	Illustrated with a deliberately naive mock system — the real thing arrives with real LLMs in Modules 4–5

A different attack surface than Modules 1 and 2

The attacker here never touched training data (contrast Module 1's poisoning) and never perturbed pixels (contrast Module 2's FGSM/PGD) — only the input text itself. Text's openness to arbitrary input is precisely what makes it a distinct risk from Modules 1–2's structured numbers and fixed-size images.

REAL-WORLD FINANCIAL EXAMPLE

If an attacker can manipulate what a sentiment pipeline concludes about a headline — or hijack what instructions an LLM-based report-generation system follows — financial decisions built on that output are compromised without the underlying facts changing at all.

⬇ Run this lesson in the notebook

CONCEPT 10 · CYBERSECURITY

PII Detection & Redaction

Free text leaks personal data in ways a spreadsheet of ratios never does.

What it is

Automatically finding and removing personally identifiable information — names, emails, phone numbers, ID numbers — from free text before it flows into training data, logs, or shared reports.

Why it exists — the history

2019

PII-detection tooling has existed in enterprise data-loss-prevention products for years, but often as proprietary, closed products. Microsoft open-sourced Presidio in 2019, combining pattern matching with NLP-based recognition (using the same NER mechanism as Lesson 4) — a modern, transparent option any team can run locally.

Alternatives — and why combine both

Option	Trade-off
Regex only	Transparent and fast for fixed-pattern PII (emails, phone numbers) — but a person's name has no fixed pattern at all, so regex alone structurally cannot catch it
Presidio (NLP + patterns)	Catches names via NER, patterns via regex — genuinely more complete, though not perfect (see the honest false-positive finding below)

A real false positive, not a hidden one

In the notebook's own PII-redaction lab, Presidio mis-tags a PAN code as a person's name alongside correctly catching the real name in the same sentence. This is shown directly: automated PII detection is a triage tool, exactly like Module 1's Isolation Forest — it should route text to faster, better-informed human review, not be trusted as a perfect, unsupervised final step. Region-specific identifiers (a PAN is India-specific) often need a custom recognizer added on top of a general-purpose tool.

REAL-WORLD FINANCIAL EXAMPLE

A loan officer's case note mentioning a customer's name, email and phone number cannot go untouched into a training set or a shared analytics report — this lesson's redaction pipeline is exactly the control that prevents it.

⬇ Run this lesson in the notebook

Module 3 hand-off

The notebook saves a sentiment baseline model, a fine-tuned NER pipeline recognising loan products, and redacted case-note datasets — extending RR Finance's system with a third data modality alongside Module 1's tables and Module 2's images.

What Module 4 builds on this

Module 4 (Generative AI, LLMs & RLHF) scales the Transformer mechanism from Lesson 3 — kept deliberately small and local here — up to a real, locally-hosted LLM. The prompt-injection illustration from Lesson 9 stops being a toy mock and becomes a real, red-teamed attack against a real instruction-following model.

→Continue to Module 4: Generative AI, LLMs & RLHF ←Back to all 12 modules