RR Skillverse · Free Learning Handbook
RR Skillverse Module 8 of 12 Theory Handbook Finance + AI + Multimodal

Module 8 — Multimodal AI

Same theory-first structure as Modules 1–7: for every technique below, what it is, why it exists — the specific problem it was invented to solve — what else you could use instead, and a real financial example. Every module so far has assumed RR Finance's input arrives as a table, text, or a retrieval index. Module 8 adds the input types a real lender actually deals with: phone calls, scanned documents, and photographs.

⬇ Download the Module 8 notebook ← Back to Module 7 All modules →

Recap — what RR Finance already has

Module 8 is not a fresh start. Here's exactly what it inherits before adding new input types.

MODULE 2

RR Finance's first CNN, built for cheque digit recognition — the convolutional foundation this module extends to audio and document images.

MODULE 3

Regex-based PII and field extraction from text — the same technique this module applies to OCR output from document images.

MODULE 1 & 6

The governed classifier and its fairness audit — still the destination every new input type in this module has to feed into, not a replacement for it.

What's new in Module 8
Every prior module assumed RR Finance's input was already a table, clean text, or a retrieval index. Module 8 adds three new arrival formats a real lender actually receives: a phone call, a scanned document, and a photograph — and asks how each becomes usable input to the same system.

Why multimodal, why now

Real lending intake is multimodal by necessity, not by choice.

The channel reality

Applicants call in, upload scans, and submit photo ID — a real intake pipeline has to handle all three, not assume clean tabular input arrives by default.

The honest-testing constraint

Large pretrained multimodal models (Whisper, CLIP) need internet access this sandbox doesn't have — every attempt at the real path is shown failing honestly, with a real, smaller, fully local alternative built and tested in its place.

The integration requirement

None of these new capabilities matter unless they feed the SAME governed classifier from Modules 1 and 6 — Lesson 7 wires them back in directly.

CONCEPT 1

Why Multimodal AI

Three new arrival formats, one system.

Overview
🧮 Math & Algorithm
What it is

Multimodal AI handles input that arrives in more than one form -- audio, images, and text together -- rather than assuming everything is already a clean table or a clean string.

Why it exists — the history
A practical necessity, not a research fad
Real intake systems in finance, healthcare, and government have always been multimodal -- paper forms, phone calls, photo ID -- long before "multimodal AI" was a research term. What changed is that convolutional and transformer architectures now make it practical to handle audio and images with the SAME kind of learned model already used for tabular and text data.

✓ Why this module treats each modality separately

  • Audio, document images, and vision-language matching are three genuinely different problems with three different real techniques -- bundling them under one umbrella term would hide the actual work each one requires

✗ What this module is NOT claiming

  • That one giant multimodal model replaces everything -- Lesson 7 shows a real vision-language model attempt failing honestly in this sandbox, with a smaller, real, working alternative built in its place
REAL-WORLD FINANCIAL EXAMPLE
RR Finance's phone-based loan servicing line, its scanned-document upload portal, and its photo-ID verification step are three separate real channels -- this module gives each one a real, tested technique.
⬇ Run this lesson in the notebook
No formula in this framing lesson — it sets up the three real techniques (Concepts 2-6) and the integration step (Concept 7) that make up the rest of this module.
CONCEPT 2

Audio Feature Extraction (MFCCs)

Turning a sound wave into numbers a neural network can use.

Overview
🧮 Math & Algorithm
What it is

MFCCs (Mel-Frequency Cepstral Coefficients) compress a raw audio waveform into a small set of coefficients per time frame, designed to track how the human ear actually perceives pitch and timbre -- the standard feature representation for speech tasks for decades.

Why it exists — the history
1980
Davis & Mermelstein, "Comparison of Parametric Representations for Monosyllabic Word Recognition" introduced MFCCs, building on the earlier psychoacoustic Mel scale (Stevens, Volkmann & Newman, 1937). MFCCs became the dominant speech feature representation for decades, and remain a standard preprocessing step even in modern deep-learning speech pipelines.

✓ What the notebook demonstrates

  • Real MFCC extraction, via the real librosa library, on real recordings from the Free Spoken Digit Dataset -- 3,000 genuine spoken-digit audio clips across 6 speakers

✗ What this is NOT

  • A synthetic stand-in -- this course's discipline of using real data (RR Finance's real 800-row dataset since Module 1) extends here to a real, freely available audio dataset rather than generated tones
REAL-WORLD FINANCIAL EXAMPLE
A caller reading out their loan account number digit by digit over RR Finance's phone line produces audio that needs exactly this kind of feature extraction before any recognition model can act on it.
⬇ Run this lesson in the notebook
Mel(f) = 2595 · log₁₀(1 + f/700)
The Mel scale (above) warps a raw frequency f (Hz) into a scale where equal steps correspond to roughly equal perceived pitch differences -- humans hear pitch differences non-linearly, more sensitively at low frequencies than high ones.
MFCC extraction pipeline: take the signal's short-time Fourier transform (a spectrogram), map its frequency axis through Mel-scale filters, take the log of the filter energies, then apply a Discrete Cosine Transform to decorrelate the result into a small number of coefficients (n_mfcc) per time frame.
The final output is a 2D array: coefficients x time frames -- exactly the "image-like" shape Concept 3's CNN treats as its input.
CONCEPT 3

Speech Recognition: From Whisper to a From-Scratch CNN

Extending Module 2's own CNN to a genuinely new input shape.

Overview
🧮 Math & Algorithm
What it is

Classifying WHICH digit was spoken, from its MFCC features -- the actual recognition task, built here two ways: attempting a large pretrained model (Whisper), then a small, real, from-scratch CNN when that model's weights aren't reachable.

Why it exists — the history
2022 (Whisper) vs. 1989 (CNNs for speech/vision generally)
Radford et al. (OpenAI), 2022, "Robust Speech Recognition via Large-Scale Weak Supervision" introduced Whisper, a large, general-purpose speech-to-text model trained on 680,000 hours of audio. The fallback CNN traces back to the SAME convolutional architecture family Module 2 already covered (LeCun et al., 1989) -- applied here to a spectrogram-like input instead of pixels.

✓ What the notebook demonstrates

  • A REAL attempt to load Whisper, a genuine HTTPError: 403 Forbidden when its weights can't be reached in this sandbox, and a from-scratch CNN reaching over 95% test accuracy on real, unseen spoken digits

✗ When the small CNN is the RIGHT choice anyway

  • For a narrow, closed-vocabulary task like digit entry, a small domain-specific model is often preferable to a giant general-purpose one in production -- faster, cheaper, and easier to audit, independent of what's reachable in any particular sandbox
REAL-WORLD FINANCIAL EXAMPLE
The notebook's CNN correctly recognises spoken digits from speakers it never heard during training -- exactly the generalisation a phone-based IVR system needs across RR Finance's actual caller base.
⬇ Run this lesson in the notebook
Identical 2D convolution mathematics to Module 2's CNN — see that handbook page's Math & Algorithm toggle for the full convolution and pooling formulas. The only real change here is the INPUT shape: 13 MFCC coefficients x 32 time frames, one channel, instead of a pixel image.
CONCEPT 4

Document Intelligence (OCR)

Turning a scanned form into the structured fields a model expects.

Overview
🧮 Math & Algorithm
What it is

OCR (Optical Character Recognition) converts a document image into raw text; regex-based extraction then pulls specific fields (income, loan amount, credit score) out of that text -- the same field-extraction technique Module 3 used for PII, now applied to an image-derived input.

Why it exists — the history
Tesseract: 1985 (HP) → 2006 (open-sourced by Google)
Tesseract, the OCR engine behind pytesseract, was originally developed at HP in the mid-1980s, then open-sourced and substantially improved by Google starting in 2006. It remains one of the most widely used open-source OCR engines in production document-processing systems today.

✓ What the notebook demonstrates

  • The SAME extraction pipeline run on a low-resolution and a high-resolution rendering of an IDENTICAL document, with genuinely different, honestly reported results

✗ An honest limitation, not smoothed over

  • The low-resolution version misreads a real digit (36 months read as 38) -- a genuine OCR error with real financial consequences if trusted blindly, exactly the kind of finding this course discloses rather than hides
REAL-WORLD FINANCIAL EXAMPLE
The notebook's high-resolution document extracts every field correctly; its low-resolution version -- simulating a poor-quality fax or phone-camera scan -- misses two fields entirely and silently gets a loan term wrong. Scan quality is a real, first-order variable in production document intake.
⬇ Run this lesson in the notebook
No new formula — Tesseract's internal OCR pipeline (character segmentation, then classification) is used here as a tested, real external tool, exactly as Module 5 treated FAISS or Module 3 treated spaCy's pretrained pipeline. The regex extraction step following it uses the same pattern-matching approach as Module 3's PII redaction.
CONCEPT 5

Document Tampering Detection (Error Level Analysis)

A real image-forensics attack and a real, testable defence.

Overview
🧮 Math & Algorithm
What it is

Error Level Analysis (ELA) re-saves an image at a known JPEG compression quality and measures the difference from the original -- regions edited AFTER the image was last fully saved compress differently from regions that were part of the original save, producing a detectable signal exactly where an edit happened.

Why it exists — the history
Popularised in the mid-2000s digital-forensics community
ELA exploits how JPEG's lossy, block-based compression behaves differently on freshly-edited image regions versus regions that have already been through one or more compression passes -- a well-known, widely taught technique in digital image forensics, still useful (with real limitations) today.

✓ What the notebook demonstrates

  • A real, working attack: a fraudulently edited loan amount on a real document image, followed by a real, measured ELA signal roughly 3.8x stronger inside the edited region than elsewhere in the same document

✗ An honest limitation, stated plainly

  • ELA is not foolproof -- a careful forger who re-compresses the WHOLE image after editing can weaken this signal considerably, a real constraint worth naming rather than implying ELA is a complete solution
REAL-WORLD FINANCIAL EXAMPLE
This is the image-forensics counterpart to Module 1's data poisoning and Module 5's retrieval poisoning -- a real attack on a real input channel (a submitted document image), paired with a real, honestly-scoped defence.
⬇ Run this lesson in the notebook
ELA(x, y) = |I(x, y) − JPEG_q(I)(x, y)|
I is the original image; JPEG_q(I) is that same image re-saved at a known JPEG quality q.
The difference is computed per pixel, per colour channel -- regions that were part of the original save settle into a STABLE compression pattern already, so re-saving barely changes them; freshly-edited regions have not yet "settled," so they show a larger difference.
In practice: compare the MEAN of this difference INSIDE a suspected edited region against the mean OUTSIDE it -- a substantially higher value inside is the forensic signal, as measured directly in the notebook (roughly 3.8x higher).
CONCEPT 6

Vision-Language Models: From CLIP to a From-Scratch Joint Embedding

Matching an image to a caption, without ever typing in a label.

Overview
🧮 Math & Algorithm
What it is

A vision-language model maps images and text captions into the SAME shared embedding space, trained so that matching image/caption pairs land close together and non-matching pairs land far apart -- enabling "zero-shot" classification by comparing a new image against a set of plain-text candidate captions.

Why it exists — the history
2021
Radford et al. (OpenAI), "Learning Transferable Visual Models From Natural Language Supervision" introduced CLIP, trained on 400 million real image-caption pairs scraped from the web, using a contrastive loss to align the two modalities -- the same contrastive idea this module's fallback trains from scratch, at drastically smaller scale.

✓ What the notebook demonstrates

  • A REAL attempt to load CLIP, a genuine OSError when huggingface.co isn't reachable in this sandbox, and a from-scratch contrastive model that correctly matches a brand-new, unseen document image to its plain-text caption

✗ What this is NOT

  • A claim that this small model rivals real CLIP's actual visual understanding -- this lesson is explicitly about MECHANISM, not quality, the same framing Module 3 used for its own from-scratch Word2Vec on a tiny corpus
REAL-WORLD FINANCIAL EXAMPLE
RR Finance's document intake needs to recognise WHAT TYPE of document just arrived -- loan application, bank statement, ID card, pay stub -- before any OCR or field extraction can even run; this lesson demonstrates the real mechanism behind doing that from an image and a plain-text label, with no hand-assigned category ID.
⬇ Run this lesson in the notebook
L = −(1/N) Σᵢ log[ exp(sim(imgᵢ, txtᵢ)/τ) / Σⱼ exp(sim(imgᵢ, txtⱼ)/τ) ]
This is a contrastive (InfoNCE-style) loss: for each image i in a batch, its matching caption should have the HIGHEST similarity among every candidate caption j in the batch.
sim(img, txt) is cosine similarity between the image embedding and the text embedding, both produced by their own encoder network and both normalised to unit length.
τ (temperature) sharpens or softens how strongly the loss separates the correct match from the alternatives -- CLIP learns this parameter; the notebook's fallback fixes a similar scale factor directly for simplicity.
CONCEPT 7

Multimodal Fusion

Wiring three new input types back into one governed system.

Overview
🧮 Math & Algorithm
What it is

Multimodal fusion here means something specific and modest: taking fields extracted from a document image (Concept 4) and feeding them into Module 1's ACTUAL saved classifier pipeline -- no new model, no parallel system, just a new INPUT PATH into the one RR Finance has been building since Module 1.

Why it exists — the history
This module's own throughline, not a named external technique
This lesson is deliberately NOT a sophisticated fusion architecture (there is real research on early vs. late fusion, cross-attention between modalities, and more) -- it is the much more basic, and arguably more important, discipline of making sure new capability actually connects to the governed system rather than becoming an unaudited side pipeline.

✓ What the notebook demonstrates

  • A real check of which of Module 1's FEATURES document extraction CAN supply (income, loan amount, term, credit score) versus which ones genuinely cannot come from a document image alone (previous defaults, account age, existing debt)

✗ What this is NOT

  • A claim that document intelligence alone can fully replace an application intake process -- naming the fields it CANNOT supply is as important as the ones it can
REAL-WORLD FINANCIAL EXAMPLE
A real RR Finance deployment would combine this module's document-extracted fields with the applicant's existing account history (already in Module 1's system) before the classifier ever runs -- exactly the honest gap this lesson names rather than glosses over.
⬇ Run this lesson in the notebook
No new modelling — this lesson reuses Module 1's already-fitted StandardScaler and LogisticRegression exactly as saved. The only computation is a set-membership check: which of Module 1's FEATURES a document image can and cannot supply.

Module 8 hand-off

The notebook extracts real MFCC features from a real spoken-digit dataset, extends Module 2's own CNN architecture to recognise spoken digits at over 95% test accuracy, runs real OCR-based document field extraction with an honestly-measured resolution-quality effect, detects a real document tampering attack via Error Level Analysis, builds a real from-scratch vision-language joint embedding model where CLIP itself isn't reachable, and wires the whole result back into Module 1's actual saved classifier -- with a metrics summary saved as an artifact for the next module.

What Module 9 builds on this
Module 9 (Graph Neural Networks) shifts RR Finance's lens from individual applicants to the RELATIONSHIPS between them -- fraud rings, shared addresses or devices across seemingly unrelated applications, and knowledge-graph-based reasoning. It doesn't directly reuse Module 8's audio or vision pipelines, but the same discipline carries forward: attempt the real production tool first, disclose honestly when this sandbox can't reach it, and always wire new capability back into the one system RR Finance has been building since Module 1.
←Back to all 12 modules ←Back to Module 7