Season 2, Episode 2. Part two of two. Last episode ended mid-argument: James asked where a chatbot stores the date of the Battle of Hastings, and Sarah said it doesn't store it anywhere. This is the answer.

Still no jargon and no assumed knowledge. James holds the audience's scepticism, Sarah explains, and James drags her back out of the machine room whenever it gets too clever.

What we cover:

• Where the "knowledge" actually lives when there is no database inside: a map of meaning built from billions of numbers, where related ideas sit near each other
• Why relationships are directions rather than stored facts — the step from "man" to "woman" is the same step from "king" to "queen," and nobody ever wrote a rule about gender
• Why the map isn't flat: hundreds of directions, each one a quiet question about the word, so a word is really a dot with hundreds of scores attached
• The three things worth keeping, and permission to forget the rest
• Does it understand? The honest answer is that serious people are split roughly fifty-fifty, and we don't pretend to settle it
• Why it invents things: plausibility is the target, not truth — plus the second cause, which is that we scored these systems in a way that rewards a confident guess over "I don't know"
• The lawyer who filed fabricated case law and called ChatGPT "a super search engine" — four words that contain the whole misconception
• Four differences from normal software you will actually feel: same question different answer, nobody wrote the rules, it fails politely instead of crashing, and it goes stale
• Why that makes AI a running cost rather than a purchase — less like buying a dishwasher, more like hiring someone, and only one of those two gets budgeted for
• The rule of thumb worth keeping: if you can write the rule down, write the rule down. Nobody should train a model to calculate GST
• James pushes back twice: if there's no database, why does his chatbot give him real working links — and how does a frozen model know about last week? Sarah on retrieval, live search, and the model-versus-system distinction
• Why cited answers are safer but not safe: the check stops being "is this true" and becomes "does the thing it linked me to actually say this"

The takeaway: the useful question is never "does it understand?" It's "can I rely on this without checking?" Even when it hands you a link. A chatbot is not a search engine.

Key references:

• CSET Georgetown, how language models predict the next word, in plain language: https://cset.georgetown.edu/article/the-surprising-power-of-next-word-prediction-large-language-models-explained-part-1
• MIT Technology Review, why hallucination is what these models are designed to do, and the lawyer's "super search engine": https://www.technologyreview.com/2024/06/18/1093440/what-causes-ai-hallucinate-chatbots/
• OpenAI, why training and evaluation reward guessing over admitting uncertainty: https://openai.com/index/why-language-models-hallucinate/
• Bender, Gebru et al. (2021), On the Dangers of Stochastic Parrots: https://doi.org/10.1145/3442188.3445922
• Lewis et al. (2020), the original paper on retrieval-augmented generation: https://arxiv.org/abs/2005.11401
• IBM, what retrieval-augmented generation is: https://www.ibm.com/think/topics/retrieval-augmented-generation
• IBM, narrow AI is the only kind that exists today: https://www.ibm.com/think/topics/artificial-general-intelligence

Better AI still starts with better foundations.

Send us Feedback