Why AI gets it wrong (and why that's normal)
Stop being surprised when AI is confidently wrong. Understand why it happens — and build the editor mindset so you catch it before it ships.
Confident. Polished. Wrong.
You've seen it happen. You ask the AI for something, it comes back in ten seconds looking great — clean, confident, well-structured. And somewhere in there is a statistic that doesn't exist, a product feature you don't have, or a claim that would make legal wince.
Most people react one of two ways. Either they decide the tool is unreliable and quietly go back to doing everything by hand — or they never catch the mistake at all, and keep shipping AI output as-is, trusting it a little more each time.
Both reactions come from the same place: a wrong picture of what this tool actually is.
The tool isn't broken. It's working exactly as designed. Once you understand what it's actually doing, you'll stop being surprised by the mistakes, start catching them early, and get noticeably better output. That's this whole lesson.
It's not looking anything up
When you type a question into ChatGPT, here's what's not happening: it is not searching a database of verified facts. There isn't one. Nothing is being looked up.
What it's doing is predicting. It has read a staggering amount of text — most of the internet, effectively — and it has become extremely good at one thing: guessing which words most plausibly come next. That's the entire trick. It's a far better trick than it sounds. But hold onto the distinction: it's a plausibility machine, not a truth machine.
Think of it like a copywriter who has absorbed every swipe file in existence. Every ad, every case study, every landing page ever written. Ask them for "a case study intro with a compelling stat" and they know exactly what one looks like — the shape, the rhythm, the kind of number that goes in it. So they produce one. Instantly. About anything.
Including products that don't exist and studies that were never run.
That's not the copywriter being dishonest — that's the copywriter doing precisely what they're brilliant at: producing the shape of the right answer. When the AI invents a statistic, it's the same move. Plausible was the assignment. Plausible is what you got.
One more layer for marketers specifically: since it learned from everything, its default output is the average of everything. Ask a generic question, get the industry-average answer. Keep that thought — it's the whole of Module 2.
Tokens & the context window — the short version
Two bits of jargon are worth knowing now, because they explain behavior you've already noticed. Plain English only — we go deeper in Module 4.
The chunks the AI reads and writes in — roughly three-quarters of a word each. Everything you send it is counted in as input; everything it writes back is counted out as output. The machine runs on word-chunks, in and out.
How much it can hold in mind at once — its working memory. Think a creative brief with a hard page limit: everything you want shaping the work has to fit on those pages. Your messages, its replies, that 40-page PDF you pasted — all competing for the same limited space. When the brief is full, older material falls off.
Why mention this now? Because it explains something you've probably experienced: long, sprawling chats degrading over time. That's not the tool failing — that's the brief overflowing. Module 4 turns this into practical habits.
It sounds exactly the same when it's wrong
Now the dangerous part. It's not that the AI makes mistakes — everything makes mistakes. It's how it sounds while making them.
It sounds exactly as confident when it's wrong as when it's right. Same polish. Same authority. Same clean structure. No stammer, no "I think...", no tell.
The word you've heard. All it means: the AI confidently filling a gap with something plausible instead of something true. Fancy word, simple mechanic.
Here's why this catches even smart people — especially marketers. We know better than anyone that polish reads as credibility. It's half our job. Clean design, confident copy, a specific number — that's how you make a claim land. We weaponize that wiring daily.
The AI runs that exact trick on us, all the time, without meaning to. Maximum polish, every answer, including the invented ones. It's great ad copy for a product that might not exist. Your credibility detector doesn't just fail here — it fails in the wrong direction.
Polish is not proof.
Spot the guess
Three statistics, the kind the AI hands you every day. One of them was invented — produced swipe-file style, in the same confident tone as the others. Click the one you think is the guess.
Open a fresh chat and paste: "Give me 5 statistics about [your industry], with the source for each." Pick the most specific claim it returns and search for the actual study. Before you checked — could you tell which claims were safe? That gap is what this module is about.
If everything checks out: it does that sometimes. Run it again on a nicher topic. The point isn't that it always guesses — it's that you can't tell when it does.
AI drafts. You edit.
Three shifts. None of them is a magic prompt.
Change what you think you're holding. AI output isn't an answer — it's a first draft from that swipe-file copywriter: brilliant, fast, completely unverified. You'd never publish a freelancer's first draft without reading it. Same rule, no exceptions. That's not a workaround for a flawed tool — that is the workflow, permanently.
The specifics are yours. Numbers, names, dates, statistics, product claims, quotes — anything a customer, a journalist, or legal could hold up later. If it's specific and it matters, a human verifies it. Not because the AI is usually wrong — it's usually right — but because you can't tell which specific is the guess.
Your expertise just got promoted. The AI doesn't know your audience, your positioning, or which claim gets you in trouble. You do. A marketer who knows their stuff catches the plausible-but-wrong line in five seconds. Your judgment didn't get replaced — it moved up the chain: from writing the words to directing and quality-checking them.
The reviewer's question
The risk in AI-assisted work isn't sloppy writing anymore — sloppy writing announces itself. The risk now is confident writing: clean, polished, plausible, carrying an invented fact straight through review, because polish is exactly what reviewers are trained to wave through.
If the answer is "the AI said so" — it's not done yet. Not wrong, necessarily. Just not done. Producers verify. Reviewers ask. Same mindset, both sides of the desk.
Your picture of it was.
Exercises
The claim audit ~10 MIN
Take one recent piece of AI-assisted work. Highlight every specific claim — numbers, stats, names, dates, features, quotes. Mark each: verified / unverified / wrong. The score isn't the point. The point is the feeling of looking at a claim you shipped and not knowing which column it belongs in. That feeling is the editor mindset switching on.
Reflection
- When did AI output last burn you — or nearly? What would have caught it earlier?
- Which camp have you drifted toward: trusting a bit too much, or avoiding the tool? What changes this week?
- What's the one type of claim in your role that should always get verified before it ships?
Reference card — The Editor Mindset
- The picture: a copywriter who's absorbed every swipe file ever — producing the shape of the right answer, unverified.
- The rule of thumb: AI output = first draft, never final answer.
- Always verify: numbers & stats · names, dates, titles · product features, pricing · quotes & sources · anything legal-sensitive.
- The reviewer's question: "Where did this number come from?"
- The line: Fluency is not accuracy. Polish is not proof.
Go deeper
- Generative AI exists because of the transformer — Financial Times A scrollable visual explainer of how these models actually produce text. The most accessible of the bunch. ~10 min · easiest
- Why language models hallucinate — OpenAI OpenAI's own plain-language explanation of why models guess confidently instead of saying "I don't know." ~10 min read
- Intro to Large Language Models — Andrej Karpathy The best plain-English full walkthrough of what an LLM is. For anyone who wants the complete picture. 1 hr video
- What Is ChatGPT Doing … and Why Does It Work? — Stephen Wolfram The deepest written explanation of the prediction mechanic on this list. Long, but rewards the effort. Long read
Next: what's in its head about us?
If the AI only works with what's in its head — what does it actually know about your audience, your product, your voice when you hit enter? Almost nothing. And that turns out to be the single biggest lever on output quality there is.