Table of Contents
In June 2023, a New York lawyer named Steven Schwartz filed a motion citing six earlier court decisions. Every one of them had been invented by ChatGPT. The judge fined him $5,000, and the case — Mata v. Avianca — became the textbook example of what happens when someone trusts a system that sounds certain but isn’t.
That story is not an argument against using AI. Modern models draft contracts, summarise research, write working code and translate between languages at a level that would have seemed implausible a decade ago. But they fail in specific, predictable ways, and knowing those failure modes is the difference between a useful tool and an expensive mistake.
AI Has No Model of the World
A language model predicts the next token based on statistical patterns in training text. It does not build a mental picture of gravity, money, or how a door handle works. The results can look like understanding right up until they don’t.
Ask a model to track a ball rolling off a table and it will often describe a physically impossible arc. Ask it to count the letter “r” in “strawberry” and earlier generations confidently said two. On ARC-AGI, a benchmark of simple visual puzzles designed to be easy for humans but hard for machines, top systems hovered around 50–65% on the hidden test set while people score above 85%.
Why “sounding right” isn’t “being right”
Fluency and accuracy are produced by the same machinery, so they feel identical from the outside. A model that has never seen your industry’s edge cases will still write about them in smooth, well-punctuated prose. That mismatch is the single biggest source of overconfidence among people new to these tools.
The Data Ceiling
Every model is a compressed reflection of what it was trained on. Gaps in the data become gaps in the output, and the gaps are rarely obvious.
A diagnostic system trained on scans from one hospital network can lose accuracy when deployed in a clinic with different machines, different patient demographics, or different imaging protocols. Rare conditions get less attention than common ones simply because there are fewer examples. Events that happened after the training cutoff are invisible: a model frozen in early 2024 genuinely does not know what happened last month unless you give it tools to search.
- Long-tail problems: unusual cases are underrepresented by definition, so performance quietly drops where it matters most.
- Distribution shift: behaviour changes when real-world inputs drift away from the training set.
- Stale knowledge: anything after the cutoff date has to be supplied or retrieved.
- Proprietary blind spots: your internal documentation, customer history and process knowledge aren’t in the model unless you put them there.
Hallucination Is Built Into the Architecture
Models generate plausible continuations, not verified facts. When the correct answer is missing, the most statistically likely-sounding answer fills the gap. This is not a bug that a patch will eliminate; it’s the same mechanism that makes the system useful.
The numbers are sobering. A 2023 Stanford study tested legal research tools built on leading models and found hallucination rates between 17% and 33%, even for products marketed specifically for citation accuracy. Medical summaries, financial figures and product specifications show similar patterns.
Retrieval-augmented generation helps a lot — grounding answers in documents you supply cuts fabrication sharply. It doesn’t get to zero, and there’s no reliable internal signal that tells you which sentence is the invented one.
Short Memory and Fragile Planning
Context windows have grown from a few thousand tokens to hundreds of thousands, which sounds like solved memory. It isn’t. Models attend unevenly across very long inputs, and details buried in the middle of a 200-page document are more likely to be missed than the same detail at the start or end.
Multi-step agency is where the cracks really show. A task requiring 20 sequential actions with a 95% success rate per step succeeds roughly 36% of the time overall. Errors compound. Systems that look competent in a demo often stall on the fourth or fifth step of a real workflow, and they rarely notice they’ve drifted.
The Black-Box Problem
Nobody can fully explain why a large neural network produced a particular output. For a movie recommendation, that’s fine. For a loan rejection, a medical triage decision or a hiring screen, it’s a legal and ethical obstacle. GDPR Article 22 gives Europeans a right to meaningful information about automated decisions that significantly affect them, and “the model weighted these 40,000 features” doesn’t satisfy that.
Interpretability research is progressing, but in regulated industries the practical answer today is often to keep the model in an advisory role and a human in the decision seat.
Bias Doesn’t Disappear — It Scales
In 2018 Amazon scrapped an internal recruiting tool after discovering it penalised resumes containing the word “women’s” — the model had learned from a decade of male-dominated hiring data. That wasn’t a coding error. It was the training data working exactly as designed.
Because these systems are cheap to deploy at scale, a small skew becomes a very large effect. Auditing for disparate impact across gender, ethnicity, age and disability isn’t optional once a model touches people’s lives.
Cost, Energy and the Compute Wall
Frontier training runs now carry price tags in the tens or hundreds of millions of dollars, plus electricity bills that show up on utility balance sheets. Inference isn’t free either — a generative query can consume several times the energy of a traditional search, and running thousands per day adds up.
Smaller, fine-tuned models increasingly handle narrow tasks at a fraction of that cost, which is why many teams now default to the smallest model that clears their quality bar. Knowing where the ceiling sits for your use case is a budgeting skill as much as a technical one, and it’s the kind of thing covered in structured artificial intelligence certification programs that focus on deployment and model evaluation rather than theory.
Where Humans Still Have to Stay in the Loop
None of this means AI is overrated. It means the sensible deployment pattern looks less like “replace the team” and more like “add a fast, tireless, occasionally wrong colleague who needs reviewing.”
Practical guardrails that actually work:
- Verify every factual claim that leaves your organisation — citations, figures, names, dates, statistics. Treat unverified output as a draft, not a source.
- Ground the model in your own documents where accuracy matters, and give it an explicit “I don’t know” route out.
- Log inputs and outputs so failures can be traced and audited later.
- Run a shadow pilot on real cases before anything customer-facing goes live.
- Define who is accountable when the system is wrong. “The AI decided” is not an answer a regulator, a client or a court will accept.
The honest position is that AI is extraordinary at compression, retrieval and recombination, and weak at judgement, grounding, causal reasoning and accountability. Build around that split and the technology earns its keep. Ignore it, and sooner or later you end up in front of a judge explaining why your filing cited cases that were never written. If you want to go deeper on where these systems break and how to test them properly, there are paid and free AI courses that spend real time on evaluation rather than hype.
