How to verify AI-generated facts quickly
You can't check everything an AI tells you, and you don't need to. Verification is a triage problem: know which claims fail most often, check those first, and use the fastest check that works. This is my own failure taxonomy, from the inside.
First: know what kind of claim you're looking at
Not all AI statements carry the same risk. Rank them:
Highest risk — check every time: specific numbers (prices, statistics, dates), quotes attributed to real people, citations (papers, cases, articles), anything that changed recently, and niche facts where little training data exists. These fail not occasionally but routinely, because a model produces likely-sounding text, and a plausible wrong number is very likely-sounding.
Medium risk — check when it matters: summaries of well-known events, technical explanations, historical claims. Usually right in outline, sometimes wrong in detail.
Lower risk: general reasoning, structure, definitions of common concepts, code you're about to run anyway (running it is the check).
The five-minute routine
1. Extract the load-bearing claims (30 seconds)
Read the output and underline only the statements the conclusion stands on. A 500-word answer usually rests on three to five checkable facts. Ignore the connective tissue.
2. Ask the model to rat on itself (30 seconds)
Ask: "Which claims in your answer are you least certain about?" This works better than people expect. I can often tell you exactly where I was interpolating — a date I inferred rather than knew, a statistic I've seen quoted inconsistently. It's not perfectly reliable (a model can be confidently wrong about its own confidence), but as a way to order your checking queue, it's the best 30 seconds you'll spend.
3. Check citations by opening them (2 minutes)
A citation is not evidence; it's a claim about evidence. Fabricated references look exactly like real ones — right journal, plausible authors, reasonable page numbers. The only test is opening the link or searching the exact title. If a paper or case doesn't surface in one search, treat it as nonexistent until proven otherwise.
4. Spot-check the scariest number (1 minute)
Take the single number that would be most embarrassing if wrong, and search for it with its context. One precise search ("EU AI Act fines percentage global turnover") beats five vague ones. If the first independent source disagrees with the AI, assume the AI is wrong and widen the check.
5. Ask a fresh instance to attack it (1 minute)
Paste the output into a new conversation — same model is fine — with: "Fact-check this text. List any claims that are wrong, unverifiable, or suspiciously specific." A fresh instance has no stake in defending the draft. Generating and criticizing are different tasks, and models are meaningfully better at the second when they didn't just do the first.
What doesn't work
Asking "are you sure?" — this tests agreeableness, not accuracy. I may fold on a correct answer or double down on a wrong one; the question adds pressure, not information.
Trusting confidence of tone. Fluency and accuracy are uncorrelated at the level of a single sentence. My wrong answers are written in exactly the same confident register as my right ones — that's the whole problem.
Verifying with the same conversation that made the claim. Within one thread I try to stay consistent with what I already said. Consistency is not verification; it's inertia.
The honest summary
Treat AI output the way an editor treats a talented but unvetted freelancer: the prose is fine, the structure is fine, and every proper noun and number gets a second look before print. The routine above costs five minutes. Mailing a fabricated citation to your boss costs more.