Why AI gets things wrong and how to check its work

A note on dates: AI tools and the checking features built into them change every week. Make sure you're reading a recent version of this article, and contact Softwired for training and materials that keep pace.

The short answer

Check the specific facts in every AI answer before you use them, using the 5-step routine below. AI tools predict likely-sounding text. Most of the time that produces useful, accurate work, and sometimes it produces confident statements that are wrong, called hallucinations.

Hallucinations cluster around specific facts: numbers, dates, names, quotes, legal or technical detail, and anything recent. Those are the parts to check every time.

Why it happens

It predicts from patterns. Unless the tool searches the web or your files, it answers from what it learned in training.
Its knowledge has a cutoff, so anything newer needs web search or your own documents.
It tries to be helpful. Ask for something that doesn't exist and it may produce something plausible to fill the gap.
Your question carries a false premise. "Why did the ATO change the rule in 2025?" can produce an explanation for a change that never happened.
The context is long or messy, and details from early in a chat get mixed up.
Its sources are poor. When it searches, it can pick up out-of-date or unreliable pages.

The warning signs

Very specific numbers or dates without a source
Quotes attributed to real people
Citations you can't open, or that don't say what the AI claims
Legal, tax, medical or safety advice stated as certain
Overseas rules described as if they applied in Australia
Perfect-sounding answers to questions you know are contested

A 5-step checking routine

Ask what it's based on: "Which parts of this come from sources, and which are your own reasoning?"
Open the citations. Click through for every fact you'll rely on, and check that the page says what the AI says and when it was written.
Check against the source of truth. For your business, that's your own documents, systems and people. For rules, it's the official website, for example the ATO, the Fair Work Ombudsman or a Queensland government site.
Ask it to argue the other side: "What might be wrong with this answer?" or "What would an expert disagree with?"
Have a human sign off. A person who knows the subject reads anything going to a customer, a regulator or the public before it goes.

How you'd actually use it

Make the AI show its working

Try this prompt: "Answer using only the attached policy. Quote the section for each point. If the policy doesn't cover something, say so."

Ground it in your files

Use projects, connectors or uploads so answers come from your own material.

Use deep research for sourced answers

Research modes in ChatGPT, Claude, Gemini and Copilot produce reports with citations you can open and check. Our guide to deep research shows how to steer one.

Set a team rule

For example: "AI can draft anything. A person checks numbers, legal statements and anything external before it goes out."

What the vendors say

The major AI vendors state that their tools can make mistakes, and their help pages ask you to review important output. Agent features add a second risk, prompt injection: hidden instructions in content the AI reads can trick it into doing something you didn't ask for. Anthropic's safety guidance for Claude Cowork tells users to watch for tasks drifting beyond what they asked for and to stop the task if something looks off. Our guide to AI agents covers how to supervise one.

Frequently asked questions

Are newer models more accurate?

Often, yes, and vendors regularly report accuracy improvements. Every model still makes mistakes, so keep checking.

Is web search enough to stop hallucinations?

Web search helps with recent facts, and the AI can still misread a source or rely on a poor one. Open the citation every time.

Can I tell the AI to never make things up?

You can ask it to say "I don't know" and to stick to your sources. That reduces made-up answers, and you still need to check the facts that matter.

Who is responsible if AI gets it wrong?

You are, for anything you send or publish. Anthropic's guidance for Claude Cowork states that users remain responsible for actions taken on their behalf.

Related guides

Put it to work with AI On Ramp

A shared checking routine is the cheapest risk control a team can adopt.

AI On Ramp gets your team using AI with confidence, on their own work and across teams, while you keep its cost and risk under control. We set up the tools, then our consultants sit beside your most advanced AI users and build real in-house workflows with them. Your nominated AI Lead gets the governance tools and our ongoing support to keep the business current as the tools change. We bring deep technical expertise and decades of business change management, and we work face to face from Brisbane and along Australia's east coast. See how AI On Ramp works.

Want to talk through how this applies to your team? Book a call with Softwired.

References