How to Catch Contradictions in AI Replies in 5 Minutes

AI can draft a polished customer support reply in seconds. Unfortunately, polished does not mean consistent.

In a 2024 Nature study, researchers manually reviewed 150 factual claims in AI-generated biographies and found that 45 were incorrect—a 30% error rate in that particular test. The researchers also found that generating multiple answers and comparing their meanings can help expose uncertainty (Nature).

For an indie developer, the dangerous mistakes are rarely dramatic. They look like ordinary sentences:

• “Your subscription has already been cancelled.”
• “The refund will arrive within three days.”
• “This feature is available on every plan.”
• “You can restore the deleted project from Settings.”

Each statement may sound reasonable while contradicting the customer’s message, your documentation, an earlier reply, or another sentence in the same draft.

You do not need an enterprise review system to catch most of these problems. You need five focused minutes and a repeatable checklist.

What counts as a contradiction in an AI reply?

A contradiction is more than two sentences directly disagreeing. In customer support, it includes any claim that conflicts with the information the reply is supposed to respect.

Check for five types.

1. Internal contradictions

The reply disagrees with itself.

“Your account remains active until August 31. You will lose access immediately after cancelling today.”

Both statements might be valid under different billing rules, but they cannot both describe the same account without further explanation.

2. Customer-context contradictions

The draft conflicts with what the customer said.

The customer writes:

“I can sign in on the web, but the Android app rejects my password.”

The AI replies:

“Since you cannot access your account on any device, please reset your password.”

The response has quietly changed the problem. A password reset may waste the customer’s time and hide the real mobile-app issue.

3. Policy contradictions

The reply conflicts with your current pricing, refund, privacy, or support policy.

For example, your documentation says that refunds are available within 14 days, but the AI promises one after 30 days. This is especially risky because the customer may treat the reply as an official commitment.

That risk is not theoretical. In 2024, a Canadian tribunal ordered Air Canada to pay CA$812.02 after its chatbot provided incorrect information about a bereavement-fare policy (Moffatt v. Air Canada case summary).

4. Timeline and number contradictions

Dates, plan limits, prices, percentages, version numbers, and waiting periods are easy to overlook.

A draft might say:

• the trial ends on Friday, even though the account shows Thursday;
• the customer has used 8 of 10 seats, then describe 3 seats as available;
• a refund takes 5–7 business days in one paragraph and “up to 14 days” in the next;
• a bug was fixed in version 4.2, although the release notes list version 4.3.

Numbers deserve a separate pass because the surrounding prose can make them appear trustworthy.

5. Action contradictions

The explanation and recommended action do not match.

The draft correctly identifies a server-side outage but then asks the customer to reinstall the app. Or it says no data was lost while instructing the customer to recreate missing records manually.

A useful reply should connect three things cleanly:

What happened → what is true now → what the customer should do next

If those parts do not agree, the reply needs revision.

The five-minute contradiction check

Use a timer at first. After several reviews, the process becomes almost automatic.

0:00–0:45: Extract the hard claims

Ignore tone and grammar. Scan the draft only for statements that could be proven wrong.

Mark every claim about:

• account or payment status;
• prices, limits, and plan features;
• dates and processing times;
• product behavior;
• previous customer actions;
• company policies;
• data loss, security, or privacy;
• actions your team has supposedly completed;
• promised outcomes.

A sentence such as “I understand how frustrating this is” does not require verification. “I have restored your account” absolutely does.

A quick technique is to highlight nouns, numbers, and verbs that imply certainty: is, will, has, completed, supports, includes, refunded, fixed.

0:45–1:30: Compare the draft with the customer’s message

Read the original request again before checking external sources. You are looking for facts the AI silently added, removed, or changed.

Ask:

• Did the reply preserve the customer’s device, operating system, and app version?
• Did it keep separate accounts, projects, invoices, or dates separate?
• Did it turn a question into a confirmed diagnosis?
• Did it claim the customer performed a step they only considered performing?
• Did it overlook a test the customer already tried?

For long threads, compare the draft with the latest customer message and the last concrete promise from your team. Old context can easily override newer information.

1:30–2:30: Check the source of truth

Open the smallest authoritative source that can confirm the marked claims. That might be:

• the customer’s account record;
• your current pricing page;
• refund and cancellation rules;
• release notes;
• a status page;
• the relevant knowledge-base article;
• an internal bug ticket;
• the app-store version history.

Do not verify a policy by asking the same AI to remember it. Compare the draft with the actual policy.

This is where a grounded support workflow helps. An assistant such as SupportMe can draft from a product knowledge base and previous conversations, but the important design choice is still human review: nothing should be sent until the person responsible has approved the factual claims.

The knowledge base also needs a clear owner and current information. Retrieval cannot rescue a reply if two source documents contain different refund rules.

2:30–3:15: Run the “cannot both be true” test

Read each pair of related claims and complete this sentence:

“These cannot both be true if…”

For example:

“The account is cancelled, and the next renewal will be charged. These cannot both be true if cancellation stops future renewals immediately.”

This forces you to identify the missing condition. Sometimes you will find an actual contradiction. Other times, you will discover that the reply only needs a qualifier:

“Your cancellation is scheduled for the end of the billing period, so the account remains active until August 31. You will not be charged again.”

Pay particular attention to absolute words:

• always;
• never;
• every;
• immediately;
• guaranteed;
• fully;
• permanently;
• only.

AI drafts often choose certainty when the evidence supports a narrower statement.

3:15–4:15: Challenge the risky claims

Now interrogate the draft. You can do this manually or ask an AI tool to act as a critic.

Use a prompt like this:

Review the support draft against the customer message and policy excerpt.

List:
1. Claims that directly contradict another statement.
2. Claims unsupported by the supplied information.
3. Dates, numbers, plan limits, or promises that require verification.
4. Instructions that do not follow from the stated diagnosis.

Quote the conflicting phrases. Do not rewrite the reply and do not assume missing facts.

Provide the customer message, draft, and relevant policy excerpt below the prompt.

This second AI pass is useful because varying or conflicting answers can signal model uncertainty. The Nature researchers reported semantic-entropy detection performance of roughly 0.78 to 0.81 AUROC across the model families and scales they studied. However, they also stressed that their method does not guarantee factuality. A model can repeat the same wrong claim consistently (Nature).

Treat an AI self-check as an alarm, not a verdict.

4:15–5:00: Fix, qualify, or escalate

Every questionable claim should end in one of three buckets.

• Fix it: You have an authoritative source and can replace the incorrect statement.
• Qualify it: The claim is plausible but not confirmed. Use wording such as “It looks like,” “Based on the account status,” or “This usually takes…”
• Escalate it: The answer affects billing, security, data loss, legal commitments, or another area where guessing is unacceptable.

When evidence is missing, a short honest reply is better than a complete-sounding invention:

“I can see the cancellation request, but I cannot confirm whether the renewal charge has already been submitted. I’m checking that before promising a refund.”

That sentence may feel less efficient than an instant answer. It is far more useful than sending the wrong promise quickly.

A worked example

Imagine a customer writes:

“I downgraded from Pro to Starter yesterday, but today’s invoice still shows €29. Will I lose access to my private projects?”

The AI draft says:

“Your downgrade took effect immediately, so the €29 charge will be refunded within three business days. Your private projects will remain available on the Starter plan.”

The reply contains at least four claims requiring verification:

1. The downgrade took effect immediately.
2. The charge qualifies for a refund.
3. The refund takes three business days.
4. Starter supports private projects.

Suppose your billing documentation says downgrades take effect at the next renewal, refunds require manual approval, and Starter projects must be public. Nearly every useful-looking sentence in the draft contradicts the source of truth.

A safer reply would be:

“Your downgrade is scheduled for the next renewal, which is why the current invoice still shows the Pro price. I have not initiated a refund. Before the downgrade takes effect, you will need to make your private projects public or move them to another Pro workspace.”

The revision does not merely remove false statements. It aligns the billing status, plan rules, and next action.

Why confident wording deserves extra attention

The US National Institute of Standards and Technology uses the term confabulation for AI systems that “generate and confidently present erroneous or false content in response to prompts.” Its guidance explicitly includes outputs that contradict earlier statements in the same context (NIST Generative AI Profile).

Confidence is therefore not evidence. In fact, smooth writing can make review harder because your attention shifts toward style instead of substance.

A practical review order is:

1. Facts
2. Policy
3. Logic
4. Requested action
5. Tone and grammar

Do not start by polishing an opening sentence. First make sure the reply is true.

Pros and cons of the five-minute method

Advantages

• It catches common errors without a complicated quality-assurance workflow.
• It concentrates effort on checkable claims instead of reviewing every word equally.
• It works with email, app-store replies, and in-app support messages.
• It creates useful editing feedback for systems that learn from revisions.
• It keeps the founder or support owner responsible for customer-facing promises.

SupportMe’s diff-based learning is relevant here: when you change an invented certainty into a qualified statement, that edit can teach the system both your writing style and how you prefer uncertainty to be handled.

Limitations

• Five minutes may not be enough for security, legal, medical, or high-value billing cases.
• Internal consistency does not prove that an answer is factually correct.
• A knowledge base can contain stale or mutually inconsistent documents.
• Asking the original model to review itself may reproduce the same mistake.
• The process depends on having access to a reliable source of truth.

Current benchmarks also vary enormously by task and evaluation method. Stanford’s 2026 AI Index reports hallucination rates ranging from 22% to 94% across 26 leading models in one new accuracy benchmark (Stanford HAI). That range is a reminder that there is no universal “AI accuracy rate.” Your own support topics, documentation, prompts, and review process matter more than a vendor’s headline number.

A compact checklist for daily use

Before sending an AI-drafted reply, ask:

• Does it contradict the customer’s message?
• Does one paragraph contradict another?
• Are all dates, prices, limits, and timeframes verified?
• Does it promise an action that has not happened?
• Does it match the latest policy and product documentation?
• Does the proposed fix follow from the stated cause?
• Has uncertainty been presented as certainty?
• Would I be comfortable treating every sentence as an official company statement?

If any answer causes hesitation, pause the reply and check the relevant source.

Final takeaway

Contradictions hide inside plausible sentences. The fastest way to find them is to extract hard claims, compare them with the customer’s actual message, verify them against one authoritative source, and challenge anything that sounds more certain than the evidence allows.

AI can handle the first draft. Consistency still needs an owner.