# Hallucination (AI)

> Part of the AI Garage Jargons glossary: https://aigarage.in/jargons (all terms: https://aigarage.in/jargons.md)

**Category:** LLM
**Tags:** risk, safety, beginner

When a generative AI system produces plausible but false, unsupported, or internally inconsistent content.

**In plain terms:** The AI sounds sure, but it made things up.

**Use cases:**
- Risk reviews in legal/finance
- Fact-checking pipelines
- Human approval workflows

**Why it matters:** Fluent wording can make an unsupported claim look trustworthy. In consequential workflows, that can turn a generation error into a customer, compliance, or operational failure.

**Why now:** Newer models reduce some hallucinations, but current primary research and risk guidance still treat confident false output as an unresolved problem that requires evaluation, grounding, abstention, and human review where stakes are high.

**AI Garage example:** In an AI Garage example, Shushi, a WhatsApp AI assistant, could retrieve an approved policy passage, require the answer to point to that passage, and escalate when the source does not support a requested refund exception.

**Mental model:** A language model is built to continue with plausible language. If the available evidence stops before the answer is complete, fluent generation can continue past the evidence boundary unless the surrounding system checks support or allows the model to abstain.

**Builder scenario: Stop an unsupported refund exception**

- **Context:** A customer asks a support assistant whether a workshop booking can receive a special refund. The approved policy explains standard refunds but contains no special exception for this case.
- **Failure:** The model completes a helpful-sounding answer by inventing an exception that is not present in the approved policy.
- **Intervention:** Retrieve the relevant policy, require every consequential claim to match a supporting passage, and instruct the workflow to abstain or escalate when the evidence is missing.
- **Outcome:** The customer receives either a source-supported policy answer or a clear handoff for human review, rather than a fabricated exception.

**Common misconceptions:**
- **Claim:** Adding RAG or citations eliminates hallucinations.
  **Correction:** Retrieval and citations can reduce unsupported output, but retrieval can miss the right evidence and generated citations can themselves be wrong. The workflow must verify that the cited source actually supports the claim.

**Visual model (sequence):** Fluency can continue after evidence ends. A reliable workflow makes that boundary visible and checks claims before they reach the user.

**Visual description:** A customer question reaches an approved policy that does not contain the requested exception. Instead of continuing into an unsupported claim, the guarded workflow verifies support and abstains or escalates.

**Related concepts:** [RAG (Retrieval-Augmented Generation)](https://aigarage.in/jargons/rag), [Large Language Model](https://aigarage.in/jargons/llm), [Decision Model](https://aigarage.in/jargons/decision-model)

**Video explainer:** [Hallucination: fluency can continue after evidence ends](https://nqqbubjesbyuwrdhqxnk.supabase.co/storage/v1/object/public/aigarage-public-media/jargons/ai-hallucination/v2/ai-hallucination.mp4) (46 seconds)

See how a support assistant crosses beyond an approved policy, then checks the unsupported claim, abstains, and routes the case for human review.

**Published:** 2026-10-11T23:40:14+05:30
**Captions:** [English](https://aigarage.in/jargons/ai-hallucination/captions.vtt)
**Narration:** generated

**Transcript:** A customer asks an assistant for a special refund exception.

The approved policy covers only standard refunds.

A language model keeps writing plausible text, even when the evidence runs out.

So it may invent a rule that sounds helpful and confident.

That is a hallucination, fluent text that goes beyond what the evidence supports.

Giving the model the policy helps, but it proves nothing. Even a citation can be wrong.

A guarded workflow checks each important claim against the real passage.

Here, nothing in the policy supports the exception.

So the assistant says it is unable to confirm, and sends the case to a person.

A polished sentence is no substitute for evidence.

**Last researched:** 2026-10-11

**Sources:**
- [OpenAI - Why language models hallucinate](https://openai.com/index/why-language-models-hallucinate/)
- [NIST - Generative Artificial Intelligence Profile](https://doi.org/10.6028/NIST.AI.600-1)

**URL:** https://aigarage.in/jargons/ai-hallucination