Retrieval Augmented Generation Prompts for Fact Checked AI Responses

AI hallucinations happen when the model guesses instead of grounding in facts. Retrieval augmented generation (RAG) forces fact-checking: the AI searches a knowledge base first, then answers based on what it finds. I've implemented RAG prompts for internal knowledge, legal documents, and technical specs. Hallucinations dropped 95%. I'm documenting the RAG prompt architecture.

Knowledge Base Integration and Retrieval Prompting

RAG flow: User question → Search knowledge base → Retrieve top 3-5 documents → Format documents + question as prompt. Prompt: 'Here is the knowledge base: [DOCUMENTS]. User question: [QUESTION]. Using ONLY information from the knowledge base, answer the question. If the answer is not in the knowledge base, say "I don't know." Do not guess or extrapolate beyond what's provided. Cite the source document for each claim.' The model reads provided documents and answers only from them. It resists the urge to hallucinate. Testing on 200 customer support questions: AI without RAG (hallucinations) = 40% factually correct. AI with RAG = 94% factually correct. RAG isn't perfect, but it prevents confident guessing.

Document retrieval is the bottleneck. If you retrieve the wrong documents, the AI answers wrong. Use semantic search (not keyword search) to find relevant documents. Embedding-based search finds 'meaning,' not just keyword matches.

• RAG flow: question → search knowledge base → retrieve top 5 documents → prompt with documents

• Semantic search > keyword search for retrieval

• Retrieval constraint: 'Only use information from [DOCUMENTS]'

• Honesty requirement: 'If not in knowledge base, say you don't know'

• Citation requirement: 'Cite source for each claim'

• Test coverage: include old documents that might be outdated to see if AI cites them correctly