LLM & Agentic AI

How Enterprise RAG Actually Works

Follow enterprise documents from permission-aware indexing to evidence-based answers, and see why abstention belongs in the design.

Reference architecture - illustrative, not a client deployment

Trace how enterprise documents are indexed with permissions, retrieved for an authenticated caller, checked for grounding and cited, or refused when the evidence is too weak.

Complexity
Intermediate
Drawing
Supplied
Listing
Featured
  • RAG
  • Enterprise AI
  • Search
  • Security
  • LLM
  • Azure AI Search

Overview

This reference architecture is illustrative. It is not a record of a client deployment.

Enterprise RAG separates two paths. During ingestion, documents are cleaned, chunked and embedded while source references and access-control metadata are retained in a search index. At request time, an authenticated question triggers permission-filtered retrieval, optional reranking and context assembly. The selected evidence reaches the language model; retrieval does not update model weights.

The draft then passes grounding and citation checks. These checks can catch unsupported claims or broken references, but they do not prove truth. If authorized evidence is absent or inadequate, the safe result is an explicit abstention. Microsoft documents both native document-level controls and a security-filter pattern; the latter uses identity strings in query filters and therefore still depends on the application establishing a trustworthy caller identity.

A concrete example

An employee asks for the international-travel reimbursement limit. The system searches only policy passages that employee may access. If a current passage establishes the applicable limit, the answer cites it. If the indexed policy has no rule for that country or employee group, the assistant states that the evidence is insufficient and points to the policy owner.

Common misconception

"We uploaded our PDFs into the model." The documents remain in an external retrieval system. Only selected passages enter the prompt for a particular request.

Diagram & Walkthrough

Choose a declared scenario and step, or select a component by pointer, touch or keyboard. Playback starts only when you press Play and pauses when this panel leaves the screen.

100%
How Enterprise RAG Actually Works Separate document indexing, authorized retrieval and answer generation. Ingestion runs ahead of time; at query time retrieval is permission-filtered, evidence is reranked, an answer is generated and checked for grounding and citations, or the system abstains. Retrieval does not update model weights; validation reduces risk but is not a proof of truth. How Enterprise RAG Actually Works Separate document indexing, authorized retrieval and answer generation. INGESTION · OFFLINE REQUEST PATH · QUERY TIME RESPONSE CHECKS Documents + access rules Chunks + vectors + ACLs Question + identity Permission-filtered search Authorized candidates Retrieved evidence Question + context Draft + citations Validated answer Insufficient evidence Documents +permissions data Clean, chunk, embed deterministic Search index +ACL metadata data User + identity human Authorized hybridretrieval deterministic Rerank +assemble evidence deterministic Answer generation model Grounding +citation checks deterministic Answer orabstention external Main path Data flow Control flow Feedback flow Retrieval never updates model weights. Checks reduce risk; they do not prove truth.

Main walkthrough, step 1 of 6: Index content with its permissions. idle.

Main walkthrough

The primary path through this reference architecture.

Active parts: Documents + permissions, Clean, chunk, embed, Search index + ACL metadata

Text walkthrough

This vertical reference remains readable without the interactive controls.

  1. 01Index content with its permissions

    Clean and chunk source documents, generate embeddings and store each chunk with its source and access-control metadata.

    Active parts: Documents + permissions, Clean, chunk, embed, Search index + ACL metadata

  2. 02Receive an authenticated question

    Receive the question with identity established by a trusted application layer, not by text supplied in the request.

    Active parts: User + identity, Authorized hybrid retrieval

  3. 03Retrieve only authorized evidence

    Search with the caller's enforced permission scope and return only candidates that this identity may access.

    Active parts: Authorized hybrid retrieval, Search index + ACL metadata

  4. 04Prepare the context

    Rerank the permitted candidates when useful, apply the context budget and retain source references for citations.

    Active parts: Authorized hybrid retrieval, Rerank + assemble evidence

  5. 05Generate and check an answer

    Generate a draft from the selected context, then check whether its material claims and citation references are supported.

    Active parts: Rerank + assemble evidence, Answer generation, Grounding + citation checks

  6. 06Answer or abstain

    Return the checked answer when evidence supports it, or abstain when retrieval cannot supply sufficient authorized evidence.

    Active parts: Authorized hybrid retrieval, Grounding + citation checks, Answer or abstention

Connections

  • Documents + access rulesDocuments + permissions to Clean, chunk, embeddata
  • Chunks + vectors + ACLsClean, chunk, embed to Search index + ACL metadatadata
  • Question + identityUser + identity to Authorized hybrid retrievaldata
  • Permission-filtered searchAuthorized hybrid retrieval to Search index + ACL metadatacontrol
  • Authorized candidatesSearch index + ACL metadata to Authorized hybrid retrievaldata
  • Retrieved evidenceAuthorized hybrid retrieval to Rerank + assemble evidencedata
  • Question + contextRerank + assemble evidence to Answer generationdata
  • Draft + citationsAnswer generation to Grounding + citation checksdata
  • Validated answerGrounding + citation checks to Answer or abstentiondata
  • Insufficient evidenceAuthorized hybrid retrieval to Answer or abstentioncontrol

Components

Documents + permissionsdata

Authoritative source documents together with classification, ownership and ACLs.

Responsibilities

  • Authoritative source documents together with classification, ownership and ACLs.

Inputs

  • Documents
  • Source-system access rules

Outputs

  • Content and permission metadata

Risks

  • Untracked deletions
  • Stale permission synchronization
Clean, chunk, embeddeterministic

Prepare, version and transform documents into searchable chunks and vectors.

Responsibilities

  • Prepare, version and transform documents into searchable chunks and vectors.

Inputs

  • Source content
  • ACL metadata

Outputs

  • Chunks
  • Embeddings
  • Indexable metadata

Risks

  • Loss of context during chunking
  • Index lag
Search index + ACL metadatadata

Store searchable text, vectors, source references and permission data.

Responsibilities

  • Store searchable text, vectors, source references and permission data.

Inputs

  • Chunks
  • Vectors
  • Permissions

Outputs

  • Permitted search hits

Risks

  • Outdated documents
  • Incorrect permission fields
User + identityhuman

Submit a question in a verified identity context.

Responsibilities

  • Submit a question in a verified identity context.

Inputs

  • Authentication

Outputs

  • Question
  • Trusted identity claims

Risks

  • Impersonation
  • Unauthorized group assumptions
Authorized hybrid retrievaldeterministic

Enforce permissions and retrieve lexical/semantic candidates for the specific caller.

Responsibilities

  • Enforce permissions and retrieve lexical/semantic candidates for the specific caller.

Inputs

  • Question
  • Trusted identity
  • Search index

Outputs

  • Authorized candidate evidence

Risks

  • Access-control bypass
  • Low recall
Rerank + assemble evidencedeterministic

Select and package relevant passages with source references within token limits.

Responsibilities

  • Select and package relevant passages with source references within token limits.

Inputs

  • Authorized retrieved passages

Outputs

  • Bounded evidence context

Risks

  • Missing critical qualifiers
  • Prompt injection in retrieved text
Answer generationmodel

Generate a proposed answer based on the question and selected sources.

Responsibilities

  • Generate a proposed answer based on the question and selected sources.

Inputs

  • Question
  • Evidence context

Outputs

  • Draft answer and citations

Risks

  • Hallucinated statements
  • Unsupported citations
Grounding + citation checksdeterministic

Validate reference resolution and test claim support before publishing.

Responsibilities

  • Validate reference resolution and test claim support before publishing.

Inputs

  • Draft
  • Evidence references

Outputs

  • Pass/fail decision
  • Validated draft when acceptable

Risks

  • False-positive checks
  • Checker-model errors
Answer or abstentionexternal

Deliver a supported answer or state that evidence is insufficient.

Responsibilities

  • Deliver a supported answer or state that evidence is insufficient.

Inputs

  • Approved answer or abstention policy

Outputs

  • User-visible response

Risks

  • Overstated certainty
  • Poor explanation of limitations

Scenario reference

No supporting evidence

Do not invent support or imply that abstention eliminates all hallucinations.

  1. 01Search and abstain

    The authenticated question reaches retrieval, but the returned candidates do not provide enough support, so the response follows the abstention path.

    Active parts: User + identity, Authorized hybrid retrieval, Search index + ACL metadata, Answer or abstention

Design Decisions

  • Enforce authorization during retrieval, before restricted text can enter model context.
  • Preserve document IDs, versions and passage references for traceable citations.
  • Evaluate lexical search, vector search and reranking separately on representative questions.
  • Define evidence thresholds and a no-answer path instead of relying on model confidence.

Risks & Trade-offs

Stale permissions can expose restricted text; stale indexes can cite superseded policy; poor chunking can remove qualifiers; and retrieved text can contain prompt-injection attempts. Citation formatting may look convincing even when the conclusion is unsupported. Authorization, retrieval quality, citation faithfulness, latency and cost need separate tests.

When to Use / Not Use

Use RAG when knowledge changes, answers need citations or access differs by user. Prefer a deterministic database query or conventional search for exact lookups that do not need generation. Fine-tuning can change task behavior, but it does not replace fresh, permission-aware retrieval.

Sources