AI ENGINEERING / RAG
Building a Production-Ready RAG System on Azure
From chunking and embeddings to access control and evaluation.
Key Insight: Azure gives you every piece needed for a production RAG system — search, embeddings, identity, evaluation tooling — but assembling them into something reliable is still an architecture job, not a checklist of services to enable.
Azure's AI ecosystem has matured to the point where you rarely need to build core RAG infrastructure from scratch. The harder work is wiring the pieces together correctly, especially around access control and evaluation, which is where most Azure RAG implementations I've reviewed fall short.
Storage and indexing: Azure AI Search as the retrieval layer
Azure AI Search is the natural default for the retrieval layer — it supports vector search, keyword (BM25) search, and hybrid queries combining both natively, which matters because pure vector search alone under-serves exact-match queries like product codes or policy numbers. Set up your index with both a vector field for embeddings and standard searchable text fields, and use hybrid queries by default rather than vector-only. Semantic ranking, layered on top as a reranking step, meaningfully improves relevance for the top results the LLM actually sees.
Embeddings: pick the model deliberately, and plan for change
Azure OpenAI's embedding models are a solid default, but the decision that matters more than which model is: how will you handle re-embedding when you inevitably change models or chunking strategy? Store the model version alongside each embedded chunk, and design the ingestion pipeline to be re-runnable — you will want to re-index at some point, and doing it as a one-off emergency script instead of a repeatable pipeline step is a common and avoidable pain point.
Access control: Entra ID at the retrieval layer, not just the app layer
This is where I see the most Azure RAG implementations fail a security review. It's common to secure the *application* with Entra ID (Azure AD) — users sign in, the app is protected — while the retrieval layer behind it queries a single shared index with no per-document permission check. That means anyone who can use the assistant can potentially have it surface content from documents they weren't supposed to see, just by asking the right question.
The fix is to model document-level permissions into the index itself — typically a field capturing which security groups or users can see each document — and filter every retrieval query by the calling user's Entra ID group membership, enforced server-side, not trusted to the client. This has to be designed in from the start; retrofitting per-document access control onto an existing index and ingestion pipeline is a substantial rework.
User (Entra ID) → App → Retrieval query filter: doc.allowedGroups ∩ user.groups ≠ ∅ → Azure AI Search (hybrid + semantic rank) → Azure OpenAI (grounded answer)Orchestration: keep it simple until complexity is earned
Azure gives you options from a simple orchestration layer you write yourself, to frameworks like Semantic Kernel or Prompt Flow. For a straightforward retrieve-then-generate pattern, a simple, explicit pipeline you control is often easier to debug and reason about than a heavier framework. Reach for the more structured tooling when you need multi-step reasoning, tool calling, or you're managing enough complexity that the framework's scaffolding earns its cost.
Evaluation: Azure AI Studio's evaluation tools, used deliberately
Azure AI Studio provides built-in evaluators for groundedness, relevance, and coherence — use them, but don't treat automated scoring as sufficient on its own. Build a golden dataset from real, representative questions your users actually ask (or will ask), with known-correct source documents, and track retrieval precision and recall against it as you iterate on chunking and indexing strategy. Automated groundedness scoring catches a real and common failure mode — the model answering fluently from its own training data instead of the retrieved context — and it's worth wiring into your CI pipeline so a regression in grounding gets caught before it ships.
Key takeaways
Use Azure AI Search's hybrid and semantic ranking capabilities rather than vector search alone. Version your embeddings and design ingestion to be re-runnable. Enforce document-level access control inside the retrieval query using Entra ID group membership — not just at the application's front door. Keep orchestration as simple as the problem allows. Build a real evaluation set in Azure AI Studio before launch, and wire groundedness checks into your deployment pipeline so quality regressions are caught automatically.