Grounding starts with evidence
A model without access to your systems cannot establish your current deployment rate or internal policy. Retrieval-augmented generation (RAG) supplies external context at request time. It can improve answers, but stale evidence, missed documents, or unsupported generation can still produce errors.
Implementation reference: Anthropic contextual retrieval combines contextualized chunks, lexical search, embeddings, and reranking. Evaluate the combination on your own corpus; its reported benchmark results are not a universal guarantee.
Where MCP and agents fit
MCP: connection protocol
MCP standardizes access to tools, resources, and prompts. A RAG retriever can be exposed through MCP; the protocol itself does not verify an answer or guarantee fresh data. Check authorization and tool-result timestamps.
Agent: controlled iteration
An agent can choose tools and gather more evidence. Set step, time, and cost limits, isolate untrusted retrieved instructions, and require approval for consequential actions.
Multiple agents: task split
Use specialist roles when parallel work or independent checks help. More agents add coordination cost and can share the same wrong assumptions. Compare quality and latency with a simpler workflow.
Protocol reference: Model Context Protocol introduction.
Caching: reuse the right layer
These caches solve different problems. Start with the repeated work you can safely reuse; do not add every layer by default.
| Layer | What is reused | What still happens |
|---|---|---|
| Prompt / prefix cache | Provider-side computation for a matching prompt prefix, such as stable instructions or reference text. | The model generates a new response. Follow provider rules for prefix identity, minimum length, retention, and model support. |
| Retrieval-result cache | Document IDs or chunks for a query plus its search filters, access scope, and index version. | Validate permissions and freshness, then generate an answer. Corpus or permission changes can invalidate a hit. |
| Exact answer cache | A previous response for the same complete request key, including relevant conversation and configuration. | Serve only after current access and validity checks; generation can be skipped. Identical text alone is not a sufficient key. |
| Semantic answer cache | A response to an equivalent question found through embedding similarity and strict metadata filters. | Check task equivalence and scope. Nearness in vector space can hide different dates, entities, negations, or requested actions. |
Primary references: Claude prompt caching for reusable prefixes; Redis semantic cache for response reuse with thresholds and metadata boundaries. Actual savings depend on reuse, write cost, workload, and retention.
CAG / long-context reuse
Cache-augmented generation can preload a bounded reference set into a long-context model and reuse its cached state across questions. It is a candidate for stable knowledge that fits the context budget. Large or rapidly changing corpora still need a retrieval or refresh strategy; answer quality and update cost require evaluation. Read the CAG paper.
Cache validity and safe defaults
- Scope before lookup: derive tenant, user permissions, and relevant filters from trusted server identity. Recheck authorization on hits; a client-supplied tenant label is not access control.
- Version the dependencies: include corpus/index, prompt, model, retriever, and policy versions where relevant. Purge or stop serving entries affected by updates, deletions, or revoked access.
- Use bounded lifetimes: choose TTL from the source freshness requirement, cap memory, and track eviction. TTL limits age but does not guarantee correctness between updates.
- Bypass sensitive or volatile work: do not reuse an action execution result as a new action. Live incident state, personal data, and high-impact decisions may require fresh reads and additional checks.
- Test semantic false hits: include near-matches that need different answers. Keep conversation state and source citations attached to entries; reject ambiguous reuse.
Quality, freshness, and operations
Access and trust
Apply document filters before context reaches the model. Treat retrieved text as data, not instructions. Enforce tool permissions separately and record a bounded audit trail without exposing sensitive prompts.
Evaluate the pipeline
Test retrieval relevance, grounded claims, citation support, completeness, and correct abstention. Include missing evidence, conflicting versions, prompt injection, and cross-tenant requests. Model judges help, with human-reviewed samples.
Measure real behavior
Track p50/p95 latency, cost per accepted answer, cache hit rate, stale or incorrect hits, and invalidation failures. Compare cold and warm paths. If a source fails, label any stale fallback or ask for verification.
Primary references: Azure AI Search security filters and Microsoft RAG evaluators. A citation makes a claim inspectable; it does not prove that the source or answer is correct.
Grounding Response Lab
Compare illustrative responses to the same question. These patterns can work together; they are not successive levels of guaranteed accuracy.
Try scoped answer caching
Run the same sample twice to see a miss followed by a hit. Change the scope or document version to create a separate key; advance time to see TTL expiration.
Ready 0 hits · 0 misses · 30-second TTL
Select a sample and run it.