RAG · ARCHITECTURE AND EVALUATION
RAG assistant architecture, evaluation, and limitations
RAG is more than a vector database and a model. A dependable assistant needs source and permission governance, reproducible document processing, retrieval, evidence-backed generation, a versioned evaluation set, and operational monitoring.
01 · DIRECT ANSWER
Direct answer
Build a RAG assistant as a testable chain—source, processing, retrieval, context, answer, citation—and evaluate retrieval separately from generation. RAG is useful for questions over a changing document corpus, but it does not eliminate hallucinations, repair bad sources, or replace human judgment in high-impact workflows.
02 · CRITERIA
Criteria before development
Apply these criteria to the specific workflow, data, and cost of error. They are not a universal readiness checklist.
Governed corpus
Document owners, permissions, versions, update dates, and deletion rules are known. An unmanaged folder does not become a reliable knowledge base merely because it is indexed.
Suitable question type
The user needs explanation or synthesis from textual sources. An exact record, amount, or status should come from the system of record rather than a probabilistic answer.
Verifiable answer
A citation opens a specific passage the user is allowed to see, and the interface separates retrieved evidence from model inference.
Known cost of error
Workflows are grouped by risk, with abstention, clarification, and human escalation wherever the answer cannot be safely assumed.
Evaluation set
Real questions have expected sources, useful-answer criteria, and unacceptable errors. The set is versioned with the corpus.
03 · PROCESS
Validation sequence
- 01
Map sources and access
Record systems, owners, formats, update frequency, data sensitivity, and permission rules before choosing the retrieval stack.
- 02
Build reproducible ingestion
Configure extraction, cleaning, chunking, metadata, versioning, and deletion so every retrieved passage can be traced to its source.
- 03
Evaluate retrieval first
Confirm that relevant passages reach the context on the evaluation set. A generator cannot reliably compensate for a systematically missed document.
- 04
Add generation and abstention
Generate from the available context, display citations, and define behaviour when sources are missing or contradictory.
- 05
Evaluate and observe by version
Compare retrieval, model, and prompt changes on the same set; in operation, monitor quality, freshness, latency, cost, and access violations.
04 · DECISION
Decision table
Architecture depends on the type of truth involved and the action a user will take after receiving the answer.
| Workflow | Approach | Boundary |
|---|---|---|
| An exact record, amount, or current status is required | Query an API, database, or search index | A structured value should not be paraphrased probabilistically. |
| An answer is needed from changing documents | RAG with citations and versions | The external corpus can be updated and traced independently of the model. |
| The system must perform an action | Separate authorised tool or API layer | Knowledge retrieval and permission to change data carry different risks. |
| Sources do not support an answer | Abstain or ask for clarification | A plausible model completion is not a retrieved fact. |
| The answer affects a high-impact decision | Assist with specialist review | A citation aids verification but does not transfer responsibility to the system. |
05 · MEASUREMENT
What to measure
Record the baseline workflow and collection method first. Then compare equivalent scenarios without presenting a planned target as measured impact.
Retrieval coverage
For questions with known sources, test whether the required passage reaches the results and context, and analyse why misses occur.
Context relevance
Assess whether each retrieved passage helps answer the question and whether noise displaces useful evidence.
Answer grounding
Map each material claim to cited context; track contradictions and unsupported inferences separately.
Task usefulness
A domain reviewer checks correctness, completeness, appropriate abstention, and whether the user can take the next step—not just writing style.
Operational layer
Monitor index freshness, ingestion failures, latency, cost, abstentions, and any sign that content is exposed outside user permissions.
06 · BOUNDARIES
Risks
Retrieval miss
A required document may exist but fail to reach context because of processing, query wording, filters, or ranking.
Stale or conflicting source
RAG reproduces corpus problems. Versions, source precedence, and explicit conflict behaviour are needed.
Permission leakage
Filtering after retrieval may be insufficient. Authorisation should apply before context retrieval and have dedicated tests.
Document prompt injection
Corpus text may contain instructions for the model. Retrieved data must not be treated as trusted system control.
False confidence in automated evaluation
LLM judges also fail; automated measures need calibration against human ratings and repeat validation after domain changes.
WHAT CANNOT BE CLAIMED
Limitations of the conclusion
- RAG reduces reliance on a model’s parametric memory but does not eliminate hallucinations or incorrect inference.
- A citation does not prove a claim is correct or that the selected policy document is current.
- A poor, incomplete, or unlabelled corpus cannot be repaired solely by changing the model or prompt.
- Automated metrics help compare versions but do not replace expert review of high-impact answers.
- RAG should not execute actions or bypass permissions without a separate governed tool layer.
07 · PORTFOLIO
Related projects and honest stage
The related Legal Support MCP project is an MVP concept, not a live RAG case. It is included as an example of source requirements and mandatory human escalation.
Legal Support MCP
The MVP scope is documented: an MCP server for rules, sources, templates, and versions, plus a thin AI-client skill. The intended output is a structured package of prioritised remarks, questions, source links, and draft wording, with material or ambiguous terms escalated to a lawyer.
Open project review08 · METHODOLOGY
Primary sources
These sources support the methodology and definitions. They do not validate AI SaaS Solution project outcomes.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Open sourceRAGAS: Automated Evaluation of Retrieval Augmented Generation
Open sourceARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Open sourceArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
Open source