RAG Chatbot Tutorial UK: Build a Policy‑Aware Bot on Data

Step 1 — Prepare documents

• Normalise to clean text (PDF/OCR fixes, remove boilerplate, keep section headings).
• Chunk by semantic boundaries (e.g., sections 300–800 tokens) and store titles, URLs, dates, and owners as metadata.
• Redact obvious PII early; tag sensitivity levels in metadata for downstream controls.

Step 2 — Embed and index

• Choose embeddings that work well on policy/legal prose; test on a small set first.
• Store vectors with metadata in a vector database; add filters for role, department, document state, and retention.
• Combine semantic search with a lexical layer (BM25) to catch exact‑term queries.

Step 3 — Rerank and retrieve

• Use a reranker on top‑k hits to improve relevance on long policy text.
• Apply metadata guards: only retrieve chunks the user can access and that are in‑date.
• Keep retrieval summaries short; pass only the most relevant chunks to the model.

Step 4 — Policy‑aware prompting

• Instruction: “Only answer from the provided sources. If missing, say you don’t know.”
• Citations: require per‑sentence or per‑paragraph references to chunk titles/links.
• Refusals: include clear templates for out‑of‑scope, confidential, or unsafe requests.

Step 5 — Guardrails and UX

• Pre‑prompt classifier to route harmful or HR/Legal‑restricted queries to refusal flows.
• PII masking on retrieved text; redact emails, phone numbers, IDs unless policy allows.
• Session memory should store minimal context and be erasable on request.

Step 6 — Operations

• Cache frequent answers with versioning tied to document hashes for instant invalidation.
• Log retrieval hits, final answers, refusals, and user role (hashed/pseudonymised where possible).
• Add a “Was this helpful?” signal to capture real‑world feedback for retraining retrieval.

What to measure

• Retrieval quality: precision@k, coverage of ground‑truth answers, duplication rate of chunks.
• Answer quality: groundedness (citation supports claim), completeness, and clarity.
• Safety and policy: refusal accuracy for out‑of‑scope requests; leakage rate of redacted data.
• UX performance: latency p95, stability under load, helpfulness rating from pilot users.

Module: Data Intake & Labelling

• Inventory documents, owners, and retention windows.
• Tag sensitivity, department, jurisdiction, and expiry in metadata.

Module: Safe Retrieval & Access

• Role‑based filters at query time; exclude expired or restricted docs.
• PII masking and special‑category data handling.

Module: Prompting & Guardrails

• Scope‑locked instructions, refusal templates, and mandatory citations.
• Moderation/routing for unsafe or HR/Legal‑sensitive queries.

Module: Evaluation & Audit

• Gold‑set tests for groundedness and refusals; regression checks after updates.
• Audit logs, DSAR workflows, and documented model/embedding versions.

• Over‑long chunks that dilute relevance; prefer smaller, well‑titled sections.
• No citations policy—users can’t trust or challenge answers.
• Putting policy only in the prompt; enforce at retrieval and routing.
• Skipping a gold‑set; you won’t know if changes help or harm.
• Mixing draft and approved documents in one index without clear status tags.