The bank compliance assistant is a RAG system for relationship managers and operations staff, who ask about internal anti-money laundering (AML) and know-your-customer (KYC) policies and regulatory notices. It extends the ideas of the Fraud Detection Model.
Introduction
Thousands of policy documents are split into short passages. Two typical questions show why one retriever is not enough:
- Question A: "What must happen before we onboard a senior government official?" The answer is in the politically exposed persons (PEP) policy, which never uses these words, so meaning matters
- Question B: "What does error KYC-E052 mean?" The answer is one entry in a list of near-identical error codes, so the exact token matters
Requirements and design decisions
| Requirement | Design decision |
|---|---|
| Cite the exact clause (for example "CP-12 §4.3") | Hybrid BM25 and dense retrieval with RRF |
| Policies change | Incremental re-indexing, and a filter on effective date |
| Need-to-know access | Role-based pre-filtering in the vector store, never in the prompt |
| Audit trail | Log the query, retrieved chunk IDs and versions, prompt and answer |
| Trustworthy answers | Golden set from compliance officers, and abstain below a faithfulness threshold |
| Low latency at scale | HNSW index, cross-encoder re-ranking of the top 50, and caching |
| Say "I don't know" when unsure | Instruct the model to refuse, and escalate to compliance |
Reference architecture
An offline ingestion pipeline feeds an online serving path, and each box is a separate component to monitor.
graph LR A[Ingest] --> B[Parse and chunk] B --> C[Embed] C --> D[Index with access tags] D --> E[Retrieve and re-rank] E --> F[Generate and cite]
- offline: runs on a schedule or whenever a document changes, and every chunk carries its version, date and access metadata
- online: cache frequent queries, log the chunk IDs behind every answer, and re-run the golden-set evaluation after every change
Code
The generator receives the question and the top chunks, each labelled with its ID so that the answer can cite its source:
def build_prompt(question, chunk_ids):
"""Instructions, then retrieved chunks with their IDs, then the question."""
context = "\n".join(f"[{IDS[i]}] {CORPUS[i][1]}" for i in chunk_ids)
return (
"You are a compliance assistant. Answer using ONLY the context below.\n"
"Cite the chunk ID in square brackets for every claim. "
"If the context does not contain the answer, say you do not know.\n\n"
f"<context>\n{context}\n</context>\n\nQuestion: {question}\nAnswer:"
)Notes:
CORPUSis a list of (ID, text) pairs andIDSholds the chunk IDs- the instruction "answer using ONLY the context" and the instruction to say "I do not know" are what make the answer grounded
- the retrieved text goes inside
<context>tags so that it is marked as data - a simple generator that only reads the top-ranked chunk can answer Question A from the PEP definition instead of the onboarding rule, which is why re-ranking is added
Advantages and limitations of production RAG
| Advantages | Limitations |
|---|---|
| Knowledge is updated by re-indexing, not re-training | Quality is capped by retrieval recall |
| Answers cite their sources, which supports audit | Many moving parts: chunking, index, re-ranker, prompts |
| Access control is enforced per user and per document | Evaluation needs a golden set and human calibration |
| Smaller models perform well with good context | New attack surface, since whatever is indexed can reach the prompt |
The last limitation is a real attack, see Indirect Prompt Injection.