The bank compliance assistant is a RAG system for relationship managers and operations staff, who ask about internal anti-money laundering (AML) and know-your-customer (KYC) policies and regulatory notices. It extends the ideas of the Fraud Detection Model.

Introduction

Thousands of policy documents are split into short passages. Two typical questions show why one retriever is not enough:

  • Question A: "What must happen before we onboard a senior government official?" The answer is in the politically exposed persons (PEP) policy, which never uses these words, so meaning matters
  • Question B: "What does error KYC-E052 mean?" The answer is one entry in a list of near-identical error codes, so the exact token matters

Requirements and design decisions

RequirementDesign decision
Cite the exact clause (for example "CP-12 §4.3")Hybrid BM25 and dense retrieval with RRF
Policies changeIncremental re-indexing, and a filter on effective date
Need-to-know accessRole-based pre-filtering in the vector store, never in the prompt
Audit trailLog the query, retrieved chunk IDs and versions, prompt and answer
Trustworthy answersGolden set from compliance officers, and abstain below a faithfulness threshold
Low latency at scaleHNSW index, cross-encoder re-ranking of the top 50, and caching
Say "I don't know" when unsureInstruct the model to refuse, and escalate to compliance

Reference architecture

An offline ingestion pipeline feeds an online serving path, and each box is a separate component to monitor.

graph LR
    A[Ingest] --> B[Parse and chunk]
    B --> C[Embed]
    C --> D[Index with access tags]
    D --> E[Retrieve and re-rank]
    E --> F[Generate and cite]
  • offline: runs on a schedule or whenever a document changes, and every chunk carries its version, date and access metadata
  • online: cache frequent queries, log the chunk IDs behind every answer, and re-run the golden-set evaluation after every change

Code

The generator receives the question and the top chunks, each labelled with its ID so that the answer can cite its source:

def build_prompt(question, chunk_ids):
    """Instructions, then retrieved chunks with their IDs, then the question."""
    context = "\n".join(f"[{IDS[i]}] {CORPUS[i][1]}" for i in chunk_ids)
    return (
        "You are a compliance assistant. Answer using ONLY the context below.\n"
        "Cite the chunk ID in square brackets for every claim. "
        "If the context does not contain the answer, say you do not know.\n\n"
        f"<context>\n{context}\n</context>\n\nQuestion: {question}\nAnswer:"
    )

Notes:

  • CORPUS is a list of (ID, text) pairs and IDS holds the chunk IDs
  • the instruction "answer using ONLY the context" and the instruction to say "I do not know" are what make the answer grounded
  • the retrieved text goes inside <context> tags so that it is marked as data
  • a simple generator that only reads the top-ranked chunk can answer Question A from the PEP definition instead of the onboarding rule, which is why re-ranking is added

Advantages and limitations of production RAG

AdvantagesLimitations
Knowledge is updated by re-indexing, not re-trainingQuality is capped by retrieval recall
Answers cite their sources, which supports auditMany moving parts: chunking, index, re-ranker, prompts
Access control is enforced per user and per documentEvaluation needs a golden set and human calibration
Smaller models perform well with good contextNew attack surface, since whatever is indexed can reach the prompt

The last limitation is a real attack, see Indirect Prompt Injection.