Everyone wants to put a chatbot on top of their own documents: point a model at your handbook, your tickets, your contracts, and answer any question instantly. The technique is retrieval-augmented generation, RAG, and it works, but the part that decides whether it works is not the model. It is your data. RAG retrieves chunks of your documents and hands them to a model to answer from, so the answer can only be as good as what retrieval surfaces, and retrieval is only as good as how your data was prepared. The model is a rented commodity anyone can buy. The disciplined corpus underneath is the part that is yours, and the part that is work. Here is a readiness checklist to run before you spend a dollar on it, each item marked with why it matters and what breaks if you skip it. Score yourself honestly and you will know whether you are ready to build or still need to clean house.
1. Your documents are actually readable by a machine
Machine-readable source. The source documents are clean, parseable text with their structure intact: headings, lists, and tables preserved, not a pile of scanned PDFs, screenshots, and exported slide decks. Tables and figures survive the conversion to text.
Why it is the foundation. RAG reads text, so anything the parser mangles never reaches retrieval. A scanned contract with no text layer, a table flattened into a wall of numbers, a heading lost in conversion: all of it becomes noise the system skips or misreads. Databricks, in its retrieval-quality guide, is blunt: “Poor parsing (missing tables, broken formatting) directly impacts retrieval quality” (Microsoft Learn, updated June 2026, a platform vendor that also sells the parser). This is the “garbage in” law of every data project, and it bites first here because no human is reading the page to compensate. If your knowledge lives mostly in formats a machine cannot cleanly read, that is the first project, before any model.
2. The chunking keeps each idea whole
Meaningful chunks. Documents are split into retrieval chunks that respect meaning: a chunk holds a complete thought, a question with its answer, a clause with its definition, not a paragraph guillotined at a fixed character count.
Why the boring step decides accuracy. RAG does not retrieve documents; it retrieves chunks, and a question whose answer is split across two chunks is one the system can miss even though the document holds it. A 2025 study accepted to the ACM SIGIR conference frames chunking as a measurable bottleneck in RAG quality and builds a whole evaluation method around it (Brådland et al., “A New HOPE,” arXiv May 2025), which tells you the field treats chunk quality as load-bearing, not cosmetic. Databricks advises against over-tuning chunk size and toward enriching chunks with structure and metadata, noting smaller chunks localize facts while larger ones preserve context (Microsoft Learn, June 2026). The failure mode is quiet: fluent answers that are subtly incomplete because the evidence was severed in preprocessing.
3. Every chunk carries metadata you can filter on
Filterable metadata. Each chunk is tagged with the metadata that matters for your use: source document, date, department, product, version, sensitivity. Retrieval can filter on those tags, not just match on meaning.
Why this is the highest-leverage step. Semantic similarity alone retrieves things that sound related but are wrong: last year’s pricing, another region’s policy, a draft superseded months ago. Metadata filtering cuts the search to the right slice before similarity ever runs. Databricks calls it plainly “your biggest lever for retrieval quality,” noting it can shrink the search space by ninety percent or more (Microsoft Learn, June 2026). Skip it and you inherit the most common quiet defect in document RAG: the system confidently answers from a real document that simply does not apply to the question asked.
4. Permissions follow the data into the index
Enforced permissions. Access control from the source systems is carried into retrieval, so a user only ever gets chunks they were already allowed to see. Permission checks happen in the retrieval query, not bolted on after the model has already read the text.
Why this is the one that becomes a headline. Vector databases do not enforce your file permissions by default. As one security firm put it, “RAG does not natively support access control,” and without permission metadata “any user query will be compared against all vectors in the database” (Lasso Security, August 2024). A vector-database vendor’s own guide agrees, warning that “if different users have different levels of access to data, as they do in most real-world systems, your RAG pipeline must enforce those access boundaries” (Pinecone, January 2026, naturally self-interested as a seller of the index). The failure mode is a leak: an assistant that cheerfully surfaces a salary sheet or a board memo to whoever knows how to ask, because the model never saw the access rule. If your file permissions are already a mess, RAG does not fix that; it gives it a search box.
5. Personal information is handled, not just hoovered up
Governed personal information. Before ingestion, the corpus is screened for personal information. What is not needed for the use case is removed or de-identified; what stays is governed by your privacy obligations, with consent and purpose limits intact and a third-party processing agreement in place for the AI provider.
Why the Canadian angle is sharper than people expect. Under PIPEDA, an organization “is responsible for personal information under its control,” must protect it “by appropriate security relative to the sensitivity,” and “unless the individual consents otherwise or it is required by law, personal information can only be used or disclosed for the purposes for which it was collected” (Office of the Privacy Commissioner of Canada, last modified May 2025). Loading customer records into a vector store and a model’s context is a new use, and de-identification is not a free pass: the OPC’s 2024 cross-jurisdiction study notes that under Quebec’s privacy law and Canada’s proposed federal reform (the CPPA), de-identified information is still treated as personal information, because true anonymization requires that “no individual can be identified … whether directly or indirectly,” and “it may be difficult or impossible to fully eliminate all possibility of re-identification” (OPC, October 2024). Scrub what you can, govern what remains, and never paste regulated data into a provider you have no agreement with.
6. You have an eval before you have an answer
An eval set. A small evaluation set exists, with real questions and their correct answers, that measures retrieval and generation separately. It runs before launch and again after any change to chunking, metadata, or the model.
Why this is the gate, not a formality. You cannot improve what you do not measure, and RAG fails in two distinct places that need separate tests: retrieval can miss the right chunk, or the model can ignore the right chunk it was handed. The standard metrics name both: contextual recall, “whether the retrieval context contains all the information required to produce the ideal output,” and faithfulness, the share of the answer actually supported by what was retrieved (Confident AI, October 2025, a vendor selling an eval product). Databricks puts the eval first in its own sequence: “If you don’t have evaluation in place, stop here and set it up first. Optimizing without measurement is guesswork” (Microsoft Learn, June 2026). The failure mode is the worst kind, because it is invisible: a demo that wows the room and a production system nobody can prove is right. This is the eval-first discipline behind Signal’s worked support-triage build.
7. Someone owns keeping the index fresh
A named owner. There is an owner and a routine for updating the index when documents change: new files added, edited files re-indexed, deleted files removed. Stale or superseded chunks do not linger.
Why the moat needs maintenance. A document corpus is not a one-time upload; it drifts the moment your business does. Semantic search has no sense of time, so a stale chunk scores just as high as a current one and gets served with the same confidence. When a policy changes and the old version is still in the index, the system keeps citing the version that is wrong, and it sounds exactly as sure as when it was right. This is the cousin of the documentation discipline Signal keeps returning to, including in the compliance-shaped hole: the value is not the build, it is that someone owns it after launch. An index with no owner is a confidently wrong answer waiting for the day your documents change.
The one test to run first
Before you budget a dollar, run this: take ten real questions your team actually gets, find the answers by hand in your own documents, and time it. If a competent person, knowing where to look, struggles to find clean answers in your current files, then retrieval will struggle too, because it is reading the same mess with none of the human judgment that lets your people compensate. That single exercise tells you the truth a vendor demo will not: whether your data is a moat or a liability. If the answers are there and findable, you are ready to build. If they are not, your first project is not RAG. It is the data underneath it.