Retrieval a compliance officer can sign off on

Almost every enterprise AI project eventually becomes a retrieval project. The demo is built on a model; the production system is built on whether the right paragraph made it into the context window.

This is not a controversial claim among people who have shipped these systems, but it is still routinely underweighted in planning. Teams budget months for prompt engineering and model selection, and treat ingestion as a task for whoever is free. Then the pilot goes to a business user, who asks a question whose answer lives in a footnote to an amendment to a schedule, and the system confidently answers from the superseded main clause.

The document is not a blob of text

The default RAG pipeline, extract text, split every 512 tokens, embed, top-k, works remarkably well on blog posts and remarkably badly on the documents enterprises actually run on.

Real enterprise documents have structure that carries meaning:

• Tables where a cell is only interpretable with its row and column headers, which fixed-size chunking cheerfully separates from it.
• Amendments and versions, where the current obligation is the base clause as modified by three later documents, none of which are retrievable by similarity to the user's question.
• Cross-references, "subject to Section 12.4", where the retrieved chunk is actively misleading without the chunk it points to.
• Definitions that change the meaning of an ordinary word throughout a document, and live in an appendix nobody's query will ever resemble.
• Scanned pages in the middle of otherwise clean PDFs, where OCR quality silently determines whether a section exists at all.

None of this is exotic. It describes an ordinary commercial contract, a clinical guideline, or a regulatory filing. And all of it is invisible to an evaluation set built from questions whose answers happen to sit in a single well-formed paragraph.

The practical response is unglamorous: parse structure before splitting text. Chunk on semantic boundaries, clause, section, table, rather than token count. Carry an explicit metadata envelope with every chunk: its document, version, effective date, section path, and the definitions in scope. Resolve cross-references at ingestion time so a retrieved clause arrives with the thing it depends on. Treat OCR confidence as a first-class signal, not a footnote in a log.

None of this is research. It is just work, and it is the work that determines whether the system is usable.

Retrieval has to respect permissions

There is a failure mode here that is worse than a wrong answer.

If an organisation indexes everything into one vector store and puts a chat box in front of it, that chat box is a query interface to every document any of those users could not previously see. Access control that lived in a file system, a DMS or a case-management tool does not survive the trip into an embedding index unless someone deliberately carries it across.

The version of this that gets caught in testing is a user retrieving a document they should not see. The version that does not get caught is a summary of documents they should not see, synthesised, unattributed, and impossible to trace after the fact.

So permissions have to be enforced at retrieval time, per user, against the source system's own model of who can see what, not applied as a post-hoc filter on results, and not baked into a per-team index that goes stale the day someone changes roles. This is more work and it is slower. It is also the difference between a system a security team will approve and one they will not.

The citation is the product

Once retrieval is sound, the remaining question is whether the generated text actually follows from what was retrieved. As we have written about elsewhere, the empirical picture here is poor: a Stanford evaluation of four production generative search engines found only around half of generated sentences fully supported by their citations (Liu, Zhang & Liang, 2023).

For an enterprise system the citation is not a UI nicety, it is the whole mechanism of accountability. It is what lets a reviewer check one claim in ten seconds instead of re-reading a contract. If citations are decorative, the system has produced something worse than an unsourced answer, because it looks verified.

Three things move this measurably in our experience:

Attribute at the claim level, not the response level. A paragraph with one citation at the end is unverifiable. Each assertion needs its own anchor, and assertions that lack one should be visibly marked rather than blended into the prose.

Measure faithfulness explicitly. An entailment check between each generated statement and its cited span, run as part of the evaluation suite, with a threshold the system has to clear. This is a metric that can regress silently when you change a prompt, which is exactly why it belongs in CI.

Make the source reachable in one click. Not the document, the span, in context, highlighted. If verification costs a reviewer more than a few seconds they will stop verifying, and the audit trail becomes theatre.

Refusal is part of retrieval

The last piece is the behaviour when the corpus does not contain the answer.

Public benchmarks overwhelmingly measure whether a system produces the right answer when one exists. Enterprise buyers care at least as much about the opposite case, because a plausible fabrication in a regulated workflow is the thing that ends the programme. The relevant question is not "how often is it right" but "when it is not sure, does it say so".

That behaviour has to be built and tested deliberately: evaluation cases where the answer genuinely is not in the corpus, a measured abstention rate, and a retrieval confidence signal the generation step is actually allowed to act on. A system that never refuses is not confident, it is unmeasured.

We are building an open test suite for this, because we could not find a good public one. If you have hard cases from your own domain, a question, a corpus, and the reason the honest answer is "I don't know", we would like them.

---

Working on retrieval over documents that fight back? Write to [email protected].