Production RAG vs “We Wrapped GPT”: What Founders Should Demand

Every pitch deck has an AI slide now. Most of them are a thin wrapper around a chat model with a pretty UI. That can be useful for demos. It is not a production RAG system. If you are a founder buying AI work, you need a checklist that separates theatre from something you can put in front of real users — with retrieval that respects tenancy, answers that cite sources, and costs you can explain to a board.

"We wrapped GPT" — what you're actually buying

A wrapper usually means: prompt plus API key plus chat UI, maybe a PDF upload that stuffs the whole file into context. It fails when documents are large, answers must cite sources, costs spike, latency matters, or the model hallucinates with confidence. Fine for an internal spike. Dangerous as the product — especially when customers assume the assistant "knows" your knowledge base.

Wrappers also hide operational reality. There is no ingest pipeline to update when a policy PDF changes. There is no way to prove Customer A cannot retrieve Customer B's content. There is no eval suite when you swap models. There is no refusal path when retrieval is weak — only a fluent paragraph that sounds right. Founders discover that gap after launch, when support tickets become the evaluation set.

Production RAG — what should exist before you call it done

Production retrieval-augmented generation is a system: ingest pipelines, chunking strategy, embeddings, a vector (and often hybrid) index, retrieval evaluation, prompt and version control, grounded answers with citations, rate limits, cost observability, and failure handling when retrieval returns nothing useful. The model is one component. The product is the pipeline around it.

Retrieval first, generation second is not a slogan. If the wrong chunks come back, the best model will still invent a confident wrong answer. If the right chunks come back with clear citations, even a smaller model can be useful. Budget engineering time for ingest quality, metadata, hybrid search, and re-ranking before you budget another model bake-off.

A minimal architecture founders can interrogate

You do not need a research paper. You do need a clear story: documents enter through a controlled ingest path; they are chunked with intentional overlap and metadata (tenant, product, date, permission); they are indexed for dense and keyword retrieval; a query is rewritten or filtered; top results are optionally re-ranked; the prompt only sees retrieved context; the answer cites sources users can open; low-confidence retrieval refuses or asks a clarifying question instead of guessing.

ingest -> chunk+metadata -> embed/index
query -> retrieve (hybrid) -> re-rank -> ground+cite
if retrieval weak -> refuse / clarify
else -> generate with budget + logging

That sketch is enough to ask hard questions. Where do deletes propagate? How do you re-index after a policy change? Which logs let you replay a bad answer? Who owns the API keys and the vector store? If a vendor cannot walk this diagram without waving at "the LLM," you are buying a wrapper with better branding.

Founder evaluation checklist

Data and ingest: How do documents enter the system? Updates, deletes, access control per tenant? Chunking and metadata: Why this chunk size? Can you filter by product, date, or permission? Retrieval: Vector-only or hybrid (BM25 + dense)? Top-k tuning? Re-ranking? Grounding: Do answers cite sources users can open? What happens on low-confidence retrieval? Evaluation: Golden questions? Regression set when you change embeddings or prompts? Safety and tenancy: Can Customer A retrieve Customer B's content? PII handling? Cost and latency: Token budgets, caching, model routing for cheap versus hard queries? Ops: Observability, replay of bad answers, rollback of prompt or index versions? Ownership: Your keys, your indexes, your eval harness — or a black box you cannot leave?

Questions that expose demo theatre

"Show me an answer that refuses because retrieval was weak." "Show cost for 1,000 queries last week." "Change one document and prove the old answer is gone." "Walk me through how a new hire reproduces your eval suite." Teams that built production RAG answer these without improvising. Wrapper teams change the subject to model brands.

Bring your own messy corpus to a technical session — not their curated demo pack. Ask for citations on a question that has no good source in the corpus. Ask what happens under concurrent load when the embedding provider throttles. Ask how they version prompts the same way they version application code. The goal is not to trap people. It is to see whether production thinking exists before you fund the build.

Cost and failure modes founders forget to price

Token spend is the visible line item. The expensive failures are quieter: wrong-tenant retrieval, stale documents that still answer as if current, and support load when the assistant invents policy. Price observability and eval work into the SOW. Ask for budgets per query class, caching for repeated lookups, and a plan when the embedding or chat provider throttles. Production RAG is as much ops as models — treat it that way when you compare proposals that only list "LLM integration."

What "good enough for v1" still requires

You can ship a first production RAG without every research trick. You should not ship without tenant isolation, basic hybrid retrieval, citations, an eval set you re-run on changes, cost and latency logging, and a refusal path. Those are table stakes for anything customer-facing. Fancy agent loops can wait. Leaky tenancy and ungrounded answers cannot.

At CodeFlamme we treat AI systems like any other production product: clear ownership of data paths, senior engineers on the critical path, and acceptance criteria that a founder can verify. If you need a demo for an investor meeting, say so. If you need a system users will trust with documents, demand the pipeline — not the wrapper.