Most RAG demos hide the hard parts: data freshness, permission boundaries, embedding drift, and cold-start latency on GPU nodes. Production systems need an explicit checklist before traffic arrives.
Separate ingest from query paths. Index rebuilds should be versioned and switchable. Document ACLs must survive chunking—retrieval that ignores identity is a data leak waiting to happen.
On Kubernetes, pin GPU node pools, set request/limit discipline, and watch queue depth on inference services. Pair OpenTelemetry traces across the API, retriever, and model endpoint so you can tell a slow vector store from a slow model.
Finally, measure answer quality continuously. Offline eval sets catch regressions; online feedback catches real user failures. Treat both as release criteria, not nice-to-haves.