Optimizing RAG Pipeline Latency: Where the Time Actually Goes

A RAG (retrieval-augmented generation) pipeline’s end-to-end latency is the sum of several distinct stages, each with different optimization levers — treating the whole pipeline as one opaque “LLM is slow” problem misses where the time is actually accumulating, and where optimization effort is actually worth spending.

The stages, roughly in order

1. Query embedding — converting the user’s query into a vector, usually fast (tens of milliseconds) unless using a very large embedding model or batching inefficiently.
2. Vector search / retrieval — the ANN query against your vector database, covered in this site’s vector database benchmarking article; latency here depends heavily on index type, dataset size, and recall/latency tuning.
3. Re-ranking (if used) — many RAG pipelines retrieve a larger candidate set cheaply, then re-rank with a more expensive (often cross-encoder) model to improve precision — this step can be a significant, easily overlooked latency contributor if the re-ranker itself is large or the candidate set is large.
4. Context assembly and prompt construction — usually negligible latency, but worth confirming if your pipeline does anything expensive here (large document chunking/formatting on every request rather than precomputed).
5. LLM generation (prefill + decode) — typically the largest single contributor, covered in depth in this site’s LLM inference metrics article (TTFT for prefill, then decode time scaling with output length).

Where most optimization effort actually pays off

For most RAG pipelines, LLM generation dominates total latency, particularly the decode phase for longer outputs — meaning the same levers covered in this site’s LLM cost/latency articles (capping output length, using a faster/smaller model where acceptable, leveraging continuous batching on the serving side) tend to be the highest-leverage optimizations, more so than micro-optimizing the retrieval stage, unless retrieval is unusually slow for your specific setup.

When retrieval becomes the actual bottleneck

Retrieval latency becomes the dominant factor when: the vector database is under-provisioned or poorly indexed for your data scale, a re-ranking stage uses an expensive model over a large candidate set, or multiple retrieval calls happen sequentially rather than in parallel (e.g. querying several different document collections one after another instead of concurrently). Profile your specific pipeline rather than assuming generation always dominates — measure each stage’s actual contribution before deciding where to optimize.

Parallelizing independent stages

Query embedding and any independent retrieval calls (multiple collections, hybrid keyword+vector search) can often run concurrently rather than sequentially — a pipeline that serializes inherently independent steps is leaving an easy latency win on the table before touching anything related to the LLM itself.

Streaming output to improve perceived latency

Even when total generation time can’t be reduced further, streaming tokens to the user as they’re generated (rather than waiting for the complete response) dramatically improves perceived latency — the user sees the response start appearing at TTFT rather than waiting for the full end-to-end time, even though the underlying total compute time is unchanged. This is a perception optimization, not a throughput optimization, but it matters a great deal for user-facing applications.

Caching at multiple levels

Beyond LLM-level prompt caching (covered in this site’s dedicated article), RAG pipelines can cache retrieval results for repeated or similar queries, and cache embeddings for frequently re-embedded content — caching opportunities exist at nearly every stage, not just the generation step.

Takeaway: profile each RAG pipeline stage independently before optimizing — generation usually dominates and responds well to the levers covered elsewhere on this site, but retrieval and re-ranking can become the real bottleneck under specific conditions, and only measurement reveals which applies to your actual pipeline.