Load Testing LLM APIs: A Practical Guide

Load testing an LLM API needs a different mental model than testing a typical REST endpoint — the previous articles on this site covering LLM inference metrics and continuous batching explain why; this one focuses on the practical mechanics of actually running such a test.

Building a realistic prompt distribution

Don’t test with one fixed prompt repeated — real traffic has varied prompt lengths and varied expected output lengths, and both affect prefill cost, decode cost, and how the server’s continuous batching behaves under mixed-length concurrent requests. Sample from a distribution of realistic prompt lengths and max_tokens settings matching your actual production traffic shape (or your best estimate of it pre-launch), the same parameterization principle covered for JMeter and other tools elsewhere on this site, applied specifically to prompt/output length rather than generic test data.

Measuring streaming responses correctly

If your API streams tokens (most production LLM APIs do, since it improves perceived latency as covered in this site’s RAG latency article), your load testing tool needs to specifically instrument time to first token and time to last token separately, not just total request duration — a tool or script that only measures “time until the HTTP response fully completes” collapses TTFT and decode time into one number, losing the diagnostic value of knowing which phase is actually contributing latency under load.

A practical k6 approach for streaming measurement

import http from 'k6/http';

export default function () {
  const start = Date.now();
  let firstTokenTime = null;

  const res = http.post(url, payload, {
    responseCallback: http.expectedStatuses(200),
  });
  // Custom timing logic for first-byte vs full-response time,
  // depending on your client library's streaming support
}

(The exact implementation depends on your HTTP client’s streaming support — the key requirement is capturing a timestamp at first-byte/first-token separately from total completion time, which may require a lower-level streaming client than a simple blocking HTTP call provides.)

Sweeping concurrency to find the operating curve

Rather than testing at one fixed concurrency level, run the same test at several increasing concurrency levels (similar to the “ramp and watch the knee” approach covered in this site’s LLM inference metrics article) and plot achieved throughput (tokens/sec) against p95 TTFT at each level — this reveals the actual throughput/latency curve and the concurrency level beyond which latency degrades sharply, which is the practical “capacity” answer most teams actually need.

Testing realistic concurrent connection patterns, not just request rate

Because LLM requests are long-lived (seconds, sometimes much longer, unlike typical sub-second REST calls), the relevant load dimension is often concurrent open requests more than requests per second — a load testing tool’s executor/injection profile (covered in this site’s k6 and Gatling articles) should be chosen with this in mind; an arrival-rate executor needs maxVUs/equivalent sized generously enough to hold all the long-lived concurrent requests a given arrival rate will actually produce.

Cost tracking during load tests

Since LLM API calls have a direct, often non-trivial per-token cost (covered in this site’s LLM cost calculator and related articles), a sustained load test against a real (non-mocked) LLM endpoint has a real dollar cost — estimate this before running a large-scale test, and consider testing against a cheaper or smaller model for initial load-shape exploration before running the full test against your actual production model.

What “good” looks like in results

A healthy result shows throughput scaling roughly linearly with concurrency up to some point, then a clear knee where p95 TTFT begins increasing sharply while throughput growth flattens — that knee is your practical capacity number, and it should inform both your capacity planning and your reasonable rate-limiting/backpressure configuration for production.

Takeaway: LLM API load testing needs prompt/output-length variation, separate TTFT/total-latency measurement, and a concurrency sweep to find the real operating curve — testing with one fixed prompt at one fixed concurrency level answers a much narrower question than most teams actually need answered.