VALKYRIE is the heavy tier of the lab, a pair of Tesla V100s serving a twenty-seven-billion-parameter model (Qwen3.6-27B, AWQ-quantized) through vLLM. For a while it looked like it was working as hard as it could. Both cards read 99 to 100 percent utilization under load. The obvious conclusion was that the hardware was the ceiling and the only way up was more silicon.

The obvious conclusion was wrong. The server was serving about 45 tokens per second, and the thing holding it there was a single number in a config file.

The signal that lied

nvidia-smi is the first place anyone looks, and on this host it is the first place anyone should stop looking. At concurrency 1, both V100s reported near-total utilization. That reads as saturated. But the V100 is a Volta card (SM70), and the AWQ dequantization path on that architecture is compute-bound even when the card is serving a single sequence. It is genuinely busy. It is just not busy doing very much useful work. Utilization percent measures whether the SMs are occupied, not whether they are producing tokens, and on this path the two came apart completely.

The honest measure is tokens per second, and tokens per second told a different story.

The tell

Throughput sat at 45.5 tokens per second at every concurrency level I tried. Not “roughly flat”, exactly flat. And wall-clock time scaled 1x, 2x, 4x, 8x with the request count. p95 time-to-first-token climbed to 34 seconds at eight concurrent requests.

That is not the signature of a saturated GPU. It is the signature of a server processing requests one at a time and making everyone else wait in line. The cards sat at 100 percent because each single request happened to peg them, not because the aggregate workload was maxing them out.

The cause

The bottleneck was --max-num-seqs 1. That flag caps how many sequences the scheduler will run in a single step, and at 1 the server batches nothing. Every request is serialized behind the last.

The value looked deliberate, the way a hand-tuned floor might. It was not. It was a default that had never been questioned, and the KV cache had room for far more. The vLLM startup log said so itself, in a line that had been printed on every boot the whole time:

Maximum concurrency for 32,768 tokens per request: 12.90x

The ceiling was never memory. The server was advertising headroom for roughly thirteen concurrent sequences while running one.

The fix

One line:

- --max-num-seqs 1
+ --max-num-seqs 12

Twelve is the KV-derived ceiling from that startup log, not a round number I liked. The result was 156.1 tokens per second at peak, a 3.43x jump, with 177 sustained on longer outputs, held flat over nearly five minutes with zero out-of-memory events, preemptions, or errors. Peak VRAM landed at about 28 GB on each card, balanced across the tensor-parallel pair, at roughly 207 watts and low-70s Celsius. The context window stayed at 32K and the memory-utilization target stayed where it was. There was no tradeoff to make. Full context and full concurrency coexist.

The part that is not a straight line

Here is where “just raise the number” stops being the whole answer.

Throughput on this card does not climb monotonically with concurrency. The V100 vLLM fork pins its CUDA-graph capture to batch sizes of 1 and 2. At batch 1 or 2 you get full-graph decode, around 20 to 24 milliseconds per token. At batch 3 and up, decode falls out of the captured graphs and onto a piecewise path that costs roughly 2.7 times as much per token. So throughput actually dips through the middle. Concurrency 3 measured 47.8 tokens per second, barely above the serialized single stream. It took about six concurrent requests to clearly beat two.

Which means concurrency 2 is a real place to stand if you want low latency: 1.66x the baseline throughput, first token in under a second, and you never leave the fast graphs. The right setting is not “as high as it goes”. It is a choice between a latency floor and a throughput ceiling, with a valley in between that will punish you if you land in it by accident.

What I took from it

Two things transfer past this one server. The first is that utilization percent is a dead signal on this path, and probably on more paths than people assume. If a number can read 100 percent while the useful output is a fraction of capacity, it is not the number to optimize against. Tokens per second was the only honest measure here.

The second is smaller and more embarrassing: read the startup log. The answer was printed on every boot. The server told me it had room for thirteen sequences while I was feeding it one, and I did not notice until I stopped trusting the utilization graph and started counting tokens.