The Benchmark Ends Before the Work Does
Inference throughput, response latency, and accepted business throughput measure different systems. A worked example shows why buying more generation can leave the real bottleneck untouched.
A serving system can produce more tokens per second while the business completes fewer useful tasks. There is no contradiction. The benchmark measures text leaving a model. The business needs work leaving a process, after failed attempts, checks, queues, and whatever a human has to repair.
That gap matters when choosing between purchased tokens and owned inference. A capacity figure is meaningful only inside a workload and service requirement. Otherwise the buyer is comparing engines without agreeing whether the vehicle is supposed to win a race or deliver a refrigerator.
Three clocks, three decisions
NVIDIA NIM’s Metrics documentation, version 2.0.0, defines the boundary of a model response precisely:
End-to-end request latency, or e2e request latency, measures how long it takes from submitting a query to receiving the full response.
That request measurement includes queuing, batching, and network delay. Time to first token ends earlier, at the first received token; system tokens per second aggregates output across requests. The same page notes that higher concurrency can increase system throughput while reducing per-user throughput. These definitions make it possible for a machine to improve its headline number while a person waits longer. They do not measure the later human acceptance stage.
For a person waiting in an interactive application, a fast beginning and a tolerable finish matter. For an overnight batch, aggregate completion before the deadline may matter more. Raising concurrency can improve total throughput while changing the experience of an individual request. Neither metric can be optimized on behalf of an unspecified customer.
There is another clock after the model finishes: the time until somebody can use its result. It includes retrieval and tool work, validation, retries, and any approval queue. For many business questions, that is the clock worth paying to shorten.
A deliberately small factory
Take an illustrative hour in which a system emits ten task outputs per minute, averaging 600 output tokens each. It produces 6,000 output tokens per minute, or 100 per second across those completed outputs. Assume every task requires human acceptance, and the reviewer can process only eight per minute.
Even if every output is correct, the system creates two more waiting tasks each minute. After an hour, it has added 120 tasks to the queue. Faster generation would enlarge the pile more quickly. The model has met its target; the operating process has acquired a backlog.
Now suppose only 80% of the reviewed outputs pass first time. The reviewer delivers 6.4 first-pass acceptances per minute, on average, before accounting for repair work. The remaining cases consume further capacity or remain unresolved. The 100-token-per-second headline tells the buyer none of this.
These are stipulated rates in a toy example, not measured system behavior. They expose two separate constraints: the amount a reviewer can inspect and the share that survives inspection. Improving either may be more valuable than increasing generation capacity.
For this toy hour, count only first reviews; repair work has not yet been completed. The unfinished work remains visible in the figure.
Retain the rejected work in the bill
For a declared measurement period, divide the entire attributable operating cost by the number of outcomes accepted under the declared standard. Include unsuccessful attempts, verification calls, human repairs, and an explicit allocation for the capacity committed to serving the workload.
The allocation matters when comparing owned and purchased supply. A fully occupied machine can look attractive while a lightly used production fleet does not. Report both the cost attributed to the task and the capacity assumption behind it. Moving idle hardware into another budget does not improve its economics.
Keep the unresolved count visible too. A team can make cost per accepted outcome look excellent by serving only easy cases and leaving the difficult work outside the denominator. That may be a sensible routing policy, but the scope and remaining work have to be explicit.
My note on tokenizers and perplexity supplies a concrete denominator problem. In TF3’s preprocessing comparison, Romanian BPE averaged 304.89 tokens per sentence, while Romanian Unigram averaged 340.35, about 11.6% more. That does not establish which tokenizer makes a business workflow cheaper; it shows why token counts from different tokenizers are not automatically the same measuring stick. Inspect the denominator before interpreting the number. A token is an implementation unit, not a standard unit of customer value.
Buy capacity against the limiting stage
John D. C. Little’s 1961 paper, A Proof for the Queuing Formula: L = λW, relates the mean number of units in a system to the arrival rate multiplied by mean time inside it, under the paper’s stationarity and finite-mean assumptions. The growing backlog here is deliberately not stable, so I am not applying that steady-state relation to it. The elementary calculation is sufficient: ten arrivals minus eight reviews leaves two waiting outputs per minute. Naming the queueing result is useful only if its conditions travel with it.
For an investment decision, define the task mix, demand pattern, acceptance threshold, and latency requirement first. Then locate the stage that prevents more acceptable work from leaving the process. It may be inference. It may be a tool dependency, a data problem, or the person everybody has described as “in the loop” without checking their calendar.
Hardware belongs in the proposal when it changes that constraint or provides a valuable reserve. A faster model at the front of an unchanged approval queue is a very efficient way to manufacture waiting.