Introduction
MLPerf Inference's benchmark suite needs to keep pace with how AI is actually deployed. The original tests covered image classification, object detection, speech recognition, recommendation, and single-turn text generation — important workloads, but ones that miss one of the fastest-growing production uses of large language models: multi-turn agentic inference, where a coding assistant or workflow agent reads context, calls tools, and iterates across many dependent turns rather than answering a single query. MLCommons notes this shifts the serving problem in several ways: context and KV-cache pressure grow across a trajectory, KV-cache reuse becomes a key optimization, output lengths swing from short tool calls to long reasoning traces, and throughput has to be measured as closed-loop progress rather than independent request rate. The new Agentic Inference benchmark adds this workload class to the MLPerf Endpoints framework, keeping MLPerf's general measurement principles while defining model choice, dataset composition, multi-turn load generation, output validation, and rules for optimizations like prefix caching and speculative decoding.
Model selection
The benchmark needed long-context, reasoning-capable models that stress agentic serving behavior, so MLCommons chose two: Kimi K2.6, a large mixture-of-experts model (1T parameters, 32B active) representative of leading agentic systems, evaluated with an Eagle3-based speculative-decoding head; and Qwen3.6-35B-A3B, a smaller model (35B parameters, 3B active) built on a new Gated DeltaNet architecture with native multi-token-prediction decoding. Both share a 262,144-token context window and are run with the same methodology and dataset, but each produces its own separate result — the two models are never combined into a single score.
Dataset and task selection
The reference dataset combines two agentic domains chosen to stress different parts of the serving stack, totaling 613 multi-turn trajectories — 113 agentic-coding trajectories and 500 agentic-workflow trajectories — with roughly 30,000 client-issued turns and a similar number of assistant turns across the set. The coding traces come from the DeepSWE dataset built by DataCurve (datacurve.ai): a user reports a software issue and the agent investigates a repository with bash commands, reads code, runs tests, and iterates, producing deep trajectories (a median of dozens of turns) with steadily growing context. The workflow traces, contributed by Workato, are synthetic traces modeled on Workato's production experience running enterprise customer-support agents that retrieve orders, check policies, or escalate cases — shallower trajectories, but built on a large shared system prompt with many tool definitions. Coding traffic stresses KV-cache capacity and long-context scheduling; workflow traffic stresses shared-prefix reuse and routing locality; combining both prevents a serving system from optimizing for only one traffic shape.
Client Design
MLCommons added multi-turn support to MLPerf Endpoints so submitters can benchmark a standard OpenAI-compatible serving endpoint (vLLM, SGLang, TensorRT-LLM, or similar) end to end. The client manages closed-loop replay (one conversation issues a turn at a time and waits for the full response before the next), controls target concurrency without breaking turn order, inserts dataset-defined inter-turn delays without counting them as serving latency, sends a stable session ID per trajectory so routers can preserve KV-cache locality, uses deterministic 'cache salting' to allow reuse within a trajectory while blocking invalid reuse across trajectories, reconstructs future prompts from the pre-recorded dataset rather than live model output to keep runs reproducible, and clears generated tokens from the KV cache so results don't depend on which system originally generated the traces.
Performance metrics
Results are reported as a Pareto curve, with each point a fixed-concurrency benchmark run. The y-axis is output tokens per second per system (aggregate throughput); the x-axis is output tokens per second per user, i.e. how fast an individual agent progresses through its task. As concurrency rises, a system can complete more total work even as each individual agent slows down — the Pareto curve is meant to make that trade-off visible.
Accuracy metrics
Accuracy is checked at three levels so that faster results can't come from shorter or lower-quality answers: an output-sequence-length check that the mean response length matches the workload's expected distribution; an inline accuracy check comparing performance-run outputs against ground truth (including coding tool-call checks and workflow intent-code checks); and a standalone accuracy check using 200 tasks from SWE-bench Verified to confirm end-to-end task-solving capability. All three must pass a defined threshold for every point on the Pareto curve. MLCommons published tentative accuracy figures (subject to change): inline accuracy of 63.08% for Kimi K2.6 versus 55.86% for Qwen3.6-35B-A3B; standalone accuracy of 76.5% versus 67.0%; and OSL per-turn mean ranges of [404, 494] tokens versus [355, 434] tokens, respectively.
Conclusion
MLCommons frames agentic inference as one of AI's fastest-growing applications and a substantially harder serving problem than single-turn generation, with growing context, strict turn dependencies, tool-mediated delays, shared prefixes, long-tail trajectories, and accuracy checks that must run on the very configuration used for performance testing. By combining deep coding trajectories with shared-prefix workflow trajectories, the new Agentic Inference benchmark aims to measure whether serving systems can keep real multi-turn users moving forward while still using hardware efficiently.
Sources
- Agentic Inference for MLPerf Inference — MLCommons