Speeding up end-to-end inference with self-improving agents

Introduction

We build co-inventors: self-improving agents that design and optimize high-performance infrastructure. Applied to AI inference, they optimized the entire inference stack for two of the largest open-source LLMs: DeepSeek v4 Pro and GLM 5.2 running on NVIDIA B200s with vLLM v0.23. They improved both throughput and interactivity by up to 16% across multiple max-concurrency levels, while preserving model behavior through distribution-level correctness checks. Each concurrency level took the agents about a day to optimize.

As of vLLM v0.26, most of the changes made by our agents have no upstream equivalent. The changes span the entire inference stack, including kernels, schedulers, load-balancers, and configuration settings.

A single fast kernel doesn't make a fast system (Amdahl's law), and it doesn't necessarily make a correct one. That's why we optimize the whole stack at once rather than individual components.

Speed only counts if correct model behavior is preserved, and that can be subtle to check. Customizations can quietly alter model outputs, yet still give the same accuracy on standard benchmarks like GSM8K or GPQA. We use stringent distribution-matching checks rather than functional tests or benchmark scores.

Our agents learn over time. Every experiment produces reusable insights that the agents use to improve their invention process across models and workloads. For example, our agents learned to avoid distributed deadlocks that would have cost 44 minutes in unnecessary wait time, generalizing a lesson learned from optimizing DeepSeek v4 Pro applied to GLM 5.2.

What makes optimizing inference systems hard?

Agents must consider millions of lines of code containing tunable parameters and algorithm implementations, plus countless server configuration settings, making the search space enormous. Small changes in the right places can produce large gains, but the hard part is finding them and knowing what to prioritize.

Modern inference systems depend on many interacting factors:

• Software: OS, kernels (e.g., PTX, CuTeDSL, CUDA C, and Triton), high-level scripting (e.g., PyTorch), Python glue code, runtime, scheduling, caching, load balancing, interconnect, client-server configurations.
• Hardware: GPU, CPU, memory, disk, intra- and inter-node connectivity.
• Runtime dynamics: input distributions, caching behavior, memory pressure, throttling, network communication.

Because these systems are highly dynamic, optimizing them requires continuous evaluation and profiling; static code analysis is not enough. But end-to-end evaluations are slow and computationally expensive: a single one can take hours, spanning compilation, server launch, warm-up, and request processing.

As the agents continually iterate on evaluation results, our evaluation system uses a hierarchical decomposition of end-to-end and micro benchmarks to keep feedback loops short.

The invention loop

Our co-inventors run a rigorous discover + build → evaluate → verify → self-improve loop using many agents in parallel. They explore and implement optimization strategies, evaluate whether each speedup is real, verify that it preserves the model's outputs, and distill what they learn for the next run.

Discover and build

Our agents propose optimization strategies using dynamic profiling, static analysis, and learned experiences via a self-improvement mechanism. They prioritize optimization targets by weighing potential speedups against the impact on numerical accuracy. This narrows the search space and helps balance exploration versus exploitation.

Evaluate

Our agents get feedback from evaluations in a sandbox that replicates the real production serving environment. There, they test customized vLLM serving stacks, dispatching to their own optimized kernels and code in place of the stock implementations.

For models with hundreds of billions to trillions of parameters running on complex workloads, measurement is slow and inherently noisy. Shifts in batching behavior and hardware effects like GPU power throttling introduce variance in performance evaluations.

An agent optimizing against noisy evaluations will report speedups that are not real. We identify and mitigate sources of measurement noise to ensure that the agent gets clean feedback while optimizing. We then apply statistical analysis and anti-cheating checks so that a measured gain holds up in production.

Verify

A speedup is only useful if it sufficiently preserves the model's behavior. Small output errors can compound over long sequences, degrading quality.

We use a stringent correctness check based on model output distribution matching. These checks involve statistical consistency tests on token-level probability distributions between the optimized model and the stock model. We evaluate this across a diverse multilingual corpus of prompts from the AllenAI Common Crawl C4 dataset.

Verification cadence matters as well: too frequent wastes compute, and too rare lets agents drift off track. A validated evaluation method makes fast, trustworthy iteration possible. This is the same thorough philosophy behind our Safe Rust migration work.

Self-improve

Our agents distill insights from their experiments and reuse them across models, updating their own reasoning, tactics, and knowledge over time. These insights go beyond basic fact-retrieval: the agents use their experience to change their own workflow and discovery process.

Optimization examples

Looking ahead

We are building agents that work on practical system design and optimization. This requires them to work reliably over long time horizons, with the highest levels of rigor.

Our co-invention process compounds, driven by these reinforcing elements:

1. Accurate observations about the full system in the loop enable agents to reason about system-level improvements.
2. Correctness makes each performance gain shippable and a source of meaningful insights.
3. Autonomy lets the agents search a space too large for humans to explore alone.
4. Learning carries these insights forward, so each run gives the agents more experience to draw from.

Scalable verification remains an open problem, and advancing it is necessary to make autonomous optimization practical at frontier scale. We enforced a tight numerical tolerance in this early phase, which gives us a solid foundation to generalize to broader notions of correctness.

Our agents can optimize under more general approximation tolerances. We are actively applying our co-inventors across different use cases and deployment patterns.

If you want our agents to optimize your high-performance systems, contact us at hello@asari.ai.

If you're passionate about building agents that ship verified systems code, join us!