Let AI Write an Inference Engine Overnight, Reaching 16k tok/s in One Night

An inference engine is the service layer that handles requests, manages caches, and runs models. All engines share most of their skeleton; what truly differs is the model-specific forward pass and kernels. Eric Zhang's Resin enters through this gap: the skeleton (scheduling, KV cache, prefix cache, metrics) is written in Rust, while the model-specific parts are left for an AI agent to write. The code is at github.com/ekzhang/resin.

Usage is straightforward. Import the Resin core, have the agent write a forward pass optimized for the target hardware (GPU/TPU/AMD) for the model, and add the kernels you want. Done. A standalone engine.

Eric has been working on this for the past month. DeepSeek-V4.1-Flash is already within 5–10% of DeepInfer. Yesterday he wrote a skill, had Codex work through the night, and by the next morning it delivered the GLM-5.3-Flash engine: 60k input tok/s and 16k output tok/s on B200. This engine includes MTP, CUDA graphs, and TP on Blackwell. The core Rust part is about 3,000 lines and covers scheduling, KV/SWA/recurrent state, prefix cache, multimodality, and dynamic batching.

This is not an isolated experiment. vLLM core maintainer Woosuk Kwon reposted that vLLM is also moving in the same direction: isolating model-specific code is already going smoothly, and it is considering replacing more of the stack with Rust. He shared a concept diagram of Model Runner V2's ModelState abstraction: a stateful API that encapsulates model-specific logic and provides hooks for extra model inputs, attention metadata, and CUDA graph capture. It is almost the same philosophy as Resin.

Why is this noteworthy? Previously, letting AI write an inference engine from scratch generally produced unreliable results. Eric's explanation: every time you reinvent the wheel, the complex logic of the scheduler and serving layer gets repeated over and over. A shared core narrows the problem down to a single verifiable goal—write a fast forward pass. Some peers pointed out that kernels are the hardest part to share. Resin's trade-off is: share the skeleton, and leave kernels for AI to generate.

A commenter added a key constraint: the agent must first match the reference implementation's logits (e.g., the Hugging Face version) on a set of prompts before it is eligible to chase speed. Otherwise, an engine produced overnight could be fast but wrong. Another view: treating primitives as implementation references rather than framework contracts, and letting the agent derive the architecture from first principles, is more useful than giving it a set of API constraints. Someone else said AI agents perform best on well-scoped tasks. One hard problem, one beatable score. This probably explains why most previous vibe-coded inference engines failed: the task was too open-ended.

Resin's README is honest: it is not production-grade software; for stability, use SGLang or vLLM. But the direction is already clear—the general parts of engines are becoming infrastructure, and AI is taking over the model-specific parts.