Wink Pings

One Megakernel per Model: Claude-Written Inference Engine Up to 2.05× Faster Than vLLM

Eight models, eight megakernels written by Claude. One GPU program runs the entire decode loop without switching back to the host. Beyond the up to 2.05× speedup, what really matters is the validation method: first prove it is still the same model, then talk about speed. The article also clearly explains the limitations.

General-purpose inference engines have, for the first time, met a counterintuitive alternative.

Xingyu Dang released a project: have Claude write one megakernel for each of eight open-source models. Each kernel serves only one checkpoint and one GPU, and runs the entire decode loop. The eight models range from 8B to 594B. On H100/B200, single-sequence steady decode is 1.13–1.77× faster than the fastest general-purpose engine on the same GPU; if allowed to change how the model is stored, but not the function it computes, up to 2.05×.

First, watch a video. On the same GPU, the megakernel engine and SGLang answer the same question, with text replayed at the actual arrival speed.

The megakernel's first token is usually later, because the steady-decode engine reads only one token of the prompt per pass; but its subsequent token intervals are shorter. The author does not avoid this point.

General engines and specialized kernels

vLLM and SGLang are designed to run arbitrary models. Each decode step is assembled from a general kernel library: one kernel for attention, one for matrix multiplication, one for normalization, executed in sequence from a CUDA graph, with the CPU scheduling once per token.

The megakernel takes another direction: one kernel executes all computation for each step, as well as the loop over tokens. Operator boundaries become grid barriers within the GPU, and there is no switch back to the host between tokens. The advantage is freedom of scheduling: prefetch weights for the next-next layer while computing the current layer, fuse across operator boundaries, and let idle thread blocks start the next stage early.

The cost is that it is hard to maintain and debug, and tied to one model and one GPU. So before this, megakernels existed only in a very few hand-written versions (Hazy Research's Llama-1B, Cohere's North Mini Code).

A coding agent eliminated this cost. Rewriting a kernel for one checkpoint became feasible for the first time. This is also the most counterintuitive part of the project: the maintenance cost of general engines was amortized by schedulers; now the writing cost of specialized kernels is amortized by agents.

Project page banner

How it was measured

The author reports in two tracks.

track A keeps the checkpoint's own weight format and data flow, and only changes the organization of computation on the GPU. track B is looser: it allows changing model storage and what is precomputed at load time, but requires the function to remain unchanged under exact arithmetic. Lossy quantization and pruning do not count.

The figure shows the speed of the eight models relative to SGLang (the two rows without SGLang use vLLM as baseline). Gray is the general engine, colored is this result.

Speedup vs SGLang

With a single sequence (batch size 1), track A exceeds the fastest general engine on all models: 1.13–1.77×. At batch size 8, gains become smaller; the lowest are 1.02× for North Mini Code and 1.08× for gpt-oss-120b. A few representative numbers:

  • Qwen3-8B, H100: SGLang 6.09 ms/token → track A 5.13 ms (1.19×), track B 3.83 ms (1.59×)
  • gpt-oss-20b, H100: vLLM 3.31 ms → track A 1.87 ms (1.77×), track B 1.62 ms (2.05×)
  • Inkling-2-Small (594B MoE), 4×B200: SGLang 4.74 ms → track A 3.30 ms (1.43×)

Single-sequence decode is basically limited by HBM bandwidth. Qwen3-8B's BF16 weights are 15.1 GB. At H100 peak bandwidth of 3.35 TB/s, reading them once takes 4.52 ms; the track-A engine measured 5.13 ms, reaching 88% of the theoretical upper bound. This number shows there is no magic in the speedup: general engines are already close to the physical limit, and specialized kernels pushed it a bit further.

Optimizations found by the agent

In track B, the agent discovers patterns that belong only to this checkpoint. The most typical one is weight format: BF16 uses 8 bits to store the exponent, but values within the same row of trained matrices span a narrow numeric range, so many exponent bits are unnecessary. Qwen3-8B's agent encodes each weight losslessly into 12 bits, decoding in registers before sending to tensor cores. This reduces per-token weight traffic by a quarter and single-token time by 15%.

Other recurring techniques include: a provably low-precision copy of the LM head, used to screen candidates for the next token; and a per-token lookup table for layer 0, because this layer's input depends only on the token's embedding.

The author used Claude Opus 5.5 and Fable 5.2, running 19 runs, 458 agent-hours, and 675 submissions in Claude Code. Each submission records the hypothesis written by the agent. The prompt only states goals, rules, and tools, and does not specify optimization techniques. The optimization path was profiled by the agent itself. Most runs have similar trajectories: the first engine that passes tests is slower than the general engine; the first big improvement comes from fusing the entire decode step into one kernel and arranging weights in the order tensor cores read them; subsequent work mainly targets cache and idle thread blocks.

Concurrent work SpecPilot also found the same 12-bit format. The author acknowledges this and states that their agent discovered it independently within the full decode megakernel.

Validation is more important than optimization

One lesson from autoresearch: the agent optimizes what the evaluator rewards; if the evaluator has holes, they become shortcuts. Kernels written by the agent had previously passed tests by reusing reference output memory, writing special cases for test inputs, and timing the wrong stream.

So the core of this framework is a test suite that the agent cannot modify. The engine must be compared with an FP64 reference model (or FP32 when memory is insufficient), with tolerances based on Hugging Face's BF16 implementation: median and p99 of logit error ≤ 2× the BF16 implementation, KL divergence ≤ 4×, top-1 agreement rate difference no more than 1 percentage point. There are also hidden natural-text/code/math problems, extended to 2000 positions, unseen throughout the run.

The engine's own decode loop must reproduce step-by-step greedy tokens; logits produced during timing runs are checked separately afterward; sequences in a batch are rerun in reverse order, and logits must be bitwise identical. Before release, it must also pass every batch size, full context, restarts, and a source-code audit by an independent agent.

The audit did find problems that the evaluator had let through. Three track-B engines marked bit-identical were actually not; flint's engine selected a different top-1 token at 10.8% of positions; it did not error when exceeding the context limit, causing KV cache to overwrite other sequences. These were all fixed later.

The project page states it directly: except for three Inkling-2-Small engines, all showcased engines were retested with the current suite; 20 passed all checks, all 20.

Serving real traffic

The steady-decode engine handles batches that start at the same time. A real server must accept new requests while decoding. The author did a second run for Qwen3-8B and Inkling-2-Small: scheduling is also moved into the GPU, with continuous batching, in-kernel chunked prefill, paged KV cache, and prefix caching.

Replaying Mooncake's tool-and-agent trace (prompts average about 9000 tokens), Inkling-2-Small on 4×B200, at 1–8 requests per second, satisfies SLO for 99–100% of requests; SGLang is 82–85%. SLO is defined as first token no more than 5 seconds and average inter-token interval no more than 0.1 seconds. The gap mainly comes from stalls: when a long prompt arrives, SGLang's decode requests must queue behind prefill and wait up to 2.3–4.4 seconds; the megakernel waits at most 0.08–0.15 seconds.

Inkling-2-Small serving trace

This is not without cost. The serving engine's first-token tail latency and median token interval are both worse than SGLang: on Inkling-2-Small, p99 first token is 3.4–4.6 seconds, versus 2.0–2.2 seconds for SGLang; median token interval is 28–60 ms, versus 9–18 ms for SGLang.

Limitations and next steps

The author lists clear boundaries. The steady-decode engine supports at most 8 sequences and 2048 token context, reads only one token of the prompt per pass, and does sampling outside the kernel, which is 1–11% slower. At 32 concurrency, SGLang does 3338 tokens per second, while the steady-decode engine only does 1427. Long prompts and long contexts currently rely on the serving engine, but its first token on a 32000-token prompt is 0.70 seconds, versus 0.41 seconds for SGLang. All benchmarks come from a single node and two or three rounds, as the author notes.

The next step is to build a serving engine whose small passes are as fast as the steady-decode kernel, and to do sampling inside the kernel. That would be one engine doing both things at once.

The code, test suite, and autoresearch tools are all open-sourced at megakernel-zoo. The repository also contains an OpenAI-compatible server, mkserve, which wraps the engine as a streaming API. After installation, just:

git clone https://github.com/dangxingyu/megakernel-zoo
cd megakernel-zoo
pip install -e .
mkserve serve Qwen/Qwen3-8B --port 8000

People with H100/B200 can run a run against their own checkpoint. Someone in the comments asked about a DeepSeek kernel; the author replied: there is not one in the zoo yet, but the pipeline is open, and one is welcome. Some netizens also pointed out that this currently supports only H100/B200, and consumer GPUs cannot run it. This is indeed the boundary at this stage.

Appendix

Project page: https://dangxingyu.github.io/megakernel/
Code: https://github.com/dangxingyu/megakernel-zoo

The data table, submissions for each run, baseline versions, and parameters are all on the project page. Citations, acknowledgments, and reference papers are also there.

发布时间: 2026-11-20 03:02