Two 4090s Support 8 AI Agents: The 1.08 Million Tokens per Hour Math Needs to Be Broken Down
Two RTX 4090s serve 8 Qwen3.8-27B sessions locally, with aggregate throughput of 300+ tokens/s and a theoretical 1.08 million tokens per hour. Break it down to examine the roles of vLLM scheduling, MTP speculative decoding, and INT8 quantization, and also discuss the limits of this figure.
Two RTX 4090s running 8 Qwen3.8-27B sessions simultaneously, with aggregate generation throughput of 300+ tokens/s. The original tweet multiplies this rate by 3600 seconds to get 1.08 million tokens per hour.
The multiplication is correct. The premise is that these 300+ can hold steady for an hour.
First, lay out the configuration:
- GPU: 2 × NVIDIA RTX 4090
- Model: Qwen3.8-27B-INT8-W8A16-MTP
- Inference engine: vLLM
- Concurrent sessions: 8
- Context: 128K tokens
- Aggregate generation throughput: 300+ tok/s
- Theoretical hourly output: 1.08M+ tokens
The key sentence is buried later: 300+ is the aggregate throughput of all 8 sessions combined, not 300 per session. How much each agent gets depends on request size and server scheduling. If this distinction is not clear, all the discussion that follows will go off track.
What vLLM is doing here is partitioning GPU resources among multiple requests, rather than locking the entire card to one conversation. Continuous batching lets requests from different sessions enter the same batch; whoever has tokens to generate computes together. The 8 agents can hang off the same inference endpoint: one looks up information, one reads documents, one writes code, one reviews code, each doing its own thing.
MTP, or multi-token prediction, is used with speculative decoding. Ordinary autoregressive generation rolls forward one token after another, with each new token depending on the previous one. Speculative decoding first has a draft mechanism guess several tokens ahead, then the main model verifies them in batches, accepting all the correct guesses together. What is saved is part of the serial overhead. But the speedup depends on how many guesses the main model accepts; it will not be a fixed multiplier across different prompts and loads.
INT8-W8A16 means weights use 8-bit precision and activations use 16-bit. Lowering weight precision saves VRAM while retaining more numerical information than even lower bit widths. With the 24 GB of VRAM on two 4090s, this configuration is what makes it possible to fit the 27B model and still leave room for concurrency.
The 128K context setting also needs to be viewed calmly. Just because the config says 128K does not mean 8 sessions can all fill 128K at the same time without slowing down. KV cache consumes VRAM; concurrency, context length, and generation length compete with one another.
As for cost, the original tweet did not avoid it: owning GPUs does not mean eliminating costs. Electricity, hardware, maintenance, and system management are all still there. Two 4090s are not a small expense; the case, power supply, cooling, and electricity bill are all unavoidable.
So where is the value of this setup? It is not in the words 'saving money.' Local inference uses one machine as a shared endpoint, so private code and documents do not have to be sent to an external service provider with every request. Which model to choose, how to quantize it, how to schedule it, and how to handle data are all under the machine owner's control.
Treating 1.08 million tokens as a production-capacity metric that can be reliably reproduced will most likely lead to disappointment. Treat it as a signal that consumer-grade GPUs can already support a local endpoint for multi-agent parallel work; this direction is worth watching.
发布时间: 2026-11-02 11:27