4096 Sessions in a Single Process: After pi-durable Batches Database Commits, CPU Gets Faster Too

The bill for durability always comes due later. Every step is persisted to the database, so crash recovery is clean, but in normal operation you have to wait on IO. This time, pi-durable paid the bills together.

Mario Zechner gave a number in a tweet: when a single Node process reaches 4096 sessions, the system is clearly saturated but still usable.

Let's fill in the background. The word 'durable' in pi-durable comes from persisting to the database: at every step the agent first writes state to the database, then continues. One save equals one commit. An agent run involves about 12 commits. If a crash happens midway, it recovers from the last commit. The cost is that every step has to wait for the database.

The old implementation was slow because of serial waiting. The session sent each commit to the database separately and then waited for the response. A DB round trip was 4 ms, and each session's commits queued behind those of other sessions. With 128 sessions, a run had to wait about 25 seconds before its turn. Mario's change: while one commit is in flight, the commits piling up behind it are bundled into the next DB call and sent together. With 128 sessions, one DB call carried about 33 commits. The wait dropped from about 25 seconds to 0.28 seconds.

Batching sounds simple; the trouble is dependencies. You can't just blindly pack all commits together, because individual commits may have read/write dependencies. Mario says this part is quite tricky. A commenter asked about a crash scenario: if the database has committed a batch but the caller hasn't received an ack yet, then the coordinator is killed and later recovers, can dependent reads and tool calls be replayed cleanly? Mario replied yes.

The effect isn't limited to networked databases. Local SQLite got 18% to 58% faster. Cloudflare Durable Objects read 40% fewer rows and write 13% fewer rows, with static memory unchanged.

The main chart title states the improvement as 46 ops/sec to 772 ops/sec. The small text at the bottom is more specific: local SQLite at 2048 conversations went from 88 ops/sec to 459 ops/sec; three parallel tools at 2048 conversations went from 58 ops/sec to 617 ops/sec; single-conversation performance remained unchanged.

The caveats are easy to overlook. Mario noted in a reply: the tests used an in-process fake model provider and no-op tools. This data measures the baseline compute that the harness can achieve. With real tools that consume CPU or I/O, runs/s will drop.

For developers running multiple agents, the conclusion is fairly direct. If a coordinator fans out to dozens of sub-agents through something like Postgres, you no longer need to shard sessions for throughput. The bottleneck lands on a single CPU first; the Node process will saturate first. Only when the CPU is actually maxed out should you consider sharding.

Commenter Mykyta Pavlenko summed it up quite accurately: after fixing storage, the CPU gain came for free, which shows the process was previously just waiting on commits. 4096 sessions at about 12 saves per run, all in a single Node box, is already in the range of something that can do real work. Mario thanked @she_llac for pushing him to do it.