Kimi K3.1 Pro Beats GPT-6 Astra on Benchmarks, but the First Wave of Users Isn't Convinced

Just as Kimi K3.1 Pro's benchmark scores were released, the discussion split into two camps. On Artificial Analysis it ranks very high—the material even mentions it placing ahead of GPT-6 Astra; but in the same set of discussions, it trails GPT-6 Astra by about 20 points on Terminal-Bench Science, and tests more oriented toward long-horizon exploration, such as ProgramBench, are not impressive either. Nathan Lambert and Florian Brand talked through this split on Episode 20 of the Interconnects podcast.

One background note: This time Kimi changed the naming from K3 to K3.1 Pro, and the release cadence is also more professional. Florian said that in the past, open-source models often released the model first and added vLLM support later; now inference frameworks and inference providers are ready on day one, and sometimes the model side even gets several weeks of exclusive access first. Kimi K3.1 Pro is said to have two weeks of exclusive access, with weights to be released before November 4. The release promotion is also controversial: someone posted a screenshot claiming they received instructions not to disclose paid promotion, but this has not been confirmed by multiple parties. Someone in the comments added: people throw the model into their own pile of messy repos, run it for an afternoon, and then make a judgment.

Deployment cost is another underlying thread. Someone in the comments calculated that K3.1 Pro has about 2.8T parameters; at 4-bit, the weights are about 1.4TB, which barely fits into a single 8xB200 node, leaving very tight space for KV cache. Florian also said on the podcast that 3T-class models require B300 and one model replica per node, making deployment more awkward than smaller models that can run on H100/H200. The benchmark leaderboard tells you what a model can do; the bill tells you whether it will actually get used.

The cybersecurity section is more like this week's main course. In its blog, Kimi said it does not want open weights to extend the absolute frontier of cyberattacks; K3.1 Pro's offensive cyber capabilities are deliberately kept below its pipeline capabilities and also lag behind frontier closed-source models. The approach includes lowering cyber capabilities in mid-training data. Defensive capabilities are still retained. Florian thinks this release posture is smart and also aligns with what cybersecurity practitioners actually need: first find and patch vulnerabilities in a large amount of code.

PHP2Shell is a concrete example. A years-old remote code execution vulnerability in PHP was discovered by OpenAI's Patch the Planet project and other security researchers, reported to the PHP Foundation, and after collaborative fixing, a patch was released all at once. PHP underpins a large chunk of the web, including WordPress, and many sites briefly went offline to patch. The anticipated large-scale exploitation did not happen. LiteLLM is another story: an API key was committed to GitHub for five months and later exploited by someone to spread malware.

Nathan's observation is that cybersecurity experts look at the ecosystem, while the AI world looks at open source versus closed source; the two sides are often not talking about the same thing. The comment 'AI experts see benchmarks, cybersecurity experts see bills' is very apt. Closed-source model guardrails can be updated quickly, and OpenAI and Anthropic are also expanding trusted access to defensive cybersecurity programs; usage on the open-source model side is hard to see and tends to become visible only when something goes wrong. APIs such as Abliteration.ai and Morph that host abliterated models strike Nathan as a very bad judgment call right now.

The second half of the podcast turned to hiring and daily life. Anthropic said it would no longer hire junior employees; Nathan criticized this as revealing a shortsighted security culture, while Florian said it is good news for other companies, as young PhDs are worried about having no chance to get hands on clusters. After moving to the US, Florian started using Muse to handle forms, appointments, and buying kitchen supplies—giving it a credit card to run the shopping cart and using Stripe one-time virtual cards to check out. While he complains that it is slow and the browser crashes, he admits that proactive reminders and cross-website operations are useful. Nathan, meanwhile, uses Pangram to check whether emails were written by AI and remains wary of an agent inbox.

This week's buzz feels like a stress test. The benchmark scores are impressive, while deployment costs, long-horizon exploration, the cybersecurity ecosystem, and access permissions together bring the discussion back down to earth. The full podcast is on Interconnects; YouTube link: https://youtu.be/0xiq7RHVGRE