Wink Pings

65B Dense Model Beats 975B MoE, Cost $32 Million

Ricursive Intelligence has released Ripple-65B Preview, a 65B dense model pretrained from scratch. It achieves SOTA on agentic coding tasks among US open-weight models, outperforming opponents with more than ten times the parameter count, at a total R&D cost of $32 million.

Ricursive Intelligence has released Ripple-65B Preview. A 65B-parameter dense Transformer, pretrained from scratch on 12 trillion tokens, supporting up to 1M context. Among US open-weight models, it scores 80.52 on TerminalBench 2.1, SOTA.

Opponent list: Reflection Beam 501B (80.1), Meta Muse Spark Open 816B (76.4), Thinking Machines Inkling 975B (63.8), NVIDIA Nemotron 3 Ultra 550B (56.4). A 65B dense model beat a string of MoEs more than ten times its parameter count.

In replies, a netizen did the actual math: a 65B dense model can fit on two 80GB GPUs and does not need a multi-node cluster.

image

Ricursive is a full-stack AI lab that builds self-improving systems by co-optimizing hardware, software, and models. The general variant of Ripple, an internal version, is running their AI chip design platform. This also explains why they trained a model from scratch: the goal first lands on chip design.

The cost figure is $32 million. Ricursive says this is the complete R&D expense, including from-scratch pretraining, data mixture, evaluation, debugging, synthetic data generation, post-training recipe development, and the final main training and post-training. The company claims this is one-tenth the cost of the previous agentic coding open-source SOTA. Sam Altman said in 2023 that GPT-4 training cost more than $100 million, and Dario Amodei later predicted that pretraining costs for frontier models would reach $10 billion to $100 billion. $32 million is another order of magnitude.

There is a counterintuitive point in the technical choices. Other labs generally use MoE to scale parameter count, while Ricursive chose a modern dense Transformer. The official line is that they prioritized decisions that reduce the complexity of pretraining and post-training infrastructure. The result: under simple sharding, no custom kernels, and no quantization (BF16), they maintained 40% MFU throughout pretraining and post-training. The development team expanded from 1 person to 5.

The architecture is thin-and-deep, with d_model only 5120 and 80 layers. They switched from Olmo 3's Llama-like architecture to a Gemma 3N-like architecture to improve stability at large-scale training. They increased the number of attention heads, while increasing the ratio of SWA to global layers and shrinking the SWA window to offset the cost. They additionally added FFN activation clipping, gated attention, global NoPE, and value residuals.

The model ladder is another source of efficiency. Starting from Olmo 3's model ladder, they iterated version by version from v1.0 to v1.6: StepFun's batch size and learning rate scaling laws, pre+post norm, sigmoid gated attention, global NoPE, value residuals. v1.4's 2-stream mHC was effective at small scale but did not improve at 4B scale, so it was cut. In the end, code compute efficiency improved 16x relative to the Olmo 3 ladder, and up to 55x for the hero run.

image

Post-training used a hybrid RL recipe, truncated importance sampling, score centering, linear length penalty, paired with high-quality long-horizon synthetic environments. TerminalBench 2.1 reached 80.5 with only 1.4 million rollouts, about one-seventieth of previous SOTA Reflection Beam.

image

Local running is another selling point. The 65B parameters can fit into an NVIDIA DGX Spark, and the official post also mentions MacBook Pro. They provide an NVFP4 checkpoint and Colab to help people get started. On data center GPUs, before quantization and without custom kernels, 2 B200s at batch=8 can reach 250 tok/s. Pretraining used an 8K sequence length, and the long context stage was extended to 2M.

Release channels:

Apache 2.0 license. The BF16 instruction-tuned model, NVFP4 version, and BF16 base model are all on HuggingFace. Fireworks will launch later that day.

In the benchmark table, Ripple scores 79.1 on SWE-bench Pro v2.0 Hard, while Beam 501B scores 76.5. On DeepSWE 1.1, Muse Spark Open 816B leads with 60.0, while Ripple scores 44.3. The official post also acknowledges that benchmarks are hard to reproduce: when using the official Terminus 2 harness and 128K sequence length plus summarization for Terminal-Bench 2.1, all models except Muse Spark Open scored lower. They wrote this into the model card.

In community reactions, Nathan Lambert left a comment "BIG OLMO," referring to the model ladder being based on AI2's Olmo work. Ricursive also thanked AI2, Prime Intellect, RadixArk, and NVIDIA for their open-source contributions in the blog.

A 65B model beating large-scale MoEs on a specific task does not mean parameter scale no longer matters. But there is a detail in the model card worth noting: Nemotron 3 Ultra was strong on traditional pretraining evaluations but did not translate that to agentic tasks; the official post speculates that synthetic data was relatively out-of-distribution for these tasks. The mismatch between evaluation sets and real capabilities is more worth watching than leaderboard rankings.

发布时间: 2026-11-05 08:29