Two hours, three yuan to train a 64M model? MiniMind lays out both the comments and the code
MiniMind, recommended by Akshay Pachaar, uses native PyTorch to connect pretraining, SFT, LoRA, DPO, PPO/GRPO, distillation, and Agentic RL into a single pipeline. With 64M parameters, a single 3090 GPU, and the mini data combination, it takes about 2.31 hours and costs about 3 yuan; both code and data are open source. It suits people who want to see the training pipeline clearly, and it also exposes the capability limits of small models.
Akshay Pachaar recommended MiniMind on X: a 64M-parameter small language model that implements the full pipeline from pretraining and SFT to LoRA, DPO, RL, and distillation entirely in native PyTorch, and can run on a single consumer-grade GPU. The project is open source on GitHub, created by Jingyao Gong, under the Apache 2.0 license.
Projects like this are easy to misjudge based on the headline. The tweet says 2 hours, but the repository comments add a qualification: 2 hours refers to the measured time to complete 1 epoch of the SFT stage on a single 3090, and 3 yuan refers to the GPU rental cost for the corresponding period. In the training cost table, mini pretraining plus SFT totals about 2.31 hours and about 3 yuan. The numbers are real, but you need to see the scope they apply to.
What is really worth looking at is the code organization. The main MiniMind architecture aligns with the Qwen3/Qwen3-MoE ecosystem; the tokenizer, data format, chat_template, training scripts, inference service, and OpenAI API compatibility are all in one repository, and it also integrates inference frameworks such as llama.cpp, vllm, and ollama. The core algorithms do not rely on high-level wrappers such as trl or peft; LoRA, DPO, PPO, GRPO, CISPO, and distillation are all written from scratch. For people who want to open up the black box of large models, this is more useful than asking 'what can 64M chat about?'

The pipeline covered in the repository is quite complete: Pretrain, SFT, LoRA, RLHF-DPO, PPO/GRPO/CISPO in RLAIF, Tool Use, Agentic RL, adaptive thinking, and model distillation. The data is also open source: in the mini combination, pretraining data is 1.2GB and SFT data is 1.6GB; the full mainline has 10GB of pretraining data and 14GB of SFT data. To quickly reproduce the Zero chat model, downloading the mini data is enough.

The path to a quick working run is not complicated: clone the repository, install dependencies, put pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl into ./dataset, run pretraining and SFT in sequence, and then test with eval_llm.py.
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
# 下载 pretrain_t2t_mini.jsonl 与 sft_t2t_mini.jsonl 到 ./dataset
cd trainer && python train_pretrain.py
python train_full_sft.py
cd .. && python eval_llm.py --weight full_sft
The capability boundaries also need to be made clear. The repository keeps dialogue examples from the early Zero model: basic conversation works, but factual knowledge and generalization are clearly limited, and English answers fall apart. In objective evaluations, small models like this have long hovered near the random baseline on multiple-choice tasks such as C-Eval and CMMLU; the author used LoRA to do a round of multiple-choice format alignment, and minimind-3-exam improved by an average of about 2.9 percentage points, indicating that part of the bottleneck is in the input format, not in the knowledge itself.

The RL part also comes at a cost. In the repository's lightweight Agent math ToolUse test, the agent weights got 17 out of 20 questions correct, while full_sft got 12; but the author also warns that the agent weights are more prone to factual hallucinations on open-ended Q&A. Post-training raises one capability line, and it usually taxes generality and factuality. This phenomenon is not new; MiniMind lays it out for you to see.
Project link: https://github.com/jingyaogong/minimind
Online demo: https://www.modelscope.cn/studios/gongjy/MiniMind
Video introduction: https://www.bilibili.com/video/BV12dHPeqE72
Two hours and three yuan should only be treated as an entry point. The engineering facts are in the comments, data tables, and training scripts. After going through the repository and looking back at the tweet headline, you will know which parts are promotional language and which parts are reproducible code.
发布时间: 2026-11-18 02:29