DeepSeek Lets the Model Grade Its Own Homework: Is the Reward Signal for Wish-Driven Programming Stable?

In the DeepSeek V4.1 Flash 1017 update, there is a line: reinforcement learning based on self-feedback. The official description says that this training improved the model's ability to build complete projects from scratch, including scenarios such as game development and 3D modeling—wish-driven programming tasks of this kind.

What is self-feedback reinforcement learning? In RL training, the model does not fully rely on external ground-truth answers or human-annotated rewards. Instead, it evaluates, critiques, and verifies its own outputs and uses these self-evaluations as reward signals, or as part of the reward signal, to optimize behavior. Some commenters said this is like the model talking to itself while writing code. Others said it is simply the model being its own teacher and its own referee, with less annotation.

The pain point this approach addresses is clear. Traditional RL that relies only on external outcome rewards has too sparse a signal in wish-driven programming. The user gives only a high-level description, such as building a 3D display of the Forbidden City or making a web-based racing game. The model must plan on its own, write multi-file code, tune visuals and interaction, and handle various implicit requirements. Such tasks have no clear ground-truth answer, involve many intermediate steps, and belong to long-horizon agent scenarios where early errors accumulate and final quality is also hard to automatically score with simple rules. It is difficult for the model to learn from sparse outcomes how to complete a project from beginning to end.

Self-feedback provides denser intermediate rewards and reduces reliance on external annotation and human preferences. The self-feedback mechanism in the official description lets the model self-check and self-correct during generation, assess the quality of intermediate results and final effects, and deliver a relatively complete project through short-turn interactions with fewer prompts and reference materials.

In 2025–2026, there are quite a few papers of this kind in the community. The materials list 10 papers, and the directions become clear:

1. Post-Training Large Language Models via Reinforcement Learning from Self-Feedback, which uses the model's own confidence in its answers as an intrinsic reward and generates preference data for preference optimization. https://arxiv.org/abs/2507.21931
2. RLSR: Reinforcement Learning from Self Reward, where the model acts as its own referee and does not need ground-truth answers to provide reward signals. https://arxiv.org/abs/2505.08827
3. Reflect, Retry, Reward, where the model generates self-reflection after a wrong answer and uses whether the second attempt succeeds to reward the reflection process; it uses GRPO. https://arxiv.org/abs/2505.24726
4. Self-Rewarding Language Models, where the model both generates answers and uses LLM-as-a-Judge to score itself, training iteratively. https://arxiv.org/abs/2401.10020
5. Self-Evolved Reward Learning for LLMs, where the reward model iteratively improves itself using feedback it generates itself. https://arxiv.org/abs/2411.00418
6. ReflexiCoder, which uses RL to internalize the full trajectory of generation, reflection, and self-correction into the model weights, so it can self-correct at inference time without an external execution environment. https://arxiv.org/abs/2603.05863
7. ICRL, which jointly trains a solver and a critic so that the model internalizes the improvements brought by critique as capability, with clear gains on agent tasks. https://arxiv.org/abs/2605.15224
8. Self-Credit, which addresses the credit assignment problem under sparse rewards in long-horizon agent tasks by using citations generated by the model itself for dense credit assignment. https://arxiv.org/abs/2610.28388
9. Agent-Reflex, which systematically trains an agent's proactive reflection ability by combining environment feedback and RL. https://arxiv.org/abs/2610.17953
10. ReVeal, RL with a multi-turn generation–self-verification loop that lets generation and verification capabilities evolve together. https://arxiv.org/abs/2506.11442

The comments also added LongZero, which uses data synthesized by the model itself to improve long-context reasoning ability. https://arxiv.org/abs/2610.30132v1

After reading all this, let's return to this DeepSeek update. The value of self-feedback RL is straightforward: in tasks with no ground-truth answers, long horizons, and an emphasis on delivering complete projects, external rewards are too expensive and too sparse. Having the model produce intermediate evaluations itself can make the reward signal denser and reduce reliance on human annotation and external preferences. Someone in the comments asked whether the reward signal is stable. This question cannot be avoided. When the model evaluates itself, it may be self-consistent but incorrect, and scoring bias can be amplified by training. The official description does not say external verification is entirely unnecessary. If wish-driven programming is to truly deliver, execution environments, tests, and human spot checks still need to provide a backstop. The model acting as its own referee can save annotation, but the referee and the player cannot always be in cahoots.