[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-direct-opd-weak-to-strong-distillation-en":3,"article-related-direct-opd-weak-to-strong-distillation-en":30,"series-research-241fc8a1-6245-42f1-9be1-83f031963ec8":78},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":29},"241fc8a1-6245-42f1-9be1-83f031963ec8","direct-opd-weak-to-strong-distillation-en","Direct-OPD reuses weak-model RL gains for stronger models","\u003Cp data-speakable=\"summary\">48.3% to 62.4% on AIME 2024 shows Direct-OPD can transfer RL gains from a weak model to a stronger one.\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>Research org\u003C\u002Fstrong>: Unspecified in arXiv abstract\u003C\u002Fli>\u003Cli>\u003Cstrong>Core data\u003C\u002Fstrong>: 48.3% to 62.4% on AIME 2024\u003C\u002Fli>\u003Cli>\u003Cstrong>Breakthrough\u003C\u002Fstrong>: Distills the post-RL policy shift via log-ratio on on-policy states\u003C\u002Fli>\u003C\u002Ful>\u003Cp>\u003Ca href=\"\u002Ftag\u002Freinforcement-learning\">Reinforcement learning\u003C\u002Fa> with verifiable rewards has become a useful way to improve language-model reasoning, but it gets expensive fast when you want to apply it to every new strong model. The problem is simple: the target model has to generate many rollouts during training, and that makes post-training a bottleneck as models scale.\u003C\u002Fp>\u003Cp>This paper asks a practical question engineers will care about: if a smaller model can learn from RL cheaply, can a larger model reuse that learning without paying the full RL cost again? The answer here is yes, at least to a meaningful extent. The authors propose a weak-to-strong path that treats the weaker model’s RL run as a source of reusable supervision rather than just a final teacher checkpoint.\u003C\u002Fp>\u003Ch2>What problem this paper is trying to fix\u003C\u002Fh2>\u003Cp>Standard RLVR works well for reasoning, but it is not cheap to repeat. Every time you want to improve a new strong model, you need fresh rollouts on that model during training. That is exactly the part that becomes painful when the model is large, because generating those rollouts is expensive.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783405975437-yy2o.png\" alt=\"Direct-OPD reuses weak-model RL gains for stronger models\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The obvious shortcut is to train a smaller model with RL first and then distill it into a bigger one. But the paper says that directly distilling the post-RL weak teacher is not enough. Why? Because the teacher’s final policy is a mix of two things: the useful changes RL produced, and the limits of the smaller model itself. If you imitate that final policy directly, you may also inherit those size-related constraints.\u003C\u002Fp>\u003Cp>That distinction matters. In practice, developers do not just want a smaller model that got better; they want the improvement signal itself to move cleanly across model scales. This paper is about extracting that signal.\u003C\u002Fp>\u003Ch2>How Direct-OPD works in plain English\u003C\u002Fh2>\u003Cp>The method is called Direct On-Policy Distillation, or Direct-OPD. Instead of copying the weak model after RL, it compares the weak model before RL and after RL. The gap between those two checkpoints is treated as the thing worth transferring.\u003C\u002Fp>\u003Cp>More specifically, Direct-OPD uses the post-RL teacher and its own pre-RL reference, then takes their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells you which actions RL made the weak model more or less likely to take. Then that signal is applied to the stronger student on the states the student itself visits during training.\u003C\u002Fp>\u003Cp>That on-policy detail is important. The student is not just copying a static answer distribution from the teacher. It is learning from the teacher’s RL-induced shift while acting in its own training trajectory. The paper also emphasizes that this approach does not require training an explicit reward model or running sparse-reward RL on the target model.\u003C\u002Fp>\u003Cp>So the method is not “imitate the weak model” in the usual sense. It is more like “extract the delta RL created, then use that delta as training signal for the strong model.” That is a cleaner story for transfer than checkpoint distillation alone.\u003C\u002Fp>\u003Ch2>What the paper actually shows\u003C\u002Fh2>\u003Cp>The strongest concrete result in the abstract is a \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> improvement on AIME 2024. Direct-OPD boosts Qwen3-1.7B from 48.3% to 62.4% in just 4 hours on 8 A100 GPUs. That is the clearest number in the paper notes, and it gives a sense of both the gain and the training cost.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783405976395-z8v5.png\" alt=\"Direct-OPD reuses weak-model RL gains for stronger models\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The authors also say Direct-OPD consistently leverages weaker teachers to improve stronger target models. They report that it outperforms step-matched direct RL, and that it enables the sequential composition of multiple policy shifts. In other words, you can stack more than one learned improvement signal rather than treating each RL run as a one-off.\u003C\u002Fp>\u003Cp>What the abstract does not give is a full benchmark table, a broad model list, or exact comparisons across many tasks. So while the reported result is promising, the public notes here do not let us quantify how universal the gains are across reasoning datasets or model families.\u003C\u002Fp>\u003Ch2>Why developers should care\u003C\u002Fh2>\u003Cp>If you are building reasoning systems, the real cost is not just training one model once. It is repeating post-training every time you move to a new base model, a new size tier, or a new deployment target. This paper points to a way of reusing RL work instead of paying for it again from scratch.\u003C\u002Fp>\u003Cp>That could matter in workflows where you already have a smaller model that is cheap to train, but you want a stronger model to inherit the same reasoning improvements. Direct-OPD suggests that the useful part of RL is not only the final teacher policy; it is the directional change between pre-RL and post-RL behavior.\u003C\u002Fp>\u003Cp>For practitioners, that means a possible new post-training pattern: run RL where rollouts are cheap, then transfer the learned policy shift to a larger model using on-policy distillation. If it holds up beyond the reported setting, that could reduce the need for expensive sparse-reward RL on every target model.\u003C\u002Fp>\u003Ch2>Limitations and open questions\u003C\u002Fh2>\u003Cp>The abstract is encouraging, but it still leaves important questions open. We do not get a full picture of how well the method scales across different tasks, how sensitive it is to the choice of teacher, or how it behaves when the weak model’s RL run is noisy or unstable.\u003C\u002Fp>\u003Cp>There is also a conceptual limit baked into the setup: the method depends on having a useful weak-model RL run in the first place. If the smaller model cannot learn the right policy shift, there may be nothing good to transfer. And because the abstract does not include broader benchmark numbers, we cannot tell from these notes how consistent the gains are outside the highlighted AIME 2024 result.\u003C\u002Fp>\u003Cp>Still, the core idea is strong and practical: treat RL as something that produces reusable supervision, not just a better checkpoint. That framing could influence how teams think about post-training pipelines for reasoning models, especially when compute budgets are tight and model sizes keep growing.\u003C\u002Fp>\u003Cul>\u003Cli>RLVR is expensive to repeat on every strong model because it needs many rollouts.\u003C\u002Fli>\u003Cli>Direct-OPD transfers the RL-induced policy shift rather than imitating the final weak teacher.\u003C\u002Fli>\u003Cli>The paper reports Qwen3-1.7B improving from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs.\u003C\u002Fli>\u003C\u002Ful>","Direct-OPD lifts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 by distilling RL gains from a weaker model.","arxiv.org","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.05394",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783405975437-yy2o.png","research","en","af1699b9-52dd-414b-a951-32e02a760b75",[17,18,19,20,21],"reinforcement learning","distillation","reasoning models","on-policy training","weak-to-strong",[23,24,25],"Direct-OPD transfers the change RL makes to a weak model, not just the final weak model.","The method uses a pre-RL\u002Fpost-RL log-ratio as dense implicit reward for the strong student.","The abstract reports a 48.3% to 62.4% gain on AIME 2024 for Qwen3-1.7B.",0,"2026-07-07T06:32:32.065298+00:00","2026-07-07T06:32:32.049+00:00","3103988e-c4fe-45e3-98ab-846500c9d507",{"tags":31,"relatedLang":37,"relatedPosts":41},[32,34,35],{"name":19,"slug":33},"reasoning-models",{"name":18,"slug":18},{"name":17,"slug":36},"reinforcement-learning",{"id":15,"slug":38,"title":39,"language":40},"direct-opd-weak-to-strong-distillation-zh","Direct-OPD 讓弱模型 RL 成果可重用","zh",[42,48,54,60,66,72],{"id":43,"slug":44,"title":45,"cover_image":46,"image_url":46,"created_at":47,"category":13},"9d0b4bb8-5567-4f21-9249-5ec5f81a0245","label-free-real-bogus-classification-uncertainty-en","Label-free real-bogus classification with calibrated uncertainty","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783407776599-fpjx.png","2026-07-07T07:02:32.349971+00:00",{"id":49,"slug":50,"title":51,"cover_image":52,"image_url":52,"created_at":53,"category":13},"9d81e592-6c7e-48e4-8e0d-10e4c74d595f","camvla-calibration-free-view-robust-vla-en","CamVLA makes robot policies view-robust","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783404180114-4njo.png","2026-07-07T06:02:31.927927+00:00",{"id":55,"slug":56,"title":57,"cover_image":58,"image_url":58,"created_at":59,"category":13},"7fdb9c93-55a6-4f1a-968a-6e683b200191","data-link-layer-osi-layer-2-en","Data link layer: OSI layer 2 explained","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783348379590-4qi2.png","2026-07-06T14:32:29.07434+00:00",{"id":61,"slug":62,"title":63,"cover_image":64,"image_url":64,"created_at":65,"category":13},"7f13f21d-e4cd-4ed5-bf73-10c328681594","osi-model-explained-networking-layers-en","The OSI model still explains networking well","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783346592828-6a7d.png","2026-07-06T14:02:38.619749+00:00",{"id":67,"slug":68,"title":69,"cover_image":70,"image_url":70,"created_at":71,"category":13},"c2d5749a-f8c3-460f-bdcc-019aa1bf2552","evaluation-protocols-fine-tuned-llms-2026-en","Evaluation Protocols for Fine-Tuned LLMs in 2026","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783101774147-96vm.png","2026-07-03T18:02:25.049896+00:00",{"id":73,"slug":74,"title":75,"cover_image":76,"image_url":76,"created_at":77,"category":13},"ca28a691-10df-40cc-86fa-4684b467c452","deepspec-data-regeneration-pipeline-qwen3-eagle3-en","DeepSpec should be treated as a data-regeneration pipeline, not a tra…","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783080168119-497r.png","2026-07-03T12:02:18.983093+00:00",[79,84,89,94,99,104,109,114,119,124],{"id":80,"slug":81,"title":82,"created_at":83},"a2715e72-1fe8-41b3-abb1-d0cf1f710189","ai-predictions-2026-big-changes-en","AI Predictions for 2026: Brace for Big Changes","2026-03-26T01:25:07.788356+00:00",{"id":85,"slug":86,"title":87,"created_at":88},"8404bd7b-4c2f-4109-9ec4-baf29d88af2b","ml-papers-of-the-week-github-research-desk-en","ML Papers of the Week Turns GitHub Into a Research Desk","2026-03-27T01:11:39.480259+00:00",{"id":90,"slug":91,"title":92,"created_at":93},"87897a94-8065-4464-a016-1f23e89e17cc","ai-ml-conferences-to-watch-in-2026-en","AI\u002FML Conferences to Watch in 2026","2026-03-27T01:51:54.184108+00:00",{"id":95,"slug":96,"title":97,"created_at":98},"6f1987cf-25f3-47a4-b3e6-db0997695be8","openclaw-agents-manipulated-self-sabotage-en","OpenClaw Agents Can Be Manipulated Into Failure","2026-03-28T03:03:18.899465+00:00",{"id":100,"slug":101,"title":102,"created_at":103},"a53571ad-735a-4178-9f93-cb09b699d99c","vega-driving-language-instructions-en","Vega: Driving with Natural Language Instructions","2026-03-28T14:54:04.698882+00:00",{"id":105,"slug":106,"title":107,"created_at":108},"a34581d6-f36e-46da-88bb-582fb3e7425c","personalizing-autonomous-driving-styles-en","Drive My Way: Personalizing Autonomous Driving Styles","2026-03-28T14:54:26.148181+00:00",{"id":110,"slug":111,"title":112,"created_at":113},"2bc1ad7f-26ce-4f02-9885-803b35fd229d","training-knowledge-bases-writeback-rag-en","Training Knowledge Bases with WriteBack-RAG","2026-03-28T14:54:45.643433+00:00",{"id":115,"slug":116,"title":117,"created_at":118},"71adc507-3c54-4605-bbe2-c966acd6187e","packforcing-long-video-generation-en","PackForcing: Efficient Long-Video Generation Method","2026-03-28T14:55:02.646943+00:00",{"id":120,"slug":121,"title":122,"created_at":123},"675942ef-b9ec-4c5f-a997-381250b6eacb","pixelsmile-facial-expression-editing-en","PixelSmile Framework Enhances Facial Expression Editing","2026-03-28T14:55:20.633463+00:00",{"id":125,"slug":126,"title":127,"created_at":128},"6954fa2b-8b66-4839-884b-e46f89fa1bc3","adaptive-block-scaled-data-types-en","IF4: Smarter 4-Bit Quantization That Adapts to Your Data","2026-03-31T06:00:36.65963+00:00"]