[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-river-llm-reinforcement-learning-without-answers-zh":3,"article-related-river-llm-reinforcement-learning-without-answers-zh":30,"series-research-93b19c63-dbfd-4277-92b5-b5a60946fd65":77},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":29},"93b19c63-dbfd-4277-92b5-b5a60946fd65","river-llm-reinforcement-learning-without-answers-zh","RiVER 讓 LLM 不靠標準答案也能學","\u003Cp data-speakable=\"summary\">RiVER 證明 \u003Ca href=\"\u002Ftag\u002Fllm\">LLM\u003C\u002Fa> 可以只靠執行回饋與分數校準，在沒有標準答案的任務上學出更好的策略。\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>研究機構\u003C\u002Fstrong>：arXiv 摘要未明確標註\u003C\u002Fli>\u003Cli>\u003Cstrong>核心數據\u003C\u002Fstrong>：ALE rating rank 提升 8.9%\u003C\u002Fli>\u003Cli>\u003Cstrong>突破點\u003C\u002Fstrong>：分數校準式強化學習\u003C\u002Fli>\u003C\u002Ful>\u003Cp>這篇論文想處理一個很實際的問題：很多強化學習流程，尤其是訓練 \u003Ca href=\"\u002Fnews\u002Fclaude-code-terminal-main-workflow-guide-zh\">code\u003C\u002Fa> model 的流程，都默認你手上有正確答案。但現實裡不一定有。你可能只有執行結果、分數，或某種可\u003Ca href=\"\u002Fnews\u002Fmoneygram-solana-validator-infrastructure-not-marketing-zh\">驗證\u003C\u002Fa>但不是「標準答案比對」的回饋。RiVER 就是在這種情境下，嘗試把 LLM 訓練起來。\u003C\u002Fp>\u003Cp>它的主張很直接：沒有 ground-truth answer，不代表不能做 \u003Ca href=\"\u002Ftag\u002Freinforcement-learning\">reinforcement learning\u003C\u002Fa>。重點是，你不能把原始分數直接丟進訓練迴圈就算了。分數要先校準，讓\u003Ca href=\"\u002Fnews\u002Fdanceopd-on-policy-generative-field-distillation-zh\">模型\u003C\u002Fa>學到的是穩定的偏好，而不是被不同題目的分數尺度牽著走。\u003C\u002Fp>\u003Ch2>這篇在解什麼痛點\u003C\u002Fh2>\u003Cp>傳統的 RLVR，也就是 reinforcement learning with verifiable rewards，很吸引人，因為它有清楚的獎懲訊號：對就加分，錯就扣分。問題在於，這套做法通常還是依賴 ground-truth answers。只要題目沒有已知正解，標準做法就卡住了。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782454671897-i8l3.png\" alt=\"RiVER 讓 LLM 不靠標準答案也能學\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>RiVER 把焦點放在 score-based optimization tasks。這類任務不是問「答案對不對」，而是讓系統執行候選方案，再根據表現拿到一個分數。這種監督訊號在工程上很常見，像是 heuristic 類問題或其他以績效計分的任務，都很像這種情境。\u003C\u002Fp>\u003Cp>但論文指出，直接拿原始分數做 group-relative RL 會出問題。它點出兩個失真來源：scale dominance 和 frequency dominance。前者是不同測資或題目的分數尺度差太多，更新方向被扭曲；後者是弱解樣本太常出現，反而壓過少數但更好的候選方案。\u003C\u002Fp>\u003Ch2>RiVER 到底怎麼做\u003C\u002Fh2>\u003Cp>RiVER 是 Ranking-induced VERifiable framework。名字已經把核心說白了：它不是只看分數高低，而是把執行回饋轉成排名導向的學習訊號。\u003C\u002Fp>\u003Cp>方法的關鍵是 calibrated reward shaping with instance-wise comparisons。白話講，就是每個題目先用自己的脈絡來看分數，不把所有題目的原始數值硬拿來互比。這樣可以避免某些題目的分數天生比較大，卻在訓練時把別的題目蓋掉。\u003C\u002Fp>\u003Cp>RiVER 也不是把所有解法都扁平成二元的 pass\u002Ffail。它會強調排名前面的 solver，但同時保留其他有效解的 bounded feedback。這一點很重要，因為如果只獎勵第一名，訓練訊號會太窄；但如果所有有效解都差不多，弱解又可能因為樣本多而主導更新。RiVER 要處理的，就是這個平衡。\u003C\u002Fp>\u003Cp>所以它的核心不是單純「有分數就能訓練」，而是「分數要先經過校準與比較，才適合拿來做 RL」。這也是它和一般 raw score 訓練最大的差別。\u003C\u002Fp>\u003Ch2>論文實際證明了什麼\u003C\u002Fh2>\u003Cp>這篇摘要沒有公開完整 \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> 細節，所以我們只能根據摘要裡列出的結果來看它證明了什麼。作者用 12 個 AtCoder Heuristic Contest 任務訓練，並在 Algorithm Engineering Benchmark、LiveCodeBench 和 USACO 上評估。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782454673412-fjd7.png\" alt=\"RiVER 讓 LLM 不靠標準答案也能學\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>最直接的數字是 ALE-Bench。RiVER 讓 Qwen3-8B 和 GLM-Z1-9B-0414 的 ALE rating rank 分別提升 8.9% 和 9.4%。這代表它不只是把訓練環境內的分數拉高，至少在摘要提供的結果裡，確實有可見的排名增益。\u003C\u002Fp>\u003Cp>更值得注意的是轉移效果。論文說，雖然訓練只用沒有 ground-truth 的任務，但 backbone 也在 exact-solution benchmarks 上變好。摘要裡報告的絕對平均提升是 LiveCodeBench 2.4%、USACO 3.5%。\u003C\u002Fp>\u003Cp>這裡的訊息很關鍵。很多方法在訓練集上看起來很強，但一換到真正需要正確解的 benchmark 就掉下去。RiVER 的摘要說，baseline 用 raw execution scores 雖然能改善 ALE rating，卻無法轉移到 exact-solution benchmarks。這暗示校準步驟不是裝飾，而是決定模型有沒有學到可泛化能力的核心。\u003C\u002Fp>\u003Ch2>對開發者有什麼意義\u003C\u002Fh2>\u003Cp>如果你在做 code model 或其他需要執行回饋的系統，RiVER 提供了一個很實際的設計方向：訓練訊號不一定要來自完美標註。只要能穩定執行、穩定評分，就有機會把這個訊號轉成 RL 可用的獎勵。\u003C\u002Fp>\u003Cp>這對很多場景都很有吸引力。像是 ground truth 成本高、標註不完整，或根本沒有單一正解的任務，都可能適用。論文的訊息是：這些任務不必只能拿來做評估，也可以拿來做訓練，只是 reward 的形狀要設計好。\u003C\u002Fp>\u003Cp>對工程實作來說，另一個提醒是 raw score 不是天然的好 reward。即使分數是 deterministic 的，只要不同題目的尺度不一致，或候選解的出現頻率失衡，policy update 就可能偏掉。結果可能是訓練指標變好，但模型本身沒有真的更會解題。\u003C\u002Fp>\u003Ch2>這篇的限制也很明顯\u003C\u002Fh2>\u003Cp>摘要講得最完整的是方法和幾個 headline result，但很多實作細節沒有展開。它沒有提供完整 benchmark 表、訓練 compute、sample efficiency，或更細的 ablation 結果。這些都會影響你判斷方法值不值得搬進自己的 pipeline。\u003C\u002Fp>\u003Cp>另外，摘要也沒有說 RiVER 對 reward calibration 的設定有多敏感。也就是說，這套方法是不是要很精準地調參，才能穩定壓住 scale dominance 和 frequency dominance，摘要裡還看不出來。\u003C\u002Fp>\u003Cp>實務上還有一個問題是成本。instance-wise comparisons 和 ranking-based shaping 聽起來合理，但也可能比單純吃 raw score 更麻煩。摘要沒有交代額外開銷，所以在真正部署前，還得看完整論文才能評估。\u003C\u002Fp>\u003Cp>不過，這篇的主軸已經很清楚：RL for LLM 不一定要靠 ground-truth answers 才能成立。只要你能執行候選解、能穩定拿到分數，再把 reward 校準好，就有機會把這些訊號變成更廣泛的 coding 能力。\u003C\u002Fp>\u003Ch2>總結\u003C\u002Fh2>\u003Cp>RiVER 證明的不是「沒有標準答案也能隨便訓練」，而是「沒有標準答案時，reward 更需要被設計」。對做 code model 的開發者來說，這打開了一類更大的訓練場景，也提醒大家：真正的瓶頸常常不是有沒有分數，而是分數能不能教對行為。\u003C\u002Fp>\u003Cul>\u003Cli>沒有 ground-truth 的 score-based 任務，也能用來訓練 LLM。\u003C\u002Fli>\u003Cli>原始執行分數不能直接當 reward，否則會出現尺度與頻率偏差。\u003C\u002Fli>\u003Cli>RiVER 的摘要顯示，它不只改善訓練任務，也能轉移到 exact-solution benchmarks。\u003C\u002Fli>\u003C\u002Ful>","RiVER 證明 LLM 可以只靠執行回饋與分數校準，在沒有標準答案的任務上學出更好的策略。","arxiv.org","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.27369",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782454671897-i8l3.png","research","zh","c05899fc-dd62-4fad-a249-9748376c1ef2",[17,18,19,20,21],"LLM","reinforcement learning","reward shaping","execution feedback","code model",[23,24,25],"RiVER 證明沒有標準答案時，LLM 仍可透過執行回饋學習。","分數校準是關鍵，否則 raw score 會帶來尺度與頻率偏差。","摘要顯示方法有跨 benchmark 的轉移效果，但完整細節未公開。",0,"2026-06-26T06:17:26.979468+00:00","2026-06-26T06:17:26.969+00:00","0c35a120-52fc-41fc-afa3-d404eb934158",{"tags":31,"relatedLang":36,"relatedPosts":40},[32,34],{"name":17,"slug":33},"llm",{"name":18,"slug":35},"reinforcement-learning",{"id":15,"slug":37,"title":38,"language":39},"river-llm-reinforcement-learning-without-answers-en","RiVER trains LLMs without ground-truth answers","en",[41,47,53,59,65,71],{"id":42,"slug":43,"title":44,"cover_image":45,"image_url":45,"created_at":46,"category":13},"cd8b1802-2094-4f5c-89a9-230680124777","mistral-ocr-4-document-ai-structure-zh","Mistral OCR 4 把文件變結構化資料","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782468184906-6p2v.png","2026-06-26T10:02:37.422252+00:00",{"id":48,"slug":49,"title":50,"cover_image":51,"image_url":51,"created_at":52,"category":13},"a90ab5b6-f647-4cef-85af-35ff7bb21a93","autoregressive-boltzmann-generators-ditch-flows-zh","ArBG 改用自回歸做分子採樣","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782455577323-vrvt.png","2026-06-26T06:32:30.056363+00:00",{"id":54,"slug":55,"title":56,"cover_image":57,"image_url":57,"created_at":58,"category":13},"cd38b72e-b309-493d-b36f-684745ff5f7e","danceopd-on-policy-generative-field-distillation-zh","DanceOPD：把修圖技能蒸餾進同一模型","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782453784592-x1gk.png","2026-06-26T06:02:33.123618+00:00",{"id":60,"slug":61,"title":62,"cover_image":63,"image_url":63,"created_at":64,"category":13},"af1a155b-d8e6-4575-a014-959aef283098","microsoft-ai-team-collaboration-cfp-2026-zh","Microsoft 砸錢研究團隊協作 AI","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782415981776-jikr.png","2026-06-25T19:32:33.155576+00:00",{"id":66,"slug":67,"title":68,"cover_image":69,"image_url":69,"created_at":70,"category":13},"2cc1973d-a7a5-4031-8ed3-e05ca5d335fd","ai-papers-code-music-rare-disease-zh","3 篇 AI 論文：程式、音樂、罕病診斷","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782372792462-buxp.png","2026-06-25T07:32:27.274897+00:00",{"id":72,"slug":73,"title":74,"cover_image":75,"image_url":75,"created_at":76,"category":13},"f9ec6d6f-80a9-4a8e-b3ea-1eb5231aa796","new-nlp-papers-agent-memory-tool-use-zh","新 NLP 論文盯上代理記憶與工具使用","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782371888802-40t8.png","2026-06-25T07:17:39.070441+00:00",[78,83,88,93,98,103,108,113,118,123],{"id":79,"slug":80,"title":81,"created_at":82},"f18dbadb-8c59-4723-84a4-6ad22746c77a","deepmind-bets-on-continuous-learning-ai-2026-zh","DeepMind 押注 2026 連續學習 AI","2026-03-26T08:16:02.367355+00:00",{"id":84,"slug":85,"title":86,"created_at":87},"f4a106cb-02a6-4508-8f39-9720a0a93cee","ml-papers-of-the-week-github-research-desk-zh","每週 ML 論文清單，為何紅到 GitHub","2026-03-27T01:11:39.284175+00:00",{"id":89,"slug":90,"title":91,"created_at":92},"c4f807ca-4e5f-47f1-a48c-961cf3fc44dc","ai-ml-conferences-to-watch-in-2026-zh","2026 AI 研討會投稿時程整理","2026-03-27T01:51:53.874432+00:00",{"id":94,"slug":95,"title":96,"created_at":97},"cf046742-efb2-4753-aef9-caed5da5e32e","adaptive-block-scaled-data-types-zh","IF4：神經網路量化的聰明選擇","2026-03-31T06:00:36.990273+00:00",{"id":99,"slug":100,"title":101,"created_at":102},"53a0dc54-0371-4e40-8d5e-74e94a73840c","geometry-aware-similarity-metrics-for-neural-representations-zh","超越距離測量：用微分幾何重新理解神經網路","2026-03-31T06:01:01.241968+00:00",{"id":104,"slug":105,"title":106,"created_at":107},"fee7d472-a775-4b1d-bbc2-1e8bca1bbf8b","on-the-fly-repulsion-in-the-contextual-space-for-rich-divers-zh","讓AI繪圖更有創意：用排斥力提升生成多樣性","2026-03-31T06:01:25.439673+00:00",{"id":109,"slug":110,"title":111,"created_at":112},"a9901203-d69b-447b-8854-15d14eab32b4","vision-aided-beam-prediction-cnn-eca-zh","影像輔助波束預測升級 CNN","2026-04-01T10:00:25.8073+00:00",{"id":114,"slug":115,"title":116,"created_at":117},"b55e7dd4-0a24-4b3d-804d-b0309a03f498","triple-band-fss-mimo-antenna-sub-6-ghz-zh","三頻 FSS MIMO 天線瞄準 sub-6 GHz","2026-04-01T13:18:36.857305+00:00",{"id":119,"slug":120,"title":121,"created_at":122},"f68290bd-e7f3-4b30-ba22-dcd4e0130a66","openclaw-1299-repos-eight-weeks-analysis-zh","OpenClaw 1299 個 Repo 的資料解讀","2026-04-02T05:03:45.208411+00:00",{"id":124,"slug":125,"title":126,"created_at":127},"ed9f80eb-eb02-4d35-8ad4-0ddf428751dd","beam-coherence-aware-combining-mmwave-mimo-zh","毫米波 MIMO 的雙階合併法","2026-04-02T05:27:26.897188+00:00"]