[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-pretrain-q-functions-online-rl-finetuning-zh":3,"article-related-pretrain-q-functions-online-rl-finetuning-zh":29,"series-research-08ceac3e-dd49-42e7-b976-e962eae021ca":74},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":11},"08ceac3e-dd49-42e7-b976-e962eae021ca","pretrain-q-functions-online-rl-finetuning-zh","Q 函數不一定要先預訓練","\u003Cp>如果你已經有一個不錯的 pretrained policy，下一步直覺常是把 Q-function 也一起預訓練好。但這篇研究說，這個直覺不一定對，甚至可能白花算力。\u003C\u002Fp>\u003Cp data-speakable=\"summary\">這篇論文指出，線上 RL 微調時，Q 函數不一定要先用單一策略預訓練；用多個策略回合做初始化，反而更有效。\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>研究機構\u003C\u002Fstrong>：arXiv 摘要未明確標註\u003C\u002Fli>\u003Cli>\u003Cstrong>核心數據\u003C\u002Fstrong>：平均提升 1.26 倍\u003C\u002Fli>\u003Cli>\u003Cstrong>突破點\u003C\u002Fstrong>：多策略回合初始化\u003C\u002Fli>\u003C\u002Ful>\u003Cp>這件事很實際。因為在強化學習裡，Q-function 不只是分數表。它會影響選動作、探索方向，還會左右微調時穩不穩。如果初始化方向錯了，後面再怎麼訓練，都可能是在修一個本來就偏掉的起點。\u003C\u002Fp>\u003Ch2>這篇在解什麼痛點\u003C\u002Fh2>\u003Cp>論文鎖定的是一個很常見的流程：先有 pretrained policy，再做 online RL fine-tuning。問題來了，這時候到底要不要先把 Q-function 也用離線資料預訓練？\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785391369404-txyj.png\" alt=\"Q 函數不一定要先預訓練\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>很多人會覺得答案是要。畢竟 policy 都先學過了，value function 也一起預熱，理論上應該更穩。但作者的觀察是，這個假設沒有大家想得那麼牢。摘要裡明確提到，天真式的 Q-function pretraining，常常和 random initialization 差不多，沒有明顯賺到。\u003C\u002Fp>\u003Cp>也就是說，問題不在於「Q-function 有沒有學過」，而在於「它是跟誰學的」。如果離線階段學到的分佈，和線上微調真正會走到的分佈不一樣，那個預訓練就可能只是把模型綁在舊策略上。\u003C\u002Fp>\u003Ch2>為什麼單一策略預訓練會失準\u003C\u002Fh2>\u003Cp>這篇研究的核心論點很白話：Q-function 不是通用的評分器，它是跟 policy 綁在一起的。你如果用某個 pretrained policy 的行為軌跡去學 Q-function，那它自然會偏向那個 policy 的世界觀。\u003C\u002Fp>\u003Cp>但線上 fine-tuning 不是原地踏步。policy 會變，動作分佈會變，最後收斂到的行為也可能和離線資料完全不同。這時候，原本那個 Q-function 就不一定還是最好的起點。\u003C\u002Fp>\u003Cp>摘要裡還特別提到，即使做了 offline value maximization，這\u003Ca href=\"\u002Fnews\u002Fsia-openai-enterprise-ai-zh\">個落\u003C\u002Fa>差仍然存在。這點很重要，因為它表示問題不只是 pretraining 不夠久，而是目標本身就不對齊。離線優化的東西，和線上階段真正需要的東西，並不是同一個。\u003C\u002Fp>\u003Ch2>方法怎麼做：IPE 是什麼\u003C\u002Fh2>\u003Cp>作者提出的解法叫做 Initialization via Policy Ensemble，簡稱 IPE。做法不複雜，但方向很明確：不要只靠單一 pretrained policy 的回合資料，而是先訓練多個不同的 policy，把它們的 rollouts 收集起來，再把這些資料池化，用來 bootstrap Q-learning。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785391366657-amsg.png\" alt=\"Q 函數不一定要先預訓練\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>換句話說，Q-function 的起點不再是某一條策略路徑，而是多條不同策略路徑的集合。這個「多樣性」就是方法的關鍵。它讓 value learning 一開始看到的行為分佈更廣，較不容易被單一 policy 的偏差鎖死。\u003C\u002Fp>\u003Cp>從實作角度看，IPE 比起改模型架構，更像是改訓練策略。重點不是把 Q-network 搞得更大，而是讓它在進入 online fine-tuning 前，先接觸更分散的行為樣本。\u003C\u002Fp>\u003Ch2>論文實際證明了什麼\u003C\u002Fh2>\u003Cp>摘要說，作者系統性地研究了：當你在 pretrained base policy 上做 fine-tuning 時，Q-function pretraining 到底有沒有幫助。結論是，天真式的 pretraining 常常沒有顯著優勢，甚至只是接近 random init。\u003C\u002Fp>\u003Cp>接著他們拿 IPE 去測，範圍是多個具挑戰性的 continuous control benchmarks。摘要給出的結果是，IPE 相對於 naive Q-function pretraining，平均有 1.26 倍的提升。\u003C\u002Fp>\u003Cp>不過，摘要沒有\u003Ca href=\"\u002Fnews\u002Fhuang-shou-pian-x-wen-gong-kai-ting-kai-fang-quan-zhong-ai-zh\">公開\u003C\u002Fa>完整 \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> 數字，也沒有列出每個任務的分數、方差或 task list。這代表我們目前只能確定「平均表現\u003Ca href=\"\u002Fnews\u002Fblack-duck-coverity-ai-era-triage-zh\">更好\u003C\u002Fa>」，還不能直接推論它在所有 RL 場景都會贏。\u003C\u002Fp>\u003Cul>\u003Cli>天真式 Q pretraining 常常不如預期\u003C\u002Fli>\u003Cli>IPE 不是單一策略資料，而是多策略 rollouts\u003C\u002Fli>\u003Cli>摘要只公開平均 1.26 倍提升\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>對開發者有什麼影響\u003C\u002Fh2>\u003Cp>如果你在做 RL 系統，這篇的訊息很直接：初始化不是小事。很多人把它當成訓練前的例行步驟，但在 value-based RL 裡，初始化本身就是演算法的一部分。\u003C\u002Fp>\u003Cp>尤其是做 online adaptation 的時候，目標通常不是重訓一個全新的 \u003Ca href=\"\u002Ftag\u002Fagent\">agent\u003C\u002Fa>，而是把既有 policy 再往前推一點。這種情境下，Q-function 應該幫助下一階段的優化，而不是只是忠實反映過去那個 policy 的行為。\u003C\u002Fp>\u003Cp>所以這篇不是在說「永遠不要 pretrain Q-function」。它比較像是在提醒：你要先想清楚，離線資料到底代表哪個 policy 分佈。如果那個分佈太窄，或者太貼近舊策略，預訓練可能不但沒幫忙，還會把後續 fine-tuning 拉歪。\u003C\u002Fp>\u003Ch2>限制與還沒回答的問題\u003C\u002Fh2>\u003Cp>先講最重要的限制：這篇摘要聚焦在 continuous control benchmarks。也就是說，目前公開資訊最能支持的，是這類任務上的結果，不是所有 RL 問題都能直接套用。\u003C\u002Fp>\u003Cp>第二個限制是，摘要沒有提供完整 benchmark 細節。沒有絕對分數，就很難判斷不同任務之間的差距，也很難知道 1.26 倍的平均提升，是穩定出現，還是少數環境拉高平均。\u003C\u002Fp>\u003Cp>第三個問題是「多個 diverse policies」到底怎麼選。摘要有說是多策略，但沒有交代 diversity 怎麼定義、需要幾個 policy、對結果敏感不敏感。這些都會影響方法能不能真的落地。\u003C\u002Fp>\u003Cp>即便如此，這篇還是提供了一個很有用的方向：如果你的 online RL pipeline 預設 Q-function 應該跟 pretrained policy 越像越好，這個假設值得重新檢查。至少在這篇研究裡，更廣的行為來源，反而讓微調起步更順。\u003C\u002Fp>\u003Cp>對台灣做 RL、機器人控制、或任何需要線上適應的團隊來說，這是個很實用的提醒。不是每個模組都該沿著「先預訓練再微調」的老路走。Q-function 的起點選錯，後面可能會多繞很多路。\u003C\u002Fp>","這篇論文指出，線上 RL 微調時，Q 函數不一定要先用單一策略預訓練；用多個策略回合做初始化，反而更有效。","arxiv.org","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.27203",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785391369404-txyj.png","research","zh","f234ef6f-2934-4e01-bc50-3132313c0d7a",[17,18,19,20,21],"reinforcement learning","Q-function","policy ensemble","online fine-tuning","continuous control",[23,24,25],"單一策略的 Q 預訓練不一定有用","多策略 rollouts 可改善初始化","摘要只公開平均 1.26 倍提升，細節未完整揭露",0,"2026-07-30T06:02:23.775888+00:00","2026-07-30T06:02:23.765+00:00",{"tags":30,"relatedLang":33,"relatedPosts":37},[31],{"name":17,"slug":32},"reinforcement-learning",{"id":15,"slug":34,"title":35,"language":36},"pretrain-q-functions-online-rl-finetuning-en","Do You Need to Pretrain Q-Functions?","en",[38,44,50,56,62,68],{"id":39,"slug":40,"title":41,"cover_image":42,"image_url":42,"created_at":43,"category":13},"59958bf8-ca7f-4d66-a170-e54eaa4c1b77","fruitfly-inspired-regression-without-heavy-models-zh","果蠅啟發回歸：用模式匹配省算力","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785394980008-nvqp.png","2026-07-30T07:02:27.028832+00:00",{"id":45,"slug":46,"title":47,"cover_image":48,"image_url":48,"created_at":49,"category":13},"f3cf3f4d-31fc-4666-8132-57c69ed66f4d","mental-world-modeling-simulating-minds-zh","世界模型不只看場景，也要看心智","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785393187520-8rd3.png","2026-07-30T06:32:29.057837+00:00",{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"c8c4d82d-7dd9-46e7-98a0-f6f2c6b86c1e","openai-agent-hack-forces-tighter-eval-controls-zh","OpenAI 事件逼你收緊 eval","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785326587078-rzpe.png","2026-07-29T12:02:45.792562+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"7d00f7e4-000b-4921-94bf-cf06b1da1ceb","care-confidence-adaptive-routing-lora-zh","CARE 用信心分派 LoRA 專家","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785308583150-1gha.png","2026-07-29T07:02:29.165929+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"0ff5f275-dc86-4cc7-924d-48ee394c81a7","pir2-reactive-real-time-flow-policies-zh","πR² 讓流式策略即時反應","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785306783936-ijjo.png","2026-07-29T06:32:33.987717+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"b66c78dd-4d97-455a-a6a7-a1f74bcdf18b","relay-opd-fixes-prefix-failure-distillation-zh","Relay-OPD 修補失敗前綴","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1785304970727-9vzo.png","2026-07-29T06:02:30.51071+00:00",[75,80,85,90,95,100,105,110,115,120],{"id":76,"slug":77,"title":78,"created_at":79},"f18dbadb-8c59-4723-84a4-6ad22746c77a","deepmind-bets-on-continuous-learning-ai-2026-zh","DeepMind 押注 2026 連續學習 AI","2026-03-26T08:16:02.367355+00:00",{"id":81,"slug":82,"title":83,"created_at":84},"f4a106cb-02a6-4508-8f39-9720a0a93cee","ml-papers-of-the-week-github-research-desk-zh","每週 ML 論文清單，為何紅到 GitHub","2026-03-27T01:11:39.284175+00:00",{"id":86,"slug":87,"title":88,"created_at":89},"c4f807ca-4e5f-47f1-a48c-961cf3fc44dc","ai-ml-conferences-to-watch-in-2026-zh","2026 AI 研討會投稿時程整理","2026-03-27T01:51:53.874432+00:00",{"id":91,"slug":92,"title":93,"created_at":94},"cf046742-efb2-4753-aef9-caed5da5e32e","adaptive-block-scaled-data-types-zh","IF4：神經網路量化的聰明選擇","2026-03-31T06:00:36.990273+00:00",{"id":96,"slug":97,"title":98,"created_at":99},"53a0dc54-0371-4e40-8d5e-74e94a73840c","geometry-aware-similarity-metrics-for-neural-representations-zh","超越距離測量：用微分幾何重新理解神經網路","2026-03-31T06:01:01.241968+00:00",{"id":101,"slug":102,"title":103,"created_at":104},"fee7d472-a775-4b1d-bbc2-1e8bca1bbf8b","on-the-fly-repulsion-in-the-contextual-space-for-rich-divers-zh","讓AI繪圖更有創意：用排斥力提升生成多樣性","2026-03-31T06:01:25.439673+00:00",{"id":106,"slug":107,"title":108,"created_at":109},"a9901203-d69b-447b-8854-15d14eab32b4","vision-aided-beam-prediction-cnn-eca-zh","影像輔助波束預測升級 CNN","2026-04-01T10:00:25.8073+00:00",{"id":111,"slug":112,"title":113,"created_at":114},"b55e7dd4-0a24-4b3d-804d-b0309a03f498","triple-band-fss-mimo-antenna-sub-6-ghz-zh","三頻 FSS MIMO 天線瞄準 sub-6 GHz","2026-04-01T13:18:36.857305+00:00",{"id":116,"slug":117,"title":118,"created_at":119},"f68290bd-e7f3-4b30-ba22-dcd4e0130a66","openclaw-1299-repos-eight-weeks-analysis-zh","OpenClaw 1299 個 Repo 的資料解讀","2026-04-02T05:03:45.208411+00:00",{"id":121,"slug":122,"title":123,"created_at":124},"ed9f80eb-eb02-4d35-8ad4-0ddf428751dd","beam-coherence-aware-combining-mmwave-mimo-zh","毫米波 MIMO 的雙階合併法","2026-04-02T05:27:26.897188+00:00"]