[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-rootly-benchmark-llama-4-trails-coding-models-zh":3,"article-related-rootly-benchmark-llama-4-trails-coding-models-zh":31,"series-research-10c48be8-a5e6-4153-87d3-573dd4b2aec4":77},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":23,"views":27,"created_at":28,"published_at":29,"topic_cluster_id":30},"10c48be8-a5e6-4153-87d3-573dd4b2aec4","rootly-benchmark-llama-4-trails-coding-models-zh","Rootly 測試：Llama 4 落後編碼模型","\u003Cp data-speakable=\"summary\">\u003Ca href=\"https:\u002F\u002Frootly.com\" target=\"_blank\" rel=\"noopener\">Rootly\u003C\u002Fa> 的 AI Labs 用 100 筆 \u003Ca href=\"\u002Ftag\u002Fgithub\">GitHub\u003C\u002Fa> bug 來測 \u003Ca href=\"https:\u002F\u002Fai.meta.com\u002Fllama\u002F\" target=\"_blank\" rel=\"noopener\">Llama 4\u003C\u002Fa>，結果顯示它在編碼任務上落後於多個競品，甚至輸給 Llama 3.3。\u003C\u002Fp>\u003Cp>這份測試於 2025 年 4 月 11 日公布，題目是讓模型從 4 個候選 pull request 中挑出正確答案。Rootly 說，這比一般大而全的基準更貼近開發者日常的 triage \u003Ca href=\"\u002Fnews\u002F8tai-jiqiren-bao-shiyan-liucheng-zuo-cheng-bihuan-zh\">流程\u003C\u002Fa>。\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>項目\u003C\u002Fth>\u003Cth>數值\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>Benchmark size\u003C\u002Ftd>\u003Ctd>100 GitHub bug issues\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Llama 4 Maverick accuracy\u003C\u002Ftd>\u003Ctd>70%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Llama 4 overall accuracy\u003C\u002Ftd>\u003Ctd>69.5%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Llama 3.3 70B-Versatile\u003C\u002Ftd>\u003Ctd>72%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>DeepSeek v3.1 gap\u003C\u002Ftd>\u003Ctd>6% ahead of Llama 4\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>GPT-4o gap\u003C\u002Ftd>\u003Ctd>18% ahead of Llama 4\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2.5-Coder-32B accuracy\u003C\u002Ftd>\u003Ctd>About 90%\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>發生了什麼\u003C\u002Fh2>\u003Cp>Rootly AI Labs 把 Llama 4 Scout、Maverick、Behemoth，和一批通用多模態模型、編碼調校模型放在同一個測試裡比。測法很直接：給一則 bug report，再給 4 個候選 PR，模型要選出正確配對。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782086568903-6jm6.png\" alt=\"Rootly 測試：Llama 4 落後編碼模型\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>這次沒有把完整 codebase 一起丟進去。Rootly 的意思很明確，這不是學術型大考，而是更像真實工單場景的快速判斷題，考的是模型能不能看懂問題、抓到修補方向。\u003C\u002Fp>\u003Cul>\u003Cli>Llama 4 在 Rootly 排名墊底，總準確率 69.5%。\u003C\u002Fli>\u003Cli>Llama 3.3 70B-Versatile 拿到 72%，比 Llama 4 還高。\u003C\u002Fli>\u003Cli>DeepSeek v3.1 比 Llama 4 高 6 個百分點。\u003C\u002Fli>\u003Cli>GPT-4o 領先 Llama 4 18 個百分點，Qwen2.5-Coder-32B 和 OpenAI o3-mini 都接近 90%。\u003C\u002Fli>\u003C\u002Ful>\u003Cp>Rootly 也提到，他們無法重現 \u003Ca href=\"\u002Ftag\u002Fmeta\">Meta\u003C\u002Fa> 對 Llama 4 在 reasoning 和 coding 上優於 GPT-4o、\u003Ca href=\"\u002Ftag\u002Fgemini\">Gemini\u003C\u002Fa> 2.0 Flash、DeepSeek v3.1 的說法。這讓這份 \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> 不只是分數比較，也變成對模型宣傳敘事的一次檢驗。\u003C\u002Fp>\u003Ch2>為什麼重要\u003C\u002Fh2>\u003Cp>對開發者來說，這種結果最直接的意義是：通用模型的名氣，不等於在編碼任務上的穩定表現。尤其是 bug triage、PR matching、incident response 這類工作，模型只要看錯一個線索，後面就可能整串偏掉。\u003C\u002Fp>\u003Cp>對團隊採購或導入來說，選型邏輯也會跟著變。若目標是寫 \u003Ca href=\"\u002Fnews\u002Fanthropic-github-repositories-claude-code-push-zh\">code\u003C\u002Fa>、挑修補方案、協助排查，Rootly 的數據更支持先看專門做 coding 的模型，例如 Qwen2.5-Coder-32B、o3-mini 或其他 code-tuned 系統，而不是只看一個通用旗艦版本。\u003C\u002Fp>\u003Cp>但這份測試也有界線。樣本只有 100 題，而且資料集開源，代表它更像一個指標而非最終裁決。真正值得追問的是，Llama 4 的設計是否更偏向廣泛對話與多模態，而不是開發者最常碰到的修 bug 工作流。\u003C\u002Fp>\u003Ch2>補充背景\u003C\u002Fh2>\u003Cp>近一年來，模型競爭已經從「誰會聊天」轉向「誰能在特定工作裡少犯錯」。對\u003Ca href=\"\u002Fnews\u002Flyra-anthropic-aws-enterprise-ai-distribution-zh\">企業\u003C\u002Fa>來說，這種差異會直接反映在工時、審查成本和自動化品質上。對產品團隊來說，benchmark 也不再只是宣傳素材，而是選模型時的實用參考。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782086568532-6ol3.png\" alt=\"Rootly 測試：Llama 4 落後編碼模型\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>這也是為什麼像 Rootly 這類偏工作流的測試會受到關注。它們不測抽象推理總分，而是看模型能不能在有限資訊下做出正確判斷。這種題型更接近真實開發環境，也更容易暴露模型短板。\u003C\u002Fp>\u003Cp>如果一個模型在一般評測裡表現亮眼，卻在 bug 對應上輸給舊版或專門模型，團隊就不該只看發布會上的總分。真正的問題不是 Llama 4 能不能聊天，而是它能不能幫工程師少花 30 分鐘找對那個 PR。\u003C\u002Fp>\u003Cp>這次 Rootly 的答案很直接：至少在這個場景裡，Llama 4 沒跑贏。下一個要問的是，Meta 會先修 benchmark，還是先修產品定位？\u003C\u002Fp>","Rootly AI Labs 用 100 筆 GitHub bug 來測 Llama 4，結果顯示它在選 PR 的編碼任務上落後於 Llama 3.3、DeepSeek v3.1 與多個 coding 模型。","rootly.com","https:\u002F\u002Frootly.com\u002Fblog\u002Fllama-4-underperforms-a-benchmark-against-coding-centric-models",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782086568903-6jm6.png","research","zh","354441d5-652c-4658-a446-14f101f5e084",[17,18,19,20,21,22],"Llama 4","Rootly","benchmark","coding models","GitHub bug","Meta",[24,25,26],"Rootly 用 100 筆 GitHub bug 測 Llama 4，結果只有 69.5% 準確率。","Llama 4 甚至輸給 Llama 3.3，且落後 DeepSeek v3.1、GPT-4o 與多個 coding 模型。","對開發者來說，這提醒大家別只看通用模型宣傳，編碼任務更該優先看專門模型。",0,"2026-06-22T00:02:22.337854+00:00","2026-06-22T00:02:22.33+00:00","0c35a120-52fc-41fc-afa3-d404eb934158",{"tags":32,"relatedLang":36,"relatedPosts":40},[33,35],{"name":17,"slug":34},"llama-4",{"name":19,"slug":19},{"id":15,"slug":37,"title":38,"language":39},"rootly-benchmark-llama-4-trails-coding-models-en","Rootly benchmark: Llama 4 trails coding models","en",[41,47,53,59,65,71],{"id":42,"slug":43,"title":44,"cover_image":45,"image_url":45,"created_at":46,"category":13},"422953c3-97a9-4981-b06b-8a8383bd7419","8tai-jiqiren-bao-shiyan-liucheng-zuo-cheng-bihuan-zh","8台机器人把實驗流程做成閉環","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782073091466-pbxi.png","2026-06-21T20:17:40.866759+00:00",{"id":48,"slug":49,"title":50,"cover_image":51,"image_url":51,"created_at":52,"category":13},"2a17250c-5c06-4d19-ac3b-67d3abe4d7c7","xtragpt-paper-revision-human-ai-collaboration-zh","XtraGPT 讓論文改稿有控制感","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782066792613-enzs.png","2026-06-21T18:32:49.170651+00:00",{"id":54,"slug":55,"title":56,"cover_image":57,"image_url":57,"created_at":58,"category":13},"411c8eae-4b17-49a6-bc60-a72749c85a3d","skill-to-lora-cuts-agent-token-overhead-zh","Skill-to-LoRA 讓技能別再吃 Token","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781993875553-e5u3.png","2026-06-20T22:17:30.645442+00:00",{"id":60,"slug":61,"title":62,"cover_image":63,"image_url":63,"created_at":64,"category":13},"e3e27211-1d3e-41d5-bc4e-828679944083","turboquant-does-not-hurt-search-quality-equal-bytes-zh","TurboQuant 在等字節預算下不會傷害搜尋品質","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781857969634-naia.png","2026-06-19T08:32:21.766491+00:00",{"id":66,"slug":67,"title":68,"cover_image":69,"image_url":69,"created_at":70,"category":13},"ed7ed094-2671-4723-8105-a89dc805f8a9","deterministic-multicalibration-optimal-sample-use-zh","確定性多重校準終於達標","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781850776591-fs2z.png","2026-06-19T06:32:28.220144+00:00",{"id":72,"slug":73,"title":74,"cover_image":75,"image_url":75,"created_at":76,"category":13},"b84a7dd2-d3f3-428c-a37f-1ac69cb01d4b","uniego-proxy-teachers-egocentric-video-zh","UNIEGO 用代理教師統一自我中心影片","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781849878221-5dnm.png","2026-06-19T06:17:31.822125+00:00",[78,83,88,93,98,103,108,113,118,123],{"id":79,"slug":80,"title":81,"created_at":82},"f18dbadb-8c59-4723-84a4-6ad22746c77a","deepmind-bets-on-continuous-learning-ai-2026-zh","DeepMind 押注 2026 連續學習 AI","2026-03-26T08:16:02.367355+00:00",{"id":84,"slug":85,"title":86,"created_at":87},"f4a106cb-02a6-4508-8f39-9720a0a93cee","ml-papers-of-the-week-github-research-desk-zh","每週 ML 論文清單，為何紅到 GitHub","2026-03-27T01:11:39.284175+00:00",{"id":89,"slug":90,"title":91,"created_at":92},"c4f807ca-4e5f-47f1-a48c-961cf3fc44dc","ai-ml-conferences-to-watch-in-2026-zh","2026 AI 研討會投稿時程整理","2026-03-27T01:51:53.874432+00:00",{"id":94,"slug":95,"title":96,"created_at":97},"cf046742-efb2-4753-aef9-caed5da5e32e","adaptive-block-scaled-data-types-zh","IF4：神經網路量化的聰明選擇","2026-03-31T06:00:36.990273+00:00",{"id":99,"slug":100,"title":101,"created_at":102},"53a0dc54-0371-4e40-8d5e-74e94a73840c","geometry-aware-similarity-metrics-for-neural-representations-zh","超越距離測量：用微分幾何重新理解神經網路","2026-03-31T06:01:01.241968+00:00",{"id":104,"slug":105,"title":106,"created_at":107},"fee7d472-a775-4b1d-bbc2-1e8bca1bbf8b","on-the-fly-repulsion-in-the-contextual-space-for-rich-divers-zh","讓AI繪圖更有創意：用排斥力提升生成多樣性","2026-03-31T06:01:25.439673+00:00",{"id":109,"slug":110,"title":111,"created_at":112},"a9901203-d69b-447b-8854-15d14eab32b4","vision-aided-beam-prediction-cnn-eca-zh","影像輔助波束預測升級 CNN","2026-04-01T10:00:25.8073+00:00",{"id":114,"slug":115,"title":116,"created_at":117},"b55e7dd4-0a24-4b3d-804d-b0309a03f498","triple-band-fss-mimo-antenna-sub-6-ghz-zh","三頻 FSS MIMO 天線瞄準 sub-6 GHz","2026-04-01T13:18:36.857305+00:00",{"id":119,"slug":120,"title":121,"created_at":122},"f68290bd-e7f3-4b30-ba22-dcd4e0130a66","openclaw-1299-repos-eight-weeks-analysis-zh","OpenClaw 1299 個 Repo 的資料解讀","2026-04-02T05:03:45.208411+00:00",{"id":124,"slug":125,"title":126,"created_at":127},"ed9f80eb-eb02-4d35-8ad4-0ddf428751dd","beam-coherence-aware-combining-mmwave-mimo-zh","毫米波 MIMO 的雙階合併法","2026-04-02T05:27:26.897188+00:00"]