[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-marginlab-claude-code-opus-46-tracker-zh":3,"article-related-marginlab-claude-code-opus-46-tracker-zh":28,"series-ai-agent-62a923b4-173d-465e-93f0-071226ba6119":86},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":11,"views":25,"created_at":26,"published_at":27,"topic_cluster_id":11},"62a923b4-173d-465e-93f0-071226ba6119","marginlab-claude-code-opus-46-tracker-zh","Marginlab 盯上 Claude Code 漂移","\u003Cp>\u003Ca href=\"https:\u002F\u002Fmarginlab.ai\u002Ftrackers\u002Fclaude-code\u002F\" target=\"_blank\" rel=\"noopener\">Marginlab\u003C\u002Fa> 做了一個每天跑的追蹤器。\u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fclaude-code\" target=\"_blank\" rel=\"noopener\">Claude Code\u003C\u002Fa> 搭配 \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-4-6\" target=\"_blank\" rel=\"noopener\">Opus 4.6\u003C\u002Fa>，每天固定測 50 題。它還會做週報和月報。講白了，就是盯著模型有沒有慢慢歪掉。\u003C\u002Fp>\u003Cp>這件事很實際。很多 coding agent 在 demo 看起來很猛。真的上線後，表現卻會飄。Marginlab 直接抓 \u003Ca href=\"https:\u002F\u002Fwww.swebench.com\u002F\" target=\"_blank\" rel=\"noopener\">SWE-Bench-Pro\u003C\u002Fa> 的子集合來跑。它不用花俏包裝。它想看的，就是使用者真正在 CLI 裡碰到的結果。\u003C\u002Fp>\u003Ch2>每天到底在看什麼\u003C\u002Fh2>\u003Cp>這個 tracker 的核心是 pass rate。也就是 50 題裡，Claude Code 解對幾題。這個數字最直觀。你一眼就知道今天有沒有掉。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147267370-bbwz.png\" alt=\"Marginlab 盯上 Claude Code 漂移\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>但 Marginlab 沒有只看分數。它還顯示 input tokens、output tokens、API cost、平均 runtime，還有總 tool calls。這很重要。因為 agent 出問題時，常常不是只掉準確率。\u003C\u002Fp>\u003Cp>有時候它會一直試。工具呼叫暴增。runtime 也變長。這種情況下，模型可能還能解題，但效率已經變差。對開發者來說，這就是成本開始失控的前兆。\u003C\u002Fp>\u003Cul>\u003Cli>每天固定跑 50 題\u003C\u002Fli>\u003Cli>每週與每月做彙總\u003C\u002Fli>\u003Cli>用 Bernoulli trials 看 pass rate\u003C\u002Fli>\u003Cli>顯示 95% 信賴區間\u003C\u002Fli>\u003Cli>直接跑 Claude Code CLI，不加自製 harness\u003C\u002Fli>\u003C\u002Ful>\u003Cp>最後一點最關鍵。很多 benchmark 一包再包，最後量到的是 wrapper，不是模型。Marginlab 直接走 Claude Code CLI。這比較接近真實開發流程。說真的，這種做法比較不會自嗨。\u003C\u002Fp>\u003Cp>它也有一個 degradation status pan\u003Ca href=\"\u002Fnews\u002Fzellij-0440-windows-remote-sessions-cli-zh\">el\u003C\u002Fa>。這裡會把樣本數和統計門檻一起放出來。不是每次掉分都算數。樣本太少時，波動本來就很大。這點比很多只會貼分數圖的頁面誠實多了。\u003C\u002Fp>\u003Ch2>為什麼現在要做這個\u003C\u002Fh2>\u003Cp>Marginlab 說，這個 tracker 是回應 \u003Ca href=\"https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fclaude-degradations\" target=\"_blank\" rel=\"noopener\">Anthropic\u003C\u002Fa> 在 2025 年 9 月的 degradation postmortem。那份說明很直接。模型上線後，表現還是可能往下掉。不是每次更新都只會變好。\u003C\u002Fp>\u003Cp>這也是很多團隊最怕的事。你昨天還覺得 agent 很穩。今天突然開始多跑幾輪。或者同一題要花更多 token。這種變化不會立刻炸掉服務，但會慢慢吃掉你的成本和信心。\u003C\u002Fp>\u003Cp>Marginlab 也強調自己是獨立第三方。它沒有跟 frontier model provider 綁在一起。這點很重要。因為很多 d\u003Ca href=\"\u002Fnews\u002Fgemini-live-gets-major-upgrade-with-3-1-flash-live-zh\">ash\u003C\u002Fa>board 很像行銷頁。分數一高就大吹特吹。Marginlab 比較像在做監控，不是在做宣傳。\u003C\u002Fp>\u003Cblockquote>“We want to offer a resource to detect such degradations in the future.” — Marginlab\u003C\u002Fblockquote>\u003Cp>這句話很直白。它不是在做排行榜。它是在做預警器。模型如果在 reasoning、tool use，或長任務表現上開始漂移，日更追蹤就有機會先抓到。\u003C\u002Fp>\u003Cp>我覺得這種工具會越來越重要。因為 agent 的問題，常常不是一次壞掉。是慢慢變鈍。等使用者抱怨時，很多跡象早就出現了。\u003C\u002Fp>\u003Ch2>數字怎麼看，才不會被騙\u003C\u002Fh2>\u003Cp>Marginlab 最有意思的地方，在於它把樣本數和統計門檻講清楚。50 題的日測，本來就很吵。它沒有假裝每天的波動都很有意義。這種誠實很少見。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147268472-6a4w.png\" alt=\"Marginlab 盯上 Claude Code 漂移\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>它的估算是這樣。50 題時，大約要 ±13.8% 的變化，才比較能過 p < 0.05。350 題時，門檻縮到 ±4.8%。1,400 題時，門檻只剩 ±2.3%。樣本越多，越能分辨噪音和真問題。\u003C\u002Fp>\u003Cp>這對 coding agent 特別重要。因為 agent 的行為很不穩。今天可能多試幾次。明天可能少試幾次。runtime、tool calls、token 數都會晃。只看單日分數，很容易誤判。\u003C\u002Fp>\u003Cul>\u003Cli>50 題：約 ±13.8% 才容易有統計意義\u003C\u002Fli>\u003Cli>350 題：約 ±4.8%\u003C\u002Fli>\u003Cli>1,400 題：約 ±2.3%\u003C\u002Fli>\u003Cli>日、週、月都會做聚合\u003C\u002Fli>\u003Cli>同時看 pass rate、runtime、tool calls、token、成本\u003C\u002Fli>\u003C\u002Ful>\u003Cp>這組數字告訴我們一件事。小樣本很適合快速掃描。大樣本才適合下結論。很多團隊在內部看板只放一條分數線，結果天天被噪音搞心態。Marginlab 至少把這件事講開了。\u003C\u002Fp>\u003Cp>如果拿競品來比，差異也很明顯。像 \u003Ca href=\"https:\u002F\u002Fswebench.com\u002F\" target=\"_blank\" rel=\"noopener\">SWE-Bench\u003C\u002Fa> 本身偏向標準化評測。\u003Ca href=\"https:\u002F\u002Fopenai.com\u002Findex\u002Fopenai-codex\u002F\" target=\"_blank\" rel=\"noopener\">OpenAI Codex\u003C\u002Fa> 這類產品展示時，通常更強調能力與體驗。Marginlab 則把焦點放在持續監控。它問的不是「能不能跑贏一次」，而是「能不能一直維持」。\u003C\u002Fp>\u003Ch2>這種監控，為什麼對台灣團隊有用\u003C\u002Fh2>\u003Cp>現在很多軟體團隊都在把 \u003Ca href=\"\u002Fnews\u002Fai-slop-flooding-streaming-services-musicians-fight-back-zh\">AI\u003C\u002Fa> agent 放進流程。寫 code、改 bug、補測試、查 issue，通通有人想交給 LLM。問題是，模型不是靜態元件。今天的表現，和下週可能不一樣。\u003C\u002Fp>\u003Cp>台灣很多團隊資源不算多。你不可能每次模型更新都人工驗證一輪。這時候就需要固定的監控機制。每天跑一小批題目，配上週報和月報，至少能先知道有沒有異常。\u003C\u002Fp>\u003Cp>這種做法也適合跟自家觀測系統搭配。你可以把 pass rate 當功能指標。把 token 和成本當財務指標。把 runtime 和 tool calls 當效率指標。三個一起看，比只看一個漂亮分數實在多了。\u003C\u002Fp>\u003Cp>再往前想一步，這也會影響採購決策。當你要選 Claude、GPT，或其他 LLM API 時，除了價格和上下文長度，也該看它在你自己的任務上，會不會隔幾天就漂一次。這不是學術問題。這是產品風險。\u003C\u002Fp>\u003Cp>我自己的判斷很簡單。未來做 agent 的團隊，會越來越像在管伺服器。不是只裝好就算了。你還要看健康度、延遲、錯誤率，還有版本變更後的差異。模型監控會變成基本功。\u003C\u002Fp>\u003Ch2>接下來該怎麼看這類 tracker\u003C\u002Fh2>\u003Cp>如果你有在用 Claude Code，我會建議先把這個 tracker 收藏起來。不要只看某一天掉分。要看連續幾天的趨勢。再對照 runtime、tool calls 和 token 數。這樣比較不會被單日噪音騙到。\u003C\u002Fp>\u003Cp>更實際一點，團隊可以自己做一個小版。固定任務集。固定執行路徑。固定記錄成本和延遲。只要你有 30 到 50 題的穩定樣本，就已經能抓到不少異常訊號。\u003C\u002Fp>\u003Cp>我猜接下來會有更多這種獨立追蹤器。原因很簡單。模型更新太快了。只看官方公告不夠。只看 demo 也不夠。你需要的是每天都在跑的數據。\u003C\u002Fp>\u003Cp>所以問題不是「模型某次考幾分」。問題是「它下週還能不能維持」。如果你的產品真的靠 coding agent 吃飯，那這種監控最好現在就做起來。\u003C\u002Fp>","Marginlab 每天跑 50 個 SWE-Bench-Pro 任務，追蹤 Claude Code Opus 4.6 的分數漂移，還會看 token、成本與工具呼叫。","marginlab.ai","https:\u002F\u002Fmarginlab.ai\u002Ftrackers\u002Fclaude-code\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1775147267370-bbwz.png","ai-agent","zh","1e86831a-5448-4953-b598-edd58f6f58d6",[17,18,19,20,21,22,23,24],"Claude Code","Marginlab","Opus 4.6","SWE-Bench-Pro","AI agent","模型監控","benchmark drift","LLM",6,"2026-04-02T16:27:31.179506+00:00","2026-04-02T16:27:31.153+00:00",{"tags":29,"relatedLang":45,"relatedPosts":49},[30,31,33,35,37,38,41,43],{"name":22,"slug":22},{"name":17,"slug":32},"claude-code",{"name":24,"slug":34},"llm",{"name":23,"slug":36},"benchmark-drift",{"name":21,"slug":13},{"name":39,"slug":40},"SWE-Bench Pro","swe-bench-pro",{"name":19,"slug":42},"opus-46",{"name":18,"slug":44},"marginlab",{"id":15,"slug":46,"title":47,"language":48},"marginlab-claude-code-opus-46-tracker-en","Marginlab Tracks Claude Code Opus 4.6 Drift","en",[50,56,62,68,74,80],{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"83c2f8f6-3710-466e-b52c-473b811f0535","how-to-set-up-openclaw-safely-zh","如何安全架設 OpenClaw","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780549368665-1t2l.png","2026-06-04T05:02:21.26625+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"0ba5b1a8-82c5-464a-bea5-9a2c8730da74","aws-devops-agent-turns-incident-chaos-into-triage-zh","AWS DevOps Agent 把事故排查變成三步","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780466689960-g1sv.png","2026-06-03T06:03:14.154923+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"841eac88-b0f0-4a4c-9e1e-efc3b5c16281","kimi-k26-live-300-agent-workflows-zh","Kimi K2.6 上線：300 代理工作流","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780430574285-hqpn.png","2026-06-02T20:02:24.972179+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"f0411957-bcdb-42d9-a267-3e90ae7d9cb1","how-to-take-a-sabbatical-at-openai-zh","怎麼申請 OpenAI sabbatical","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780398216422-8fi7.png","2026-06-02T11:02:25.74372+00:00",{"id":75,"slug":76,"title":77,"cover_image":78,"image_url":78,"created_at":79,"category":13},"37a5e429-4235-439c-9b05-bb377085462c","8-steps-build-production-rag-with-langchain-zh","8 步驟打造可上線的 LangChain RAG","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780178597493-4hz7.png","2026-05-30T22:02:48.14022+00:00",{"id":81,"slug":82,"title":83,"cover_image":84,"image_url":84,"created_at":85,"category":13},"e73c041b-852b-44c3-85aa-0f1e2e5848e3","ai-agents-hit-chaos-mode-claude-code-openclaw-zh","Claude Code＋OpenClaw 讓 AI 代理失控升溫","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780160576178-yqcs.png","2026-05-30T17:02:25.725767+00:00",[87,92,97,102,107,112,117,122,127,132],{"id":88,"slug":89,"title":90,"created_at":91},"4ae1e197-1d3d-4233-8733-eafe9cb6438b","claude-now-uses-your-pc-to-finish-tasks-zh","Claude 開始幫你操作電腦","2026-03-26T07:20:48.457387+00:00",{"id":93,"slug":94,"title":95,"created_at":96},"5bede67f-e21c-413d-9ab8-54a3c3d26227","googles-2026-ai-agent-report-decoded-zh","Google 2026 AI Agent 報告解讀","2026-03-26T11:15:22.651956+00:00",{"id":98,"slug":99,"title":100,"created_at":101},"2987d097-563f-46c7-b76f-b558d8ef7c2b","kimi-k25-review-stronger-still-not-legend-zh","Kimi K2.5 評測：更強，但還不是神作","2026-03-27T07:15:55.277513+00:00",{"id":103,"slug":104,"title":105,"created_at":106},"95c9053b-e3f4-4cb5-aace-5c54f4c9e044","claude-code-controls-mac-desktop-zh","Claude Code 也能操控 Mac 了","2026-03-28T03:01:58.58121+00:00",{"id":108,"slug":109,"title":110,"created_at":111},"dc58e153-e3a8-4c06-9b96-1aa64eabbf5f","cloudflare-100x-faster-ai-agent-sandbox-zh","Cloudflare 的 AI 沙箱跑超快","2026-03-28T03:09:44.142236+00:00",{"id":113,"slug":114,"title":115,"created_at":116},"1c8afc56-253f-47a2-979f-1065ff072f2a","openai-backs-isara-agent-swarm-bet-zh","OpenAI 挺 Isara 的 agent swarm …","2026-03-28T03:15:27.513155+00:00",{"id":118,"slug":119,"title":120,"created_at":121},"7379b422-576e-45df-ad5a-d57a0d9dd467","openai-plan-automated-ai-researcher-zh","OpenAI 想做自動化 AI 研究員","2026-03-28T03:17:42.090548+00:00",{"id":123,"slug":124,"title":125,"created_at":126},"48c9889e-86df-450b-a356-e4a4b7c83c5b","harness-engineering-ai-agent-reliability-2026-zh","駕馭工程：從「馬具」到「作業系統」，AI Agent 可靠性的終極密碼","2026-03-31T06:42:53.556721+00:00",{"id":128,"slug":129,"title":130,"created_at":131},"96d8e8c8-1edd-475d-9145-b1e7a1b02b65","mcp-explained-from-prompts-to-production-zh","MCP 怎麼把提示詞變工作流","2026-04-01T09:24:39.321274+00:00",{"id":133,"slug":134,"title":135,"created_at":136},"f2ca7720-b471-4ce5-9336-2a9ac2a876fd","amazon-bedrock-agents-multi-agent-workflows-zh","Amazon Bedrock Agents 進入多代理工作流","2026-04-01T09:30:29.945429+00:00"]