Wink Pings

A Year of Clicking Web Pages, Then Letting the Model Write Code: Browser Use’s Pivot After Being Stuck at 60%

Tabbit team’s year-and-a-half Browser Use retrospective: the screenshot-and-click success rate stayed stuck at 60%, and after models could write code reliably, the approach flipped overnight. Codex completed in under half an hour a configuration that would have taken over an hour by hand; 200+ entries were loaded in about two minutes; benchmark accuracy was only one success higher—what was saved was tokens and waiting time.

In the Tabbit team’s year-and-a-half Browser Use retrospective, there is a very clear dividing line: when the model looked at screenshots and clicked on pages, the success rate stayed around 60% for a long time; after the model could reliably write code, the approach flipped overnight.

Browser Use 转向的讨论截图

First, a note on stance. The original text comes from a tweet by yibie; the main body is the Tabbit team’s retrospective, and the ending promotes their own browser. The conclusions can be referenced, but keep the stance in mind. Original article link: https://liaocaoxuezhe.com/article/the-bitter-lesson-of-browser-use , tweet link: https://x.com/yibie/status/2121803108808638601 . A netizen summarized it even more briefly: after a year of practicing clicking locations by looking at images, they ultimately turned the operations into code.

浏览器操控的苦涩教训封面

Phase 1’s approach was very intuitive: let the Agent view web pages like a human. Browser Use 0.1 was open-sourced in November 2024, giving the model ten tools—search, open URL, click element, enter text, extract main content, switch tabs—and after each round of actions it took a new screenshot and looked at the page again. The team compressed the DOM into hundreds of tags, clipped out invisible elements, and over a year tested hundreds of model-plus-harness combinations, yet still got stuck on slowness and inaccurate clicking.

Browser Use 0.1 的看页面、选动作、执行、再看页面循环

The details are stark. Claude 3.5 Sonnet and Opus also drifted by dozens of pixels on real screenshots; Seed 1.8 was strong at clicking, but often got dropdowns wrong and was weaker in reasoning; Seed 2.0’s GUI performance actually regressed; in internal evaluations, Gemini 3.5/3.6 Flash underperformed Claude 3.7 Sonnet. On canvas pages such as Google Sheets and Feishu Bitable, DOM information did not help and could even interfere with the model’s choices.

Canvas 页面里 DOM 信息帮不上忙

Seed 1.8 once offered a glimmer of hope for GUI clicking. ByteDance added large amounts of real mobile and GUI operation trajectories to training; click-task accuracy was high, but components not found on phones seemed clumsy. This detail illustrates the problem well: an improvement in a single capability does not mean an entire web task can be completed end to end.

Seed 1.8 曾让人看到 GUI 点击的曙光

The turning point for Phase 2 came from an admin configuration task. The original author estimated that doing it manually would take over an hour; the Codex Chrome extension finished it in less than half an hour, and a ten-minute check found no omissions. After getting the first one working, 200+ configurations were loaded in about two minutes with one for loop. Codex mainly wrote code quietly, preferring Playwright, with visual coordinates and CDP only as fallbacks.

需要配置的后台界面

屏幕前的顿悟

Looking inside Codex, you can see four kinds of interfaces: Playwright first; DOM-CUA reads interactive elements; visual CUA uses coordinates as a fallback; raw CDP handles final requirements. Browser operations were abstracted into a programming problem, and code writing—what models are best at—became the main path.

Codex 把浏览器操作抽象成接口

Phase 3: Tabbit CLI launched in August 2026. The Agent controls a real browser through the CLI, preserving pages, variables, and progress, reusing login state, working in the background without taking over the window. It prefers Playwright and drops down to CDP for low-level requirements. The Browser Use camp went the opposite way, having the model write native CDP directly. Both sides agree that the Coding Agent Harness is the most mature approach; they differ only on how high the abstraction layer should be.

Tabbit CLI

The Benchmark data needs to be read clearly. BU Bench V1 used 25 questions times 3 runs, 75 runs total, with the same model, judged by a reviewer model. Tabbit had the highest accuracy, but only one more success than Codex Chrome; the original article says this cannot be used to declare an overall capability gap. The difference is mainly in cost: input tokens per correct answer were about 49% lower than Codex Chrome and about 61% lower than Agent Browser; P90 latency was 220.6 seconds versus 489.0 seconds. These are cost metrics and should not be read as an accuracy gap.

BU Bench V1 测试结果

Browser Use also turned. In April 2026, it published a post acknowledging that browser operations should not be wrapped too rigidly, giving the Coding Agent low-level CDP access and allowing it to modify its own tool code. In September, it further confirmed that the Coding Agent Harness is the most mature Browser Use approach. The paths converge; Tabbit is just more pragmatic, prioritizing Playwright’s mature abstractions and dropping lower only for special needs.

Phase 4: Jev’s counter-evidence is worth examining. On 165 screenshots, Jev’s state-judgment accuracy was 95.2%, on par with Kimi K3, with latency of 0.76 seconds versus 8.54 seconds; on 36 localization tasks, Jev got 35 right, while K3 got all of them right. But when placed into 10 tasks, only 2 of 118 steps were decided by Jev, and K3 calls did not decrease; planning first and then letting Jev click did not trigger, and Jev actually made 0 clicks. The author explicitly says the sample is small and this cannot be used to conclude that Jev is ineffective. Being faster on a single step does not mean being faster over a long horizon.

Jev 操作浏览器

The original article concludes with Sutton’s 2019 The Bitter Lesson. Researchers often encode human knowledge into systems; this works in the short term but hits bottlenecks in the long term, and breakthroughs usually come from scaling computation and search/learning. Browser Use’s experience is a footnote: the more detailed the scaffolding, the more it may constrain the model. Models are best at writing code; when browser operations are abstracted into a code problem, a tipping point appears. Give it code-execution capability, and it can organize loops and conditions; give it persistent state, and it can save intermediate results; give it low-level interfaces, and it can handle edge cases the framework did not foresee.

The Bitter Lesson

There is also a side branch: WebMCP. Websites use navigator.modelContext to proactively declare structured tools for agents to use, such as querying orders or adding products. Agents do not have to infer website capabilities entirely from the DOM. It will not immediately replace general Browser Use; many websites will not be retrofitted right away, and permissions, security, and result verification still require complete mechanisms. But it may become the next important side branch: general code interfaces cover the existing internet, while WebMCP lets willing websites provide more stable business actions.

This lesson is not easy for people building tools. Designs once considered a source of pride may also become obstacles to the next step. Tabbit’s conclusion has a promotional stance, but this shift from GUI to Coding is worth comparing against their own approaches for people building agents.

发布时间: 2026-11-15 14:29