A new paper, Deep Search in Real-World Web Environments (arXiv:2610.30181), puts 12 frontier models on 200 live-web research tasks. The best scores 39%. We rebuilt the measurement rig to test what it actually says, and the headline is not "the web is hard". It is this: giving an agent search and browse tools is the easy part. Whether it spends a finite tool-call budget well is what decides the score.
What we did
The paper is an evaluation, not a new method. Every agent gets the same tools, a hard cap of 100 tool calls and 3,600 seconds, and an LLM judge marks the final answer. We reimplemented that protocol as a small, dependency-free harness: a budget-enforcing run loop, a semantic-equivalence judge, and a reporter that computes the paper's exact metrics (per-tier accuracy, mean tool calls, budget-exhaustion rate, and the tool-failure call-share versus time-share split).
We cannot run 12 frontier models over 200 live tasks here, so we split the test. First, a deterministic offline experiment: three synthetic policies (an efficient coordinator, a cheap quitter, and a thrasher that burns the budget) driven through the same engine over 90 tiered tasks. Second, a live smoke over three OpenDS-style tasks solved via real web search, to prove the harness runs end to end on real trajectories.
Why it was worth doing
All four of the paper's central claims reproduce. Accuracy decays by tier. The thrasher makes the most tool calls (75 on average) and scores the lowest (17.8%), so effort anti-correlates with accuracy, exactly the paper's finding that its chattiest model is also its worst. Budget exhaustion is a property of the model, not the task: it ranges from 0% to 33% across our policies and peaks on the thrasher, mirroring the paper's 16.5% to 81.5% spread. And tool failures are about 5% of calls but roughly 60% of wall-time, the paper's "rare but expensive" signature.
The useful part is the operational read. Two of these signals (mean tool calls and budget-exhaustion rate) need no ground truth, so you can watch them live and flag a failing run before you have any answer to grade. A thrashing agent is visible long before its bad synthesis lands.
What's still off
This reproduces the direction of the claims, not their exact magnitudes. The synthetic policies are calibrated to the paper's reported behaviour, not real model runs, and the live smoke is three author-solved tasks, so it validates the harness rather than measuring difficulty. The judge is a normalise-and-alias matcher, fine for unique-entity answers and weaker on long free-text. A full run would need the 200-task set and a real policy model, which is the obvious next step.
What's now in the stack
opends.py: the protocol engine (budget-enforcing run loop, judge, metrics reporter).mock_run.py: the deterministic mechanism experiment, with all four claim checks passing.live_run.py: the live web smoke over recorded trajectories.- A portable lesson for our own
/deep-researchskill: instrument it with the three cheap, ground-truth-free signals this rig computes and alert when a run thrashes.