Weekly research scan [2026-07-28] - #146
Conversation
Adds three findings verified against primary sources: - WeaveBench (arXiv:2606.09426): first long-horizon hybrid-interface CUA benchmark; 114 tasks / 8 domains, best PassRate 41.2%; trajectory-aware judge shows outcome-only grading overestimates agent performance. - MAG (arXiv:2607.10079): screenshot-grounded web-agent benchmark; strongest model completes fewer than 40% of tasks. - WCAG-EM 2.0 (W3C Group Note, 23 Jul 2026): conformance evaluation methodology extended from websites to apps and other digital products. Also adds a caveat to whitepaper 5.2 noting the Hybrid Agents row is the least empirically anchored projection in the table, and that WeaveBench's judge result implies published benchmark scores are likely optimistic. Benchmark numbers remain directional projections, never presented as a BiModal A/B test. No changes to package.json, package-lock.json, or CI.
Dependabot #144 bumped prettier 3.8.x -> 3.9.6 on 2026-07-27 without reformatting the repo. Prettier 3.9 changed how it wraps TypeScript union types inside embedded code blocks, leaving `docs/implementation-guide.md` and `examples/mcp-async-tasks.md` failing `npm run format:check` on main. This is pre-existing drift unrelated to the research scan, kept as its own commit so it can be dropped or split off. Content unchanged; wrapping only.
477ce7b to
d176d8c
Compare
|
Rebased onto Two follow-ups from the rebase: 1. Reference renumbering collision (fixed). #143 renumbered 2.
Worth knowing: local Verified after rebase: |
Drop the 'strongest claim in need of the weakest support' framing. The preceding sentence already states the row rests on inference rather than evidence, and the opening sentence already calls it the least anchored projection in the table, so the phrase restated the same point a third time as an aphorism. It also overstated: the Hybrid row is the highest number in one table, not the framework's strongest claim. Epistemic content unchanged - the row is still explicitly flagged as unanchored, and the WeaveBench overestimation caution is untouched.
1 participant
Draft — for human review. Do not merge without verifying the sources below.
Weekly research scan to 2026-07-28. Three verified, material findings. Every claim is confirmed against a primary source (arXiv abstract page, W3C
/TR/document, official spec changelog). Anything I could not confirm at a primary source was cut rather than approximated — see Cut / not proposed, which is longer than the findings list this week.npm test→ 122 passed, 7 suites.prettier --checkclean. No changes topackage.json,package-lock.json, or.github/workflows.Findings
Proposed edits
docs/whitepaper.mdREADME.mdThe judgement call: the Hybrid Agents row
Whitepaper §5.2 projects 94-99% for "Hybrid Agents (UI → Protocol Handoff)" — the highest figure in the table, and the row that carries the framework's central argument. It currently has no empirical anchor at all.
WeaveBench is the closest published measurement, and it lands at 41.2%.
I did not treat that as a refutation, and the proposed text says so explicitly, because the two use "hybrid" differently:
Those are not the same pattern, and no benchmark yet isolates the second one. So the proposed wording states that the row rests on inference rather than evidence and is "the framework's strongest claim in need of the weakest support." You may consider that too self-critical for the whitepaper's voice — it is your call, and it is the one edit here I would expect you to want to reword rather than accept as drafted.
The second half of that block is less discretionary: WeaveBench's trajectory-aware judge result means the published success rates the whole table is mapped from are likely optimistic. That caveat applies to figures the framework already cites, independent of the Hybrid row.
I did not log this in
DRIFT.md. It is not a cross-artifact inconsistency — it is a claim-strength question — and PR #143 already has an unmergedDRIFT.mdedit that this would conflict with.Cut / not proposed
2026-07-28as a shipped spec. Today is the date the release was scheduled, so I checked. It has not gone final: the releases page still lists onlyMCP 2026-07-28 RCas a pre-release with2025-11-25as the latest stable, and the changelog still sits at/specification/draft/. Status is unchanged from PR Weekly research scan [2026-07-21] #143's finding — nothing new to propose.Note on window
WeaveBench's v1 is 8 June and its v3 is 6 July, so it sits slightly outside the strict 1-2 week window. It is included because the 2026-07-21 scan missed it and it speaks directly to the framework's least-supported claim. Dates are stated plainly in the reference entry rather than presented as new-this-week.
Conventions respected
Benchmark numbers remain directional projections synthesized from agent benchmarks, never presented as a BiModal A/B test — the 41.2% and sub-40% figures are reported as what those benchmarks measured about agents.
data-agent-*untouched and still supplementary. npm, not pnpm. No dated report files.