Snorkel AI vs Roo Code

This feed is Snorkel's blog, but the artifacts inside it are the product: Senior SWE-Bench, GDPval+, Cua-Bench, and a collaboration on Agents' Last Exam all land within six weeks. Roughly half the entries are evaluation results run against frontier models — Opus 5, Grok 4.5, GPT 5.5, Claude Opus 4.8 — and the other half are reading groups and conference talks arguing that measurement now lags capability. The company that built its name on weak supervision and data labeling is now selling expert-curated evaluation.

One thesis runs through every post, stated plainly by the CEO: the ability to build agents has outpaced the ability to measure them. Snorkel is monetizing that gap by owning the benchmark layer — open-sourcing the tasks, holding private splits back against contamination, and positioning its expert contributor network as the input a lab cannot cheaply replicate. The evaluation-result posts double as distribution, turning each frontier model launch into an occasion to publish a scoreboard Snorkel controls.

Expect a benchmark per additional professional domain — Cua-Bench's KiCad tasks read as the first of several vertical software targets — and a fast-turn evaluation post within days of the next frontier release. Whether the Snorkel Data Series becomes a paid product line or stays a positioning surface isn't visible in these entries.