Snorkel AI vs Writer

The output is a research and benchmarking program, not a release feed. Recent work argues that single-episode benchmarks measure the wrong thing: agents should be scored across dependent states, tool calls, simulated users, approval rules, and learning carried between tasks. Concrete artifacts back the argument — Senior SWE-Bench with 100 tasks from real pull requests and half the set held private, GDPval+ for professional reasoning, and collaboration on Agents' Last Exam with Berkeley RDI. Alongside these, Snorkel publishes head-to-head model evaluations of frontier releases.

Snorkel is moving from evaluation-as-scoring to evaluation-as-training signal: the milestone framing scores intermediate progress, and the continual-learning thread treats improvement across a task sequence as the thing being measured. Publishing benchmarks with private splits and running public model comparisons builds the position that Snorkel is the neutral scorer, which is what makes the enterprise environments business defensible. The through-line is that measurement, not model capability, is now the bottleneck.

Expect the milestone and continual-learning threads to converge into a named benchmark or environment suite with the same public-private split as Senior SWE-Bench. The feed carries research and events rather than product releases, so it does not indicate what ships in the platform.