When “Done” Isn’t Done
Five models attempted a physical clearing and wiping task. Across 150 scored trials, none completed it. The partial results and completion claims tell a more revealing story.
Read articleReal-world benchmarks for intelligent machines
Coop runs independent, industry-informed benchmarks on real robots. Compare model checkpoints, understand failures, and measure progress with each release.
Backed by
Working with researchers from
Our second benchmark
150 scored trials across five models.T2-standard-09Two views · 8× playback
Top view
Front view
2 / 3 objects placed · 0% ink erased
Illustrative trials at 8× speed. Results beneath each clip refer to that trial only. Starting conditions differ. Each run failed the full-task criteria.
How we benchmark
Trust starts with the protocol
Industry operators inform what we test. Model developers use the results to improve their policies. Every evaluation defines the robot configuration, task conditions, and scoring rules upfront.
01Turn workflows into testable tasks
Industry workflows inform the tasks and success criteria. With your team, we agree on the protocol, confirm hardware compatibility, and select the checkpoints to evaluate.
02Make every trial count
Run compatible checkpoints on the same robot with matched starting conditions and documented resets. Record successes, failures, and interventions to make comparisons useful.
03Put results to work
Get task-level comparisons, failure analysis, and trial recordings to guide your next development cycle. Repeat the evaluation as your model evolves; findings apply to the tested hardware and configuration.
The Coop journal
Research & insights
The questions, methods, and physical work behind our benchmarks.
Spur: A High-Level Controller for the Unitree R1
A high-level motor controller for the Unitree R1 did not exist. So we built it. Spur keeps the robot balanced while its arms and hands work, plans every motion before anything moves, and is now open source.
Read article
Coop’s First Benchmark: A Sign Wasn’t Enough
Four models, one humanoid, 220 trials. What a DO NOT ENTER sign revealed about instruction following, physical execution and reported success.
Read article
The 99% Problem: Why Robot Benchmarks Stopped Telling Us Anything
Why physical evaluation matters, where simulation falls short, and what benchmark saturation hides.
Read article