Why we’re building Coop

The problem

Training results can tell a team a lot about its model. Turning those results into evidence of what a robot can reliably do is a different process.

That process starts with deciding what to test. A task can make for a compelling demonstration without telling you much about the work someone actually needs a robot to do. Picking up an object once is one thing. Picking it up consistently across the positions, orientations, and surrounding clutter it might encounter is another.

Then there’s the hardware. A robot has to be paired correctly with the model. Camera placement, sensor configurations, control frequencies, action spaces, and other details have to match what the policy expects. Mismatches can produce results that look like model failures but are actually integration failures.

Once the hardware is up and running, someone has to calibrate the workspace precisely enough that a trial run today can be meaningfully compared with a trial run next week. Then the trials have to happen, enough times to say something useful.

Seems easy enough, right?

In the lab, an object gets knocked over. A gripper slips halfway through a task. Someone has to reset the scene the same way, over and over, and log every human intervention so a helping human hand isn’t counted as a robot success.

A single successful demonstration tells you that something worked once, under those conditions. It doesn’t say whether the policy succeeds 95% of the time or 5% of the time. It doesn’t tell you whether the result holds when the lighting changes, the object moves, or the robot gets a different gripper.

And when two model versions are tested under different conditions, it becomes difficult to tell whether the model improved or the test simply got easier.

None of this is a novel observation. Researchers and robotics teams have been working on these problems for years. Building a useful benchmark requires more than choosing a task and recording a success rate. The task has to matter, the comparison has to be fair, and the result has to make clear what it actually establishes.

At Coop, that’s the work we want to do.

We’re building independent, industry-specific benchmarks that help model developers understand:

1. What their models can do consistently.
2. Where and how they fail.
3. Whether the next version is actually better.

What we’re building

Our starting point is simple. We talk to people in industries where robots could take on useful work, understand the tasks they want automated, and turn those requirements into measurable tests.

Those conversations inform what we benchmark. Model developers are the teams we’re building the evaluations for.

The goal is to give a developer something concrete to work with. If a new checkpoint improves overall success but struggles more with certain object positions, that matters. If a model completes a task only because someone keeps stepping in, that matters too. A single score can hide both.

We’re starting with a focused set of experiments on the Unitree R1. As we develop benchmarks for specific industries, we’ll expand the tasks and robot configurations we can evaluate.

The hardware matters here. A result on one robot is evidence about that tested setup. It is not automatically a promise about another robot with a different gripper, camera, or controller. We’ll be explicit about those boundaries and expand our coverage as we establish additional configurations.

Before anything runs, there’s plenty of unglamorous work to do. That happens to be our favorite part.

We confirm compatibility, define the task, and decide what actually counts as success. We specify which starting conditions will vary and which parts of the experiment stay fixed. Then we test compatible models or checkpoints under matched conditions, enough times that the result carries substance.

What comes back to the developer is the entire receipt:

1. Task-level results and comparisons between checkpoints.
2. The recordings and logs behind those results.
3. A breakdown of what failed, and how.
4. Every time a human had to step in.
5. The hardware and conditions the results hold under.

When the next checkpoint is ready, we can run the evaluation again. Over time, those comparisons should help teams see what is improving, what is regressing, and where to focus their next development cycle.

What comes next

Future posts will cover our methods in more detail: how we choose tasks, how we make comparisons meaningful, and what our experiments teach us, including the ones that don’t go as planned. As that work develops, we intend to publish more formal research reports and papers.