Insights

Robotics · Sep 24, 2026

Physical AI Will Be Judged by Its Failure Modes

In robotics, average performance is a poor guide to safety. What matters is how the system behaves when its perception is wrong.

SkyNet AI

The average is a comfortable fiction

A perception system can be 97 percent accurate and still be unsuitable for a physical task if its errors cluster in a specific condition: reflective surfaces, low light, occlusion, or motion. The average hides exactly the information that determines whether the system is safe.

Evaluate by condition

Condition-stratified evaluation changes the conversation. Instead of a single number, the team produces a map of where the system works and where it does not. That map is directly usable: it defines operating conditions and exclusion zones.

Knowing when to stop

A system that estimates its own uncertainty and hands control back is more valuable than one that is marginally more accurate but confidently wrong. This is a design choice, and it has to be made early, because it shapes the entire control architecture.

Simulation is not validation

Simulation is useful for training and for controlled adversarial evaluation, but simulated results must be reported with their fidelity assumptions. A policy that succeeds in a twin has demonstrated something about the twin. Whether it transfers is an empirical question that requires physical testing.

The discipline here is unglamorous: define conditions, test in them, report honestly, and restrict operation to the envelope you have actually verified.