Map the Failure Boundary Traditional software relies on unit tests. Non-deterministic AI, from LLM agents to RL policies, relies on static benchmarks and single-trajectory evaluations. But knowing a system succeeded once doesn’t tell you how reliably it will succeed again. We built Monte to map the failure boundaries of decision-making systems. As a first test of the instrument, we froze a trained Unitree Go1 locomotion policy and tested it against 6,400 combinations of floor friction and lateral push. The result is a map of where the policy recovers and where it falls. For each condition, Monte spins up the physics, runs repeated rollouts, and estimates the probability of surviving a five-second episode. The same system lets us probe individual worlds, inspect the resulting trajectories, and re-run the experiment after retraining. 6,400 rollouts · 20×20 conditions · 16 trials per cell · 126 s of GPU time At a glance
- the failure boundary is broad and variable: a band, not a threshold
- below μ ≈ 0.15, the policy usually falls before the push matters
- retraining on pushes + low friction moved the boundary outward in 60 cells and worsened none
- the full experiment, including retraining and the repeat sweep, cost $42 Monte runs the sweep, the statistical analysis, the live probe, and the surface comparison. The result The boundary is diagonal: grippier floors tolerate harder pushes. On medium friction the policy survives pushes up to roughly its own bodyweight (about 125 N for this 12.7 kg robot) and fails beyond that. An earlier back-of-the-envelope estimate had put the boundary near 15 N; the measured value is several times higher, which is why this study reports only measured numbers. The surprising failure mode. Below μ ≈ 0.15 the map goes dark at the left edge: the robot falls at a 10% bodyweight nudge, or with no push at all. On ice the policy can barely walk. That is a different failure mode from being knocked over (the gait itself collapses), and it lives...