NIST Proposes Benchmark for Humanoid Robot Performance

The development of commercially successful humanoid robots has sparked a global race, and with it, the need for a standardized way to evaluate their performance. The National Institute of Standards and Technology (NIST) has taken a significant step towards addressing this need by proposing a comprehensive method to assess the minimum expected physical capabilities of humanoid robots. NIST's proposal, which builds on its previous collaboration with the Defense Advanced Research Projects Agency (DARPA), outlines a low-footprint set of locomotion and manipulation tasks that use previously defined and standardized test methods and performance metrics.

These tasks are designed to represent the minimum capabilities required for humanoid robots in various applications, including industrial, household, and healthcare settings. According to Aaron Prather, director of the Robotics & Autonomous Systems Program at ASTM International, this benchmark is long overdue, as the industry has seen significant investment in humanoid platforms, including Tesla's Optimus, Figure, Agility, Apptronik, and Unitree, without a standardized way to measure their capabilities. The proposed benchmark, developed in collaboration with industry and the research community, includes an apparatus for testing humanoid capabilities.

NIST plans to build a limited number of these testing apparatuses for free distribution to U.S. humanoid robot manufacturers and established regional testing facilities. The institute also plans to publish the designs and 3D models of the apparatus for use as a physical and/or virtual testbed for robot training and control development.

By aggregating test results, NIST aims to showcase the state of the art of humanoid robot capabilities. Robot manufacturers that choose to test their robots could receive an apparatus to run their own tests or test their robots at NIST or a participating facility. The test results will be collected under pre-approved data-sharing agreements to protect intellectual property and any attribution if needed.

Interested parties can learn more or participate by contacting NIST through a contact form or directly at benjamin.beiter@nist.gov and kamel.saidi@nist.gov. This move by NIST comes as other organizations, such as Fraunhofer IPA, have also proposed their own benchmarks for humanoid safety and development. With a standardized performance benchmark in place, the industry can expect to see more transparency and accountability in the development of humanoid robots.

The proposed benchmark is a significant step towards establishing a common language and set of expectations for humanoid robot performance, and it will be interesting to see how the industry responds to this development.