Performance Metrics and Test Arenas for Autonomous Mobile Robots

nist_arena_2000Reproducible and widely known challenges can help evolving fields by providing reference problems with measures of performance which allow researchers to compare implementations, communicate results, and leverage each other’s work. The National Institute of Standards and Technology (NIST) developed the Reference Test Arenas for Autonomous Mobile Robots to focus research efforts, provide direction, and accelerate the advancement of mobile robot capabilities. These arenas, modeled from buildings in various stages of collapse, allow objective performance evaluation of robots as they perform a variety of urban search and rescue (USAR) tasks in both physical and virtual domains.

A reality arena, composed of an above and below ground structure covering approximately 300 square meters, further challenges those implementations that demonstrate successful performance in the physical arenas by forcing them to encounter a more rigorous and real-world environment.

Additional arenas are currently being designed to complete the spectrum of reference test environments. A “Confined Space Maneuver” arena is proposed that will be capable of testing shape shifting robotic implementations and other necessary robotic capabilities, along with evaluating an operator’s situational awareness. An outdoor “Scout Arena” is also in development that will challenge lightweight Unmanned Ground Vehicles (UGVs < 225 kg) by presenting an equestrian style obstacle course complete with ditches, tall grass, woods, rocks, and other outdoor obstacles (both natural and man-made). It is the intent that these test arenas will comprise a standardized reference facility that will assist in the development and evaluation of various mobile robotic systems.

Since 2000, the NIST USAR Arenas have inspired not only replicas to be fabricated, but they have provided the foundation for the annual Rescue Robot Competitions that are currently part of the RoboCup – USOpen, RoboCup– Japan Open, RoboCup, and the International Joint Conference on Artificial Intelligence (IJCAI)/American Association for Artificial Intelligence (AAAI) conferences. They have traveled throughout North America hosting the AAAI/IJCAI Robot Rescue Competitions for the past four years while the RoboCup community has constructed Japanese and Italian replicas to host their Robot Rescue Competitions for the past two years. NIST will be will be involved in the administration and/or support of the 2004 events including the RoboCupRescue – USOpen (New Orleans, USA), RoboCupRescue – Japan Open (Osaka, Japan), RoboCupRescue2004 (Lisbon, Portugal) and AAAI Rescue (San Jose, USA) Robot Competitions.

The presentation (.pdf format) below presents an overview of the NIST Reference Test Arenas for Autonomous Mobile Robots (both physical and virtual), their proliferation throughout the world and the past international Robot Rescue Competitions they have inspired. Additional and detailed information regarding the active arenas and the competitions may be obtained by clicking on the appropriate link on the left.

Leave a Reply

Your email address will not be published.