METRICS (Metrological Evaluation and Testing of Robots in International CompetitionS) organises challenge-led and industry-relevant competitions in the four Priority Areas (PAs) identified by ICT-09-2019-2020: Healthcare, Infrastructure Inspection and Maintenance (I&M), Agri-Food, and Agile Production.
Within METRICS, AIRLab is in charge of the ACRE (Agri-food Competition for Robot Evaluation) competition, dedicated to benchmarking agricultural robots.
Benchmarking means objectively measuring the performance of a robot when executing a task. Being able to benchmark robot systems is necessary to compare their performance, and thus to better understand their strenghts and weaknesses. Both research and industry need this to progress.
However, when dealing with autonomous agents benchmarking is tricky. How to devise testing procedures that yield objective results? What metrics capture the key points of the robot performance? How can robots that perform the same complex action differently be compared?
AIRLab has been working on these issues for a long time, accumulating experience in both methodology and real-world benchmark design, setup and execution.
Over the years, we participated -and are participating- to many European projects about robot benchmarking, including RAWSEEDS (FP6), RoCKIn (FP7), RockEU2 (H2020), RobMoSys (H2020), EUROBENCH (H2020), SciRoc (H2020), METRICS (H2020).
MADROB (Modular Active Door for RObot Benchmarking) and BEAST (Benchmark-Enabling Active Shopping Trolley) are benchmarks for autonomous robots aimed at measuring their capabilities and performance when dealing with devices that are common in human environments.
MADROB is focused on opening doors; BEAST considers the problem of pushing a shopping trolley. Both make use of a device with the same features of its real counterpart, fitted with sensors (to assess the actions of the robots on it: e.g., force applied to the handle of the door, precision in following a trajectory with the cart) and actuators (to introduce disturbances simulating real-world phenomena: e.g., wind pushing the door panel, stone under the trolley’s wheel).
Beyond the hardware and software, MADROB and BEAST also comprise procedures and performance metrics that enable objective evaluation of the performance of robots, as well as comparisons between different robots and between a robots and humans.
The ERL is the last evolution of a long-standing line of work about benchmarking through competitions: i.e., rethinking robot competitions to transform them into tools for objective evaluation of the performance of autonomous robot systems. All this without taking away the elements of fun and challenge that make competitions interesting.