The Artificial Intelligence and Robotics Lab of Politecnico di Milano
METRICS (Metrological Evaluation and Testing of Robots in International CompetitionS) organises challenge-led and industry-relevant competitions in the four Priority Areas (PAs) identified by ICT-09-2019-2020: Healthcare, Infrastructure Inspection and Maintenance (I&M), Agri-Food, and Agile Production.
Within METRICS, AIRLab is in charge of the ACRE (Agri-food Competition for Robot Evaluation) competition, dedicated to benchmarking agricultural robots.
Benchmarking means objectively measuring the performance of a robot when executing a task. Being able to benchmark robot systems is necessary to compare their performance, and thus to better understand their strenghts and weaknesses. Both research and industry need this to progress.
However, when dealing with autonomous agents benchmarking is tricky. How to devise testing procedures that yield objective results? What metrics capture the key points of the robot performance? How can robots that perform the same complex action differently be compared?
AIRLab has been working on these issues for a long time, accumulating experience in both methodology and real-world benchmark design, setup and execution.
Over the years, we participated -and are participating- to many European projects about robot benchmarking, including RAWSEEDS (FP6), RoCKIn (FP7), RockEU2 (H2020), RobMoSys (H2020), EUROBENCH (H2020), SciRoc (H2020), METRICS (H2020).
Contact: Matteo Matteucci
MADROB (Modular Active Door for RObot Benchmarking) and BEAST (Benchmark-Enabling Active Shopping Trolley) are benchmarks for autonomous robots aimed at measuring their capabilities and performance when dealing with devices that are common in human environments.
MADROB is focused on opening doors; BEAST considers the problem of pushing a shopping trolley. Both make use of a device with the same features of its real counterpart, fitted with sensors (to assess the actions of the robots on it: e.g., force applied to the handle of the door, precision in following a trajectory with the cart) and actuators (to introduce disturbances simulating real-world phenomena: e.g., wind pushing the door panel, stone under the trolley’s wheel).
Beyond the hardware and software, MADROB and BEAST also comprise procedures and performance metrics that enable objective evaluation of the performance of robots, as well as comparisons between different robots and between a robots and humans.
Contact: Matteo Matteucci
For additional details: http://eurobench2020.eu/developing-the-framework/modular-active-door-for-robot-benchmarking-madrob/, http://eurobench2020.eu/developing-the-framework/benchmark-enabling-active-shopping-trolley-beast/
Plug&Bench’s is a subproject of European project RobMoSys.
Plug&Bench expanded the RobMoSys framework with new elements enabling the definition of standardized and easy to use performance benchmarks.
Contact: Matteo Matteucci
For additional details: https://robmosys.eu/plugbench/
The ERL is the last evolution of a long-standing line of work about benchmarking through competitions: i.e., rethinking robot competitions to transform them into tools for objective evaluation of the performance of autonomous robot systems. All this without taking away the elements of fun and challenge that make competitions interesting.
Contact: Matteo Matteucci
For additional details: https://www.eu-robotics.net/robotics_league/
SciRoc is a EU-H2020 funded project supporting the European Robotics League (ERL), whose aim is to bring ERL tournaments in the context of smart cities.
Contact: Matteo Matteucci
For additional details: https://sciroc.eu/