Workshop @ ERF2018:
A unified scientific framework for robotics benchmarking: not a dream
Organizers: Diego Torricelli, Jose L Pons

When: Thursday 15, 10:45-12:15.
Where: ERF Conference, Tampere, Finland

Motivation and objective:

Human-centred robots such as exoskeletons, humanoids, prostheses, are becoming increasingly relevant worldwide. Many prototypes are moving out of the lab into everyday applications, in a wide range of market domains, e.g. Healthcare, Manufacturing and Consumer. The Robotic Research Agenda has emphasized benchmarking as an important instrument to assess the Technology Readiness Level (TRL) and to quantify how robotic solutions match user needs. Recent international efforts (e.g. European Robotics League, Robocup, Cybathlon, DARPA Robotic Challenge) have confirmed the interest of the scientific and industrial communities in evaluating (and comparing) the performance of robotic systems in real-life environments.

However, a consolidated benchmarking methodology for Robotics has not been reached yet. Current benchmarking approaches are focused on generic, goal-level performance indicators, which can hardly describe performance in sufficient detail to allow recognizing and quantifying the technical causes of performance. When existing, metrics are numerous and overlapping, and experimental procedures are defined from each lab independently. At the same time, most of the existing databases of robotic performance are not standardized, i.e. they lack of task labelling, data segmentation, data format, physical objects dimensions, type of measurement system. All these factors undermine the reproducibility of results and impede researchers and companies to build upon previous knowledge.

To fill these gaps, the EUROBENCH project will create the first European framework for the application of benchmarking methodology on robotic systems. The framework, specifically focused on bipedal robotic technologies, will include methods and tools to measure robotics performance on a rigorous, quantitative and replicable way. The successful achievement of this goal will put Europe at the forefront of the evaluation of robotic systems, facilitating the process of bringing innovative robotic technologies forward to market.

The EUROBENCH project will offer funding opportunities to external entities, a.k.a. Third Parties, to participate in the development and validation of the different components (both Hardware and Software) of this unified framework. To this aim of this workshop is to involve the Robotic Community in this process, to identify strategic priorities and start contact with interested Third Parties and stakeholders.

This workshop is supported by the web-based community on “benchmarking bipedal locomotion” (


The workshop will be organized in sub-groups to focus discussions on the following aspects:
– Users’ needs: what do we really expect (want) from benchmarking?
– The System Abilities: are we talking the same language? (how to make System Ability Levels really quantifiable)
– A unified framework: similarities and differences across domains
– To compete or not to compete? That is the question… (the role of competition as benchmarking tool)

Groups will report the entire audience with the main conclusions, which will be then collected by the organizers and taken as a starting point towards a European roadmap for benchmarking.

Agenda: The workshop will be held on Thursday 15, 10:45-12:15.

10:45 – 11:15 Introduction and presentation from EUROBENCH Coordinator (Diego Torricelli) and PO (Anne Bajart)
11:15 – 11:45 Discussion in sub-groups (brainstorming & synthesis)
11:45 – 12:10 Reporting by group moderators and general discussion with the audience
12:10 – 12:15 Wrap up

Outcomes from group discussions will be collected by the organizers and taken as a starting point towards a European roadmap for benchmarking. All this information will be made available publicly on the “benchmarking bipedal locomotion” website ( and the EUROBENCH website (

Confirmed participants:

Anne Bajart, Project Officer, Head of Sector, European Commission
Pedro Lima, European Robotics League (ERL)
Fabio Bonsignorio, IEEE RAS TC-Pebras
Matteo Matteucci, Politecnico di Milano
Carlos Hernandez Corbato, H2020 ROSIN Project
Matthew Studley, European Robotics League (ERL) & H2020 SciRoc Project
Nikos Tsagarakis, Italian Institute of Technology (IIT)
Carlos Perez Martínez, Altran, Spain
Amit Kumar Pandey, Softbank Robotics, Paris, France
Jose Gonzalez, Ottobock, Germany
Freygarður Þorsteinsson, OSSUR, Iceland
Jan Veneman, Tecnalia, Spain
Jinoh Lee, Italian Institute of Technology (IIT)
Jaap Buurke, Roessingh Research and Development, The Netherlands
Tamim Asfour, Karlsruhe Institute of Technology (KIT), Germany
Nicola Vitiello, Scuola Superiore S.Anna (SSSUP), Italy
Francesco Ferro, PAL Robotics, Spain


About the author:

Neural Rehabilitation Group , Cajal Institute, Spanish National Research Council (CSIC), Spain

Workshop @ ERF2018 “A unified scientific framework for robotics benchmarking: not a dream”