Benchmarking bipedal functions of humanoids robots: towards a unified framework
Venue and date: Seoul, Korea, November 3, 2015
- Diego Torricelli (CSIC, Spain), firstname.lastname@example.org
- Katja Mombaur (University of Heidelberg, Germany), email@example.com
- Nikos Tsagarakis (IIT, Italy), firstname.lastname@example.org
- Jose Gonzalez (CSIC, Spain), email@example.com
Humanoids robots are entering an exciting era. An increasing number of solutions are moving out of the lab, approaching real scenarios and home-based assistive applications. In this context, the quantitative assessment of the technology is crucial for its correct introduction in the market. Great efforts are now devoted to establish systematic frameworks to evaluate the performance of robots under all their facets, i.e. locomotion, manipulation, human-robot interaction, disaster-responding, etc.
In the framework of the three European projects H2R, Koroibot and Walkman, we developed a benchmarking scheme for the evaluation of bipedal functions of humanoid robots . The scheme has been designed for its application across different research laboratories and in several perturbed conditions. This scheme can be considered as a complementary tool to the current competitions-based approach (e.g. DRC).
With this workshop, we want to share this preliminary scheme with the Humanoids Community, and start working towards the definition of a common benchmarking perspective. We believe that reaching an international consensus will be extremely beneficial to boost the process of finding reliable methods to test and compare different systems and identifying robust metrics to measure the Technology Readiness Level (TRL) of the solutions.
This workshop comes after a series of actions directed to involve the international community in the definition of real needs and critical issues related to benchmarking of lower limb functions. These actions included: 1) a web-based survey , which received more than 160 responses from robotics and clinical researchers, 2) several sessions and workshops in the WR and humanoids communities [3-6], 3) an international network (, ) specifically addressed to this topic, to support the clustering of people and ideas. All these efforts resulted in the publication of a unified benchmarking scheme , With this workshop we would like to make a step forward, and bring together different researchers already active in the development evaluation methods and protocols, with the goal of identifying the critical issues in translating these approaches into consented benchmarks.
– Diego Torricelli/Jose Gonzalez (CSIC, Spain), Title: “A unified framework for the benchmarking of humanoid bipedal functions”
– Katja Mombaur (University of Heidelberg, Germany), Title: “Benchmarking the similarities between humans and humanoids. The Koroibot Project”
– Nikos Tsagarakis (IIT, Italy), Title: “Benchmarking humanoid stability during push perturbations. The Walk-Man Project”
– Oliver Stasse (LAAS-CNRS, France), Title: “Benchmarking the HRP-2 humanoid robot in different walking situations”
– Russ Tedrake (MIT, USA), Participation to be confirmed
– Vittorio Lippi (Freiburg University, Germany), Title: “Benchmarking human-like postural control of humanoids during perturbed conditions”
– Joo H. Kim (New York University, USA), Title: “”Humanoid vs. Human: Energy Expenditure and Balance Stability of Walking””
– Francesco Nori (IIT, Italy), Title: “wholeBodyInterface: a software abstraction layer for benchmarking whole-body motion control”
 D. Torricelli, J. Gonzalez, J. Veneman, K. Mombaur, N. Tsagarakis, A.J. del-Ama, Á. Gil-Agudo, J.C. Moreno, and J.L. Pons, Benchmarking bipedal locomotion in humanoids, wearable robots and humans: a unified scheme, IEEE Robotics and Automation Magazine (in press).
 Benchmarking bipedal locomotion – A survey. Still accessible at: http://goo.gl/forms/FL9Pd1xXgb
 2014 International workshop on wearable robots (www.werob2014.org), Session on “Benchmarking, Regulatory and funding aspects of WRs”
 European Robotics Forum 2015 (www.erf2015.eu), Session on “Replicable robotics research and benchmarking”
 2013 IEEE-RAS International Conference on Humanoid Robots, Workshop on “Benchmarking of human-like robotic locomotion” (http://www.h2rproject.eu/humanoids2013)
 2014 IEEE-RAS International Conference on Humanoid Robots, Workshop on “Benchmarking of bipedal locomotion” (http://orb.iwr.uni-heidelberg.de/koroibot/?page_id=492)
 Mailing list on “benchmarking bipedal locomotion” (https://listas.csic.es/wws/info/benchmarking_list)
 Webpage on benchmarking bipedal locomotion www.benchmarkinglocomotion.org