Workshop @ ISEK:
How good is my robot? The increasing importance of benchmarking in wearable robotics research
Organizers: Diego Torricelli, Jan Veneman, Jose Luis Pons

When: June 29, 2018 (14:00 – 17:00)
Where: ISEK 2018, Dublin

Wearable robots for gait assistance and rehabilitation, such as exoskeletons and prostheses, are becoming increasingly available in the market. These robotic systems are currently tested according to self-defined procedures and metrics, which can hardly be transferred across different laboratories and platforms. This makes it very difficult, if not impossible, to compare similar systems to each other. Benchmarking is a powerful tool that can help researchers and developers to overcome this problem. Recent efforts, mostly based on competitions (e.g. Cybathlon), have raised the attention for this topic. However, a consolidated benchmarking methodology for Robotics is still not available. The definition of standardized metrics and protocols to evaluate the effectiveness of wearable robotics technology is a crucial yet very complex issue, which involves multiple perspectives at technical, clinical and usability aspects.
The recently funded H2020 project EUROBENCH is working on the creation of a unified methodology for the assessment of wearable robotics performance, and will soon offer third party funding opportunities for research groups working on the following topics:

– Methods and protocols to test human and human-like locomotion performance in unstructured environments
– Equipment and test beds to allow replicability of benchmarking protocols across labs
– Simulation and modelling approaches to measure human-robot interactions
– Creation and use of databases to quantify human and robot performance in complex environments
– Testing safety of wearable robots
– Testing physiological and subjective impact of robotic use

This workshop wants to explore and discuss the main challenges behind robotic benchmarking as a follow-up of previous discussions workshops at WeRob2014, ICORR 2015, and WeRob2016, ICORR2017 conferences. The workshop is promoted by the H2020 Project EUROBENCH  (starting on January 2018), and supported by the benchmarking bipedal locomotion hub  and the COST Action CA16116 “Wearable Robots for Augmentation, Assistance or Substitution of Human Motor Functions”

Confirmed speakers:

Diego Torricelli – Spanish National Research Council (CSIC), Spain – “The EUROBENCH project: new funding opportunities in wearable robots benchmarking”

Short biosketch: Diego Torricelli received the Mechanical Engineering MS and the Biomedical Engineering PhD degrees from the University of Roma TRE in 2004 and 2009 respectively. Since 2010 he is with the Neural Rehabilitation Group of the Spanish National Research Council (CSIC), in Madrid, Spain. His research line spans the understanding of human motor control principles related to coordination, the development of quantitative measures of neuromotor impairments, and the development of bio-inspired machines for neural rehabilitation. He has been scientific manager of the FP7 project H2R ( and is currently the coordinator of the international community “Benchmarking Bipedal Locomotion” (

Jan Veneman – Hocoma AG, Switzerland – “Benchmarking related to safety of wearable and rehabilitation robots”

Abstract: This presentation will elaborate on the new developments in safety standardization under ISO TC299 robotics, on so far this applies to robots that are connected to limbs in order to exchange mechanical power with the human body, such as exoskeletons and rehabilitation robots. The main focus will be on approaches to functional testing, which is a type of benchmarking of performance that is related to the safety of the device.

Short biosketch: Jan Veneman works currently works for Hocoma AG, Switzerland, as Technical Project Lead for the development of Lower Extremity Rehablitation robots. Until recently, he was affiliated to Tecnalia Research and Innovation, where he was project manager and senior researcher on rehabilitation robots and wearable robots. Since several years he has been involved with regulatory issues, safety testing and benchmarking. Since 2013 he was coordinating the EC-FP7 consortium research Project BALANCE, on the development of a robotic exoskeleton with self-balancing control. He was elected Action Chair of COST Action 16116 on Wearable Robots and is active since 2012 as international expert in several ISO and IEC standardization working groups related to medical robotics, especially exoskeletons.

Nicola Vitiello – Superiore Sant’Anna (SSSA), Italy – “Ergonomics in lower-limb exoskeletons: the case study of the Active Pelvis Orthosis”

Abstract: Ageing population affects society welfare sustainability. The ageing of the population is one of the most critical challenges current industrialized societies will have to face in the next years, and threatens the sustainability of our social welfare. Among many diseases, gait disorders are common and often devastating companions of ageing, leading to reductions in quality of life and increased mortality. This presentation will introduce the research activities carried out at the BioRobotics Institute of Scuola Superiore Sant’Anna in the last years about the development of an Active Pelvis Orthosis for gait support.

Short biosketch: Nicola Vitiello is Associate Professor with The BioRobotics Institute (Scuola Superiore Sant’Anna, SSSA, Pisa, Italy) where he leads the Wearable Robotics Laboratory. He is co-author of more than 60 ISI/Scopus papers and co-inventor of more than 15 patent applications. He served as the Scientific Secretary of the EU FP7 CA-RoboCom project, and he was the scientific project coordinator of the EU FP7 CYBERLEGs project. Currently he is the scientific project coordinator of the H2020-ICT-CYBERLEGs Plus Plus project, the national project MOTU funded by INAIL, and is partner of the H2020-ICT-AIDE and H2020-FoF-HUMAN projects. On 2015, he co-founded IUVO Srl, a spin-off company of SSSA.

Jaap Buurke – Roessingh Research and Development, The Netherlands – “Benchmarking the clinical impact of rehabilitation robots”

Abstract: The objective of the European project EUROBENCH is to develop a framework for benchmarking of humanoid and wearable robotics ( To identify key system abilities for wearable robotics such as exoskeletons and prostheses, reference data on locomotion in healthy and impaiored subjects are gathered. The experiments shall include different patient groups (e.g. stroke patients, amputees) and various walking conditions that are relevant and achievable for the patients.
Movement impairments frequently lead to asymmetric gait, often including unequal step lengths. Wearable robotics such as exoskeletons and prosthetics aim to support normal gait. However, the asymmetry in pathological gait can be a functional compensation to maintain stability. Using gait trainers to train symmetric gait might have a negative effect on gait stability. Therefore, there is a need for adaptation of the wearable robot to the patient’s specific limitations and needs.

Short biosketch: Professor Jaap Buurke worked as a physical therapist for more than 25 years. He received his PhD in 2005 from the University of Twente for his work on recovery of gait after stroke. Currently he is the track coordinator of the “Rehabilitation Technology research cluster” at Roessingh Research and Development. He holds the chair of “Technology supported human movement analysis” at the University of Twente and is adjunct professor at North Western University Chicago (USA). His research focuses on human movement analysis with specific expertise in kinesiology (neuromuscular control and biomechanics) after stroke. He is actively involved in a diversity of (inter)national projects focusing on motor control, movement analysis, rehabilitation robotics and active assistive devices.

Freygardur Thorsteinsson – OSSUR, Iceland – “Benchmarking of prosthetic devices: a market-driven perspective”

Abstract: Currently, companies producing medical devices are required by regulations to show the functionality of their devices. At the same time there is lack of methods and standards to do this in a systematic manner.  The current talk discusses how this is being addressed in a company producing prosthetic devices and what are the pros and cons of this approach.

Short biosketch: Mr. Freygardur Thorsteinsson is a project manager in the R&D department of Össur hf. He got his B.Sc. and B.Sc.Hons. degree in chemistry from the University of Iceland and M.Sc. degree in chemical engineering at the Technical University of Denmark in Copenhagen 1989. He worked as a material scientist at the Technological Institute of Iceland for 10 years before joining Össur in summer 1999. His work has involved leading Ossur’s part in several international research projects among other duties. He has been active in work on testing and standards and he served 8 years on the board of Eurolab Iceland, thereof 2 years as chairman. He was representing Ossur as a member of ISO/TC 168/WG 3 working group on testing standards in prosthetics from 2009-2013. He is currently (from 2012) a board member for the National Research Fund in Iceland.

Carlos Rodriguez-Guerrero – Vrije Universiteit Brussels, Belgium – “Benchmarking lower limbs wearable robots with compliant actuation”

Abstract: In recent years, compliant actuators have been widely adopted by the wearable robots community. Compliant actuators can adequately address the issues of safety, robustness, wearability and overall system efficiency. As much as this technology have matured over the last years, there are no standardized benchmarking tests yet that can be applied to this emerging technology. In this presentation we’re going to discuss some of the aspects that are more susceptible to benchmark in compliant actuators and present some of the proposed solutions and methodologies.

Short biosketch: Carlos Rodriguez-Guerrero obtained his PhD with his work entitled “Psychophysiological Feedback Control in Physical Human-Robot Interaction” in 2012. He is now a senior researcher at Vrije Universiteit Brussel working at the Robotics and Multibody Mechanics group (R&MM) developing control systems for assistive exoskeletons with novel actuation systems. He’s been appointed as a guest professor at the VUB, in the fields of Biomedical robotics control and Biomechanics.


About the author:

Neural Rehabilitation Group , Cajal Institute, Spanish National Research Council (CSIC), Spain

Workshop @ ISEK “How good is my robot? The increasing importance of benchmarking in wearable robotics research”