Benchmarking: the next challenge in Humanoid Robotics Research
Organizers: D. Torricelli, K. Mombaur, N. Tsagarakis
When: November 15th, 2017. Full day, starting at 8:30 AM.
Where: Birmingham Repertory Theatre, Suite 3
Achieving stable, versatile and efficient walking is one of the current challenges of humanoid research. Biologically inspiration is gaining relevance in the design of bipedal machines, due to the outstanding abilities of humans in coping with out-of-the-lab conditions, in presence of variable and unexpected perturbations. Nevertheless, the process of transferring human abilities to machines is a complex process, which involves questions from scientific and engineering perspectives, such as: “Which are the human properties that should be transferred to the humanoid? How do limitations of the current technology (e.g. actuators, number of DoF, etc..) influence this process? Can the cause-effect relationship between a given solution and the resulting performance be quantified?” Benchmarking may constitute a powerful tool for researchers and developers to answer these questions.
In the framework of five European projects (H2R, Koroibot, Walkman, Biomot and Balance) we recently developed a benchmarking framework for the evaluation of bipedal functions . In past workshops, we shared this preliminary scheme with the Humanoids and related communities [3-8]. In this workshop, we will gather worldwide experts in the field of humanoid locomotion to discuss emerging methods, protocols and devices for the assessment and comparison of walking and postural robotic performance.
This WS is linked with the Frontiers Research Topic: “Assessing bipedal locomotion: towards replicable benchmarks for robotic and robot-assisted locomotion”
Why benchmarking? Introduction and new funding opportunities
Biosketch: Diego Torricelli received the Mechanical Engineering MS and the Biomedical Engineering PhD degrees from the University of Roma TRE in 2004 and 2009 respectively. Since 2010 he is with the Neural Rehabilitation Group of the Spanish National Research Council (CSIC), in Madrid, Spain. His research line spans the understanding of human motor control principles related to coordination, the development of quantitative measures of neuromotor impairments, and the development of bio-inspired machines for neural rehabilitation. He has been scientific manager of the FP7 project H2R (www.h2rproject.eu) and is currently the coordinator of the international community “Benchmarking Bipedal Locomotion” (www.benchmarkinglocomotion.org).
University of Heidelberg
Benchmarking bipedal locomotion for humanoid and wearable robots
Biosketch: Katja Mombaur is a full professor at the Institute of Computer Engineering (ZITI) of Heidelberg University and head of the Optimization in Robotics & Biomechanics (ORB) group as well as the Robotics Lab. She holds a diploma degree in Aerospace Engineering from the University of Stuttgart and a Ph.D. degree in Mathematics from Heidelberg University. She was a postdoctoral researcher in the Robotics Lab at Seoul National University, South Korea. She also spent two years as a visiting researcher in the Robotics department of LAAS-CNRS in Toulouse. Katja Mombaur is coordinator of the newly founded Heidelberg Center for Motion Research. She also is PI in the European H2020 project SPEXOR and the Graduate School HGS MathComp as well as in several national projects. Until recently, she has coordinated the EU FP7 project KoroiBot and was PI in the EU projects MOBOT and ECHORD–GOP. She is founding chair of the IEEE RAS technical committee Model-based optimization for robotics.
Her research focuses on understanding human movement and using this knowledge to improve motions of humanoid robots and in the interactions of humans with exoskeletons, prostheses and external physical devices. Her particular interest is on dynamic motions such as walking, running, and other kinds of motions in sports, as well as motions of daily life. She and her team use and develop dynamic models and optimization methods for motion studies, based on the assumption that human movement is optimal. In this context they are also interested in inverse optimal control which can determine what a human is optimizing in a given situation.
Italian institute of Technology (IIT)
Benchmarking humanoid resistance to push disturbance
Biosketch: He received his DEng degree in Electrical and Computer Science Engineering in 1995 from the Polytechnic School of Aristotle University, Greece, an M.Sc degree in Control Engineering in 1997 and in 2000 a PhD in Robotics from the University of Salford, UK. Before becoming a Senior Researcher at IIT with overall responsibility for Humanoid design & Human Centred Mechatronics development he was a research Fellow and then Senior Research Fellow in the Centre for Robotics and Automation at the University of Salford where he worked on haptic systems, wearable exoskeletons, rehabilitation robots and humanoids robots. He is an author or co-author of over 250 papers in research journals and at international conferences and holds 12 patents, and recipiente of several scientific awards. He has been in the Program Committee of over 60 international conferences including IEEE ICRA, IROS, RSS, HUMANOIDS BIOROB and ICAR. Nikos Tsagarakis is Technical Editor of IEEE/ASME Transactions in Mechatronics and on the Editorial Board of the IEEE Robotics and Automation Letters. Since 2013 he is also serving as a Visiting Professor at the Centre for Robotics Research (CORE), Department of Informatics, King’s College University, London, UK.
University of the West of England
Benchmarking through competition: The European Robotics League and the SciRoc Project.
Abstract: What is the ERL and where did it come from? The approach to competitions which the ERL has used, and how benchmarking is the added ingredient. Lessons learned from past iterations. Where the ERL is going; the SciRoc project and Smart Cities.
Biosketch: Matthew Studley is member of the Bristol Robotics Laboratory, and is leading an international consortium in the newly-agreed H2020 SciRoc project.
The RoboCup Humanoid League – Past Achievements, Current Challenges, Future Requirements
Abstract: The talk provides a detailed look at the RoboCup robotics soccer Humanoid League, a competition in building humanoid robots to play soccer. It looks back to 20 years of developing the functionalities for playing soccer and provides a flashlight on the current status of the robots participating in the competition. It lists current challenges and sketches the development roadmap to reach the goal of a soccer game between humans and humanoid robots in the year 2050. The talk covers technical as well as organisational aspects and how the activities serve as driver and benchmark for robotics activities.
Biosketch: Holds a degree in Electrical Engineering from Technical University Braunschweig. His PhD thesis was on ‘Event-Flow Architectures for Embedded Systems’. He has been working in the fields of industrial automation, circuit design and systems engineering. Since 2002 he is full professor at Ostfalia University. His field of research and teaching is robotics, human-robotinteraction and virtual reality. His student teams have been participating in RoboCup since 2007. Among other activities, he currently is member of the executive committee of the RoboCup federation and speaker of the German RoboCup regional committee.
Karlsruhe Institute of Technology (KIT)
Open Source Datasets and Tools for Human, Humanoid and Exoskeletons Motion Performance Evaluation
Abstract: A fundamental requirement for benchmarking is the availability of suitable datasets, tools and frameworks, which allow comparison between methods and algorithms. In this talk, I will present several open source datasets and tools for performance evaluation in the context of human motion understanding as well as humanoids and exoskeleton motion generation. This include 1) the KIT whole-body human motion database with its tools for unifying representation of human subject(s) and objects as well as 2) the KIT Language-Motion Dataset for studying the mapping between human motion and natural language. In addition, I will present a new framework for comparative analysis of human motion segmentation algorithms.
Bioskech: Tamim Asfour is full Professor at the Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology (KIT) where he holds the chair of Humanoid Robotics Systems and is head of the High Performance Humanoid Technologies Lab (H2T). His current research interest is high performance 24/7 humanoid robotics. Specifically, his research include the engineering humanoid robot systems, grasping and dexterous manipulation, learning from human observation and sensorimotor experience as well as on the mechano-informatics of humanoids as the synergetic integration of mechatronics, informatics and artificial intelligence methods to create integrated complete humanoid robot systems. He is developer of the ARMAR humanoid robot family and is leader of the humanoid research group at KIT since 2003. www.humanoids.kit.edu
Benchmarking Human-Humanoid Social and Cognitive Interaction: some results and the grand challenges – An industrial perspective
Abstract: Never before in the history of robotics, social robots, including humanoid robots, have been so close to us, in our society. We are ‘evolving’, so as our society, lifestyle and the needs. AI has been with us for decades, and now embodied in humanoid robots, penetrating more in our day-to-day life. All these are converging towards creating a smarter eco-system of living, where social humanoid robots will coexist with us in harmony, for a smarter, healthier, safer and happier life. Such robots are supposed to be socially intelligent and behave in socially expected and accepted manners. Therefore, benchmarking and evaluating such high level capabilities of these humanoid robots need much more attention and investigation. The open questions from scientific and R&D perspective are manifold. The talk will reinforce this need and from R&D and industrial perspective of social robots and provide some pointers of initial efforts in this direction. The talk will conclude with some open challenges ahead, including social and ethical issues and emphasize on the greater need of a bigger and multi-disciplinary effort.
Biosketch: Dr. Amit Kumar Pandey is Head Principal Scientist (Chief Scientist) at SoftBank Robotics Europe (formerly Aldebaran Robotics), Paris, France, also serving as the scientific coordinator (R&D) of its various collaborative projects. Earlier for 6 years he worked as researcher in Robotics and AI at LAAS-CNRS (French National Center for Scientific Research), Toulouse, France. His Ph.D. thesis in Robotics (title: Towards Socially Intelligent Robots in Human Centered Environment), is the second prize winner (tie) of the prestigious Georges Giralt Award for the best Ph.D. Thesis in Robotics in Europe, awarded by euRobotics (the European Union Robotics Community). His current research interest includes Socially Intelligent Robots, Human Robot Interaction (HRI), Robot’s Cognitive Architecture and Lifelong Learning. On these aspects, he has been actively contributing as principal investigator, researcher, and industrial scientific coordinator in various national and European Union (EU) projects, as well as involved in their design and proposal. He has published more than 40 scientific research papers in international journal, conferences and workshops. Among other responsibilities, he is the founding coordinator of Socially Intelligent Robots and Societal Applications (SIRo-SA) Topic Group (TG) of euRobotics, and an active contributor in the Multi-Annual Roadmap (MAR) and Strategic Research Agenda (SRA) of euRobotics, which aim to shape the future of robotics in Europe in collaboration with European Commission (EC) through PPP SPARC (one of the largest civilian-funded robotics innovation programme in the world). He is also the recipient of Pravashi Bihari Samman Puruskar 2014 (Non Residential Bihari Honour Award), for Science, Technology and Education, one of the highest level civilian honors, awarded by the state of Bihar, India.
Human-likeness from human experiments and human-robot comparisons
Abstract: Human-like posture control can be seen as a basis for robust and versatile sensorimotor behavior in complex ‘real world’ settings. Aiming at human-likeness in the postural control of humanoid robots may currently proceed from an existing taxonomy of motor skills classification. Future extensions of it may help to diagnostically target specific sensor or control functions. Such an approach will profit from establishing sets of human-likeness definitions at the level of sensors and control functions. This contribution addressed the following points:
– Brief overview: Identified basic features of the human posture control system.
– Basic tests of the human-derived posture control system:
Testbed, Stimuli, Analyses, Human-Robot comparisons, Robustness with Sensor failure, Versatility (combinations of stimuli, external and self-produced).
– Testing a human-derived posture control system on different robots.
– Comparing different human-inspired posture control systems in a robot.
Biosketch: Study of medicine at Münster University, Germany, post-doc in neurophysiology at Pisa University, Italy. Neurology training and positions at Ulm University, Germany. Professorship for Experimental Neurology and Clinical Neurophysiology at the Neurological University Clinic, Freiburg, Germany, since 1985. Scientific focus is on how humans use sensory information to perceive and control their spatially oriented behavior and how to formalize the experimental results into dynamic models. This led to a model of human sensorimotor control, which then was embodied into humanoid robots for human-robot comparisons.
Benchmarking motion generation capabilities on the HRP-2 humanoid
Abstract: During this talk we will describe a set of experiments realized with the humanoid robot HRP-2 to benchmark its motion generation capabilities.The work was performed in collaboration with the french national laboratory on metrology (LNE). We tested the range of temperature the robot was able to handle. The commercial walking pattern generator provided by Kawada Robotics as well as one of our recent pattern generator were tested under various slopes and pushes. We also tested the stabilizer commercially available on various horizontal sinusoidal motions. We will show that the controller was able to maintain the robot stable despite reaching strong perturbations.
Biosketch: Olivier Stasse is a senior CNRS researcher at LAAS CNRS, Toulouse, in the Gepetto group. His main research topic is motion generation on humanoid robots. In 2000, he received a Ph.D. in Intelligent Systems from the University of Paris 6 under the supervision, and the French Habilitation to Supervise Research (HDR) in Robotics (2013) from the University of Toulouse III. From 2000 to 2003, he was assistant professor at the Univ. of Paris XIII. From 2003 to 2011, he was with the Joint French-Japanese Robotics Laboratory (JRL) between the CNRS and the AIST in Tsukuba, Japan.
Italian institute of Technology (IIT) & DeepMind
The benchmarking of physical interaction and AnDy’s roadmap towards proficient human-robot collaboration
Abstract: In this talk Francesco Nori will discuss his approach in modelling physical human-robot interaction. At the basis of this approach the idea is to model interaction as the coupling between two mechanical systems: the agent and the partner. While the agent has direct access to its internal variables (e.g. proprioception and applied forces), it can only indirectly infer the partner’s internal variables. Still these variables are fundamental to generate proficient collaborative motions during physical interaction tasks. In this sense modern robot have a blind spots since they are limited in observing human whole-body dynamics. The recently funded H2020-ICT project AnDy will resolve this blind spot by developing a sensor suit able to observe human whole body dynamics in real- time, by learning ergonomic and anticipatory models from the big data sets this suit generates, and by incorporating these models in on-line control to make collaboration more efficient.
Biosketch: Francesco Nori was born in Padova in 1976. He received his D.Eng. degree (highest honors) from the University of Padova (Italy) in 2002. During the year 2002 he was a member of the UCLA Vision Lab as a visiting student under the supervision of Prof. Stefano Soatto, University of California Los Angeles. During this collaboration period he started a research activity in the field of computational vision and human motion tracking. In 2003 Francesco Nori started his Ph.D. under the supervision of Prof. Ruggero Frezza at the University of Padova, Italy. During this period the main topic of his research activity was modular control with special attention on biologically inspired control structures. Francesco Nori received his Ph.D. in Control and Dynamical Systems from the University of Padova (Italy) in 2005. In the year 2006 he moved to the University of Genova and started his PostDoc at the laboratory for integrated advanced robotics (LiraLab), beginning a fruitful collaboration with Prof. Giorgio Metta and Prof. Giulio Sandini. In 2007 Francesco Nori has moved to the Italian Institute of technology where he is currently hired as a Tenure Track Researcher. His research interests are currently focused on whole-body motion control exploiting multiple (possibly compliant) contacts. With Giorgio Metta and Lorenzo Natale he is one of the key researchers involved in the iCub development, with specific focus on control and whole-body force regulation. Francesco is currently involved in two FP7-EU projects: CoDyCo as coordinator and Koroibot as principal investigator. In the past he has been investigator in ITALK, VIACTORS and Robotcub.
|8:30 – 9:00||Diego Torricelli||Why benchmarking? Introduction and new funding opportunities|
|9:00 – 9:30||Katja Mombaur||Benchmarking bipedal locomotion for humanoid and wearable robots|
|9:30 – 10:00||Nikos Tsagarakis||Benchmarking humanoid resistance to push disturbance|
|10:00 – 10:30||Coffee Break & Poster Session|
|10:30 – 11:00||Matthew Studley||Benchmarking through competition : The European Robotics League and the SciRoc Project|
|11:00 – 11:30||Reinhard Gernt||The RoboCup Humanoid League – Past Achievements, Current Challenges, Future Requirement|
|11:30 – 12:00||Tamim Asfour||Open Source Datasets and Tools for Human, Humanoid and Exoskeletons Motion Performance Evaluation|
|12:30 – 13:30||Lunch Break|
|13:30 – 14:00||Amit Kumar Pandey||Benchmarking Human-Humanoid Social and Cognitive Interaction: some results and the grand challenges – An industrial perspective|
|14:00 – 14:30||Thomas Mergner||Human-likeness from human experiments and human-robot comparisons|
|14:30 – 15:00||Olivier Stasse||Benchmarking motion generation capabilities on the HRP-2 humanoid|
|15:00 – 15:30||Coffee Break & Poster Session|
|15:30 – 16:00||Francesco Nori||The benchmarking of physical interaction and AnDy’s roadmap towards proficient human-robot collaboration|
|16:00 – 17:00||All attendees||Discussion in groups|
|17:00 – 17:30||Organizers||Wrap-up & Conclusions|
The following posters will be presented during the two Coffee breaks, in the REP Mezzanine area, at 10:00-10:30 and 15:00-15:30:
– M. Brandao, K. Hashimoto, A. Takanishi, LegOpt: Legged-robot posture and trajectory Optimization Benchmark
– D. M. Lofaro, P. P. Weis, E. Wiese, Experiments confirming gaze triggered effects in embodied social robots
– V. Lippi, T. Mergner, Experimental evaluation of a Bio-inspired Modular System for Posture Control
– D. M. Lofaro et al. Experimental Setup and Approach for Co-Robot Locomotion when Mechanically Coupled to a Human Partner
Spanish National Research Council (CSIC), Spain
University of Heidelberg, Germany
Italian Institute of Technology, Italy
 D. Torricelli, J. Gonzalez-Vargas, K. Mombaur, N. Tsagarakis, J. C. Moreno, and J. L. Pons, “Benchmarking Bipedal Locomotion. A Unified Scheme for Humanoids, Wearable Robots, and Humans,” IEEE Robot. Autom. Mag., vol. 22, no. 3, pp. 103–115, 2015.
 Webpage on benchmarking bipedal locomotion www.benchmarkinglocomotion.org
 2013 IEEE-RAS International Conference on Humanoid Robots, Workshop on “Benchmarking of human-like robotic locomotion”
 2014 International workshop on wearable robots, Session on “Benchmarking, Regulatory and funding aspects of WRs”
 2014 IEEE-RAS International Conference on Humanoid Robots, Workshop on “Benchmarking of bipedal locomotion”
 European Robotics Forum 2015, Session on “Replicable robotics research and benchmarking”
 International conference of Rehabilitation Robotics (ICORR 2015), Workshop “Benchmarking lower limb wearable robots: towards practical and evidence-based solutions”
 IEEE-RAS Humanoids 2015, Workshop “Benchmarking bipedal functions of humanoids robots: towards a unified framework”