Talk: “Real” robots prefer priors and structure — RSS2022 workshop on Scaling Robot Learning

You can watch Georgia’s talk at the RSS2022 workshop on Scaling Robot Learning online!

And the very interesting panel discussion with George Konidaris, Aleksandra Faust, Oliver Kroemer and Georgia at:

Keynote speaker at IROS 2022

Georgia will deliver a keynote speech at IROS 2022 in Kyoto, Japan, on the 26th of October at 13:30 JST!

Here are the details of the talk.

Title: Shaping Robotic Assistance through Structured Robot Learning

Abstract: Future intelligent robotic assistants are expected to perform various tasks in unstructured and human-inhabited environments. These robots should support humans in everyday activities as personal assistants or collaborate with them in work environments like hospitals and warehouses. In this talk, I will briefly describe my research works for robotic assistants to help and support humans in need, developing specific human-robot interaction behaviors combining classical robotics and machine learning approaches. I will then explain how mobile manipulation robots are currently the most promising solution among embodied AI systems, thanks to their body structure and sensorial equipment for learning to execute a series of assistive tasks. On top of this, I will point out some key challenges that hinder autonomous mobile manipulation for intelligent assistance and discuss how structured robot learning can pave the way toward generalizable robot behaviors. Structured robot learning refers to all learning methods at the intersection of classical robotics and machine learning that aim to leverage structure in data and algorithms to scale robot behaviors to complex tasks. Finally, this talk will give insights into how my team and I leverage structured representations, priors, and task descriptions together with learning and planning in some challenging (mobile) manipulation tasks in our path for creating general-purpose intelligent robotic assistants.

6 Papers accepted @IROS2022

We got 6 papers accepted in #IROS22 at Kyoto! Well done to our students of the Computer Science, TU Darmstadt and our collaborators for this success!

1. “Robot Learning of Mobile Manipulation with Reachability Behavior Priors” by Snehal Jauhri Jan Peters and me accepted to RAL+IROS. Check: https://lnkd.in/esQpVmuV

2. “Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery” by Niklas Wilhelm Funk S. Menzenbach, Jan Peters and me. Check: https://lnkd.in/eNue6X2P

3. “Active Exploration for Robotic Manipulation” by T. Schneider, Boris Belousov, Diego Romeres Devesh Jha Jan Peters and me. Check: https://lnkd.in/ebiH5Z72

4. “Regularized Deep Signed Distance Fields for Reactive Motion Generation” by P. Liu, K. Zhang, D. Tateo, Snehal Jauhri Jan Peters and me. Check: https://lnkd.in/eEDqGGkW

5. “Learning Implicit Priors for Motion Optimization” by Alexander (Sasha) Lambert, An Thai LeJulen Urain de JesusByron BootsJan Peters and me. Check: https://lnkd.in/ebMqV6jD

6. “Monte-Carlo Robot Path Planning” by T. Dan Jan PetersJoni Pajarinen and me. Soon to be on arxiv…

Regularized Deep Signed Distance Fields for Reactive Motion Generation

Authors: Puze Liu, Kuo Zhang, Davide Tateo, Snehal Jauhri, Jan Peters and Georgia Chalvatzaki

Paper
Code
Video

Abstract

Autonomous robots should operate in real-world dynamic environments and collaborate with humans in tight spaces. A key component for allowing robots to leave structured lab and manufacturing settings is their ability to evaluate online and real-time collisions with the world around them. Distance-based constraints are fundamental for enabling robots to plan their actions and act safely, protecting both humans and their hardware. However, different applications require different distance resolutions, leading to various heuristic approaches for measuring distance fields w.r.t. obstacles, which are computationally expensive and hinder their application in dynamic obstacle avoidance use-cases. We propose Regularized Deep Signed Distance Fields (ReDSDF), a single neural implicit function that can compute smooth distance fields at any scale, with fine-grained resolution over high-dimensional manifolds and articulated bodies like humans, thanks to our effective data generation and a simple inductive bias during training. We demonstrate the effectiveness of our approach in representative simulated tasks for Whole-body control and safe Human-Robot Interaction (HRI) in shared workspaces. Finally, we provide proof of concept of a real-world application in a HRI handover task with a mobile manipulator robot.

Video demonstrations

ReDSDF for Reactive Motion Generation in different application domains

Robot Learning of Mobile Manipulation with Reachability Behavior Priors

Authors: Snehal Jauhri, Jan Peters and Georgia Chalvatzaki

Abstract

Mobile Manipulation (MM) systems are ideal candidates for taking up the role of a personal assistant in unstructured real-world environments. Among other challenges, MM requires effective coordination of the robot’s embodiments for executing tasks that require both mobility and manipulation. Reinforcement Learning (RL) holds the promise of endowing robots with adaptive behaviors, but most methods require prohibitively large amounts of data for learning a useful control policy. In this work, we study the integration of robotic reachability priors in RL methods for accelerating the learning of MM. Namely, we consider the problem of optimal base placement and the subsequent decision of whether to activate the arm for reaching a 6D target. We derive Boosted Hybrid RL (BHyRL), a novel actor-critic algorithm that benefits from modeling Q-functions as a sum of residual approximators. Every time a new task needs to be learned, we can transfer our learned residuals and learn the component of the Q-function that is task-specific, hence, maintaining the task structure from prior behaviors.

Paper
Code

Technical Talk, RAL + IROS 2022 (Best Paper Award for Mobile Manipulation)

Our Contributions

  • We propose to use a hybrid action-space Reinforcement Learning algorithm for effectively tackling the need for discrete and continuous action decisions in Mobile Manipulation
  • We learn a reachability behavioral prior for Mobile Manipulation that can speed up the learning process, and incentivize the agent to select kinematically reachable base poses when dealing with 6D reaching and fetching tasks
  • We propose a new algorithm: Boosted Hybrid RL (BHyRL) for transferring knowledge from behavior priors by modelling Q-functions as sums of residuals, while also regularizing the policy learning in a trust-region fashion

Boosted Hybrid RL

  • The concept of ‘boosting’ [1] is to combine many weak learners to create a single strong learner.
  • To learn challenging base placement tasks, we first learn simpler reachability tasks and use the learnt behavior as a prior for accelerating the learning of subsequent tasks.
  • To do this, the Q-function of every task is modelled as the sum of residuals learned on previous tasks [2],[3].
  • Thus, we can progressively learn more difficult tasks while retaining the information and structure provided by the prior Q values.
  • Additionally, we regularize the new task policy using a KL-divergence penalty with the previous policy.

Simulated tasks for 6D Reaching & Fetching

The agent learns progressively more challenging tasks and combines each of the learned behaviors:

6D_Reach_1m task

The agent needs to reach a 6D target in its vicinity (1-metre radius) by choosing an optimal base location and activating its arm for reaching

6D_Reach_5m task

The agent needs to navigate towards a 6D target that is up to 5 metres away. The 6D_Reach_1m behavior is used as a prior.

6D_Reach_3_obst task

Similar to the task above but now in the presence of 3 obstacles. The 6D_Reach_1m and 6D_Reach_5m behaviors are used as priors.

6D_Fetch_table/wall task

The agent needs to fetch an object placed on a table in the presence of a wall behind the table. The 6D_Reach_1m and 6D_Reach_5m behaviors are used as priors.

6D_Fetch_2_furniture task

The agent needs to fetch an object placed on a table in the presence of another furniture obstacle. The 6D_Reach_1m, 6D_Reach_5m and 6D_Reach_3_obst behaviors are used as priors.

6D_Fetch_multiobj task

The agent needs to fetch an object placed on a table without colliding with multiple other objects on the table. The 6D_Reach_1m and 6D_Reach_5m behaviors are used as priors.

Demonstration of zero-shot transfer of BHyRL policy

“Will this method work for my robot?”

Our training method can also work for other mobile manipulators such as the above ‘Fetch’ robot. Our code is available at: github.com/iROSA-lab/rlmmbp

People

Snehal Jauhri

Jan Peters

Georgia Chalvatzaki

{snehal, jan, georgia}@robot-learning.de

Acknowledgement

This research received funding from the German Research Foundation (DFG) Emmy Noether Programme (#448644653) and the RoboTrust project of the Centre Responsible Digitality Hessen, Germany.

References

  1. Yoav Freund, “Boosting a weak learning algorithm by majority”, Information and Computation, 121 (2):256–285, 1995.
  2. Samuele Tosatto, Matteo Pirotta, Carlo D’Eramo, and Marcello Restelli, “Boosted fitted q-iteration”, International Conference on Machine Learning, 2017.
  3. P. Klink, C. D’Eramo, J. Peters, and J. Pajarinen, “Boosted curriculum reinforcement learning,” in ICLR, 2022.

Making robots useful parts of our society

Professorship for Dr. Georgia Chalvatzaki

Since February, Georgia Chalvatzaki has been assistant professor for “Intelligent Robotic Systems for Assistance” at the Department of Computer Science. Chalvatzaki has been leading the iROSA research group since 2021 as part of the Emmy Noether Programme of the German Research Foundation. Previously, the 33-year-old researcher was a postdoctoral researcher in the Department of Intelligent Autonomous Systems in the Department of Computer Science. We asked Professor Chalvatzaki about her work:

Chalvatzaki _Georgia

Continue at: https://www.informatik.tu-darmstadt.de/fb20/aktuelles_fb20/fb20_news/news_fb20_details_251072.en.jsp