Watch, Predict, Act: Robot Learning Meets Web Videos

with Homanga Bharadhwaj, Ph.D. Student
The Robotics Institute, School of Computer Science
Carnegie Mellon University

Tuesday, January 28th, 10:00-11:00 am
310 Kelly Hall

 

Developing robots that can help us in our daily activities has been a guiding aspiration in AI for decades. Yet, general purpose robots that work out of the box, can perform tasks without manual interventions, and are safe to interact with remains an elusive goal. Since collecting robot interaction data in diverse scenarios is difficult due to operational constraints, a key challenge in robotics is being able to perform new tasks in novel scenes without requiring robot data collection in every scenario. Going beyond the standard of end-to-end imitation learning, in this talk I will describe an alternate paradigm towards this goal: combining robot-specific data with predictive planning from diverse web videos such as YouTube clips of humans doing daily chores. By learning to predict motion and contextual cues from naturally diverse web data for robotic policy learning, I will demonstrate how this recipe enables training common goal/language-conditioned policies capable of multiple in-the wild manipulation tasks in unseen offices and kitchens, across different robotic embodiments. I will conclude by laying out my vision for the coming years, encompassing the entire spectrum of learning structure from diverse web datasets, developing robot learning algorithms that can scalably use such structured predictions for broad generalization, and deploying robotic systems in-the-wild by being adaptable to deployment-time constraints and compliant with human preferences.

Homanga Bharadhwaj is a final-year PhD student in Carnegie Mellon University. His research goal is to develop embodied AI systems capable of helping us in the humdrum of everyday activities within messy rooms, offices, and kitchens, in a reliable, compliant, and scalable manner without requiring significant robotspecific data collection and task-specific heuristics. He is a Meta AI Mentorship (AIM) Fellow and was a visiting researcher at Meta for two years during his PhD. Before coming to CMU he obtained an MSc from the University of Toronto and a BTech from IIT Kanpur in India. His research has received a Best Paper in Robot Manipulation Finalist at ICRA '24, the Best Conference Paper Award at ICRA '24, the Outstanding Presentation award at NeurIPS '23 Robot Learning Workshop, and has been covered by several media outlets like TechCrunch, IEEE Spectrum, and VentureBeat among others.

Host: Suyi Li