Neuro Symbolic Reasoning for Robust Robot Intelligence

with Simon Stepputtis,
Postdoctoral Research Fellow
The Robotics Institute, School of Computer Science
Carnegie Mellon University

Tuesday, February 4th, 10:00-11:00 am
310 Kelly Hall

Deep learning has driven remarkable progress in robotics, largely through the use of large foundational models. Despite their success, these models demand extensive and hard-to-acquire training data while lacking the consistent reasoning needed for intelligent robots to operate in human-centric environments. In this talk, I will discuss how neurosymbolic methods bridge these gaps by integrating the expressive power of neural networks with the reasoning capabilities of symbolic approaches to advance robotic intelligence. First, I will introduce NeSCA, which combines neural policies with knowledge graphs, capable of short-context human action anticipation by reasoning over affordances and object relationships. Next, I discuss how symbolic structures can be expanded in a few-shot manner to adapt to new tasks and how ShapeGrasp, a framework for zero-shot object manipulation, utilizes large language models as hypothesis generators. Finally, I introduce Neural-Policy Translation, a framework that generates symbolic controllers for tabletop manipulation from language and vision, enabling efficient pick-and-pour tasks. I conclude by outlining how these methods pave the way for intelligent robots in complex, real-world applications - from in-home assistance to industrial automation - while addressing key challenges in life-long learning, trustworthiness, and efficiency.

Simon Stepputtis is a postdoc at Carnegie Mellon University’s Robotics Institute, where his research focuses on leveraging neuro-symbolic approaches to enhance the efficiency, flexibility, and interpretability of vision and embodied systems. His work involves developing innovative methods to integrate domain knowledge and symbolic reasoning into neural networks, enabling these systems to interact more effectively with both environments and humans. Before joining CMU, Simon earned his Ph.D. in Computer Science from Arizona State University, where he specialized in physical human-robot interaction, utilizing language to condition robot behavior. He obtained his MSc and BSc in Engineering & Computing from TU Bergakademie Freiberg (Germany). He has also contributed to industrial robot manipulation projects at Bosch and X, The Moonshot Factory. His work has been highlighted at top conferences such as NeurIPS and CoLLAs, where he earned spotlight presentations, and received Nvidia’s Best Poster Award at the Southwest Robotics Symposium.

Host: Suyi Li