publications

2025

1. 

   VLMgineer: Vision Language Models as Robotic Toolsmiths

   George Jiayuan Gao^*, Tianyu Li^*, Junyao Shi , Yihan Li^†, Zizhe Zhang^†, Nadia Figueroa, and Dinesh Jayaraman

   Spotlight at RSS Workshop on Robot Hardware-Aware Intelligence, 2025.

   Project Website Abs PDF

   Tool design and use reflect the ability to understand and manipulate the physical world through creativity, planning, and foresight. As such, it is often regarded as a measurable indicator of cognitive intelligence across biological species. While much of today’s research on robotics intelligence focuses on generating better control strategies, inventing smarter tools offers a complementary form of physical intelligence: moving the problem-solving onus into the tool’s geometry so that control becomes simpler.This motivates us to ask: can today’s foundation models offer useful priors to automatically invent—and effectively wield—such tools? We present VLMgineer, a framework that harnesses the creativity of Vision–Language Models (VLMs) together with evolutionary search to co-design physical tools and the control policies that operate them. We evaluate VLMgineer on a diverse benchmark of everyday manipulation scenarios that demand creative tool design and use. Across this suite, VLMgineer consistently discovers tools and policies that solve tasks more effectively and innovatively, transforming challenging robotics problems into straightforward executions. It also consistently outperforms VLM-generated designs from human specifications and existing human-crafted tools for everyday tasks. To facilitate future research on automated tool invention, we will release our benchmark and code.

2. 

   Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy For Visuomotor Imitation Learning

   George Jiayuan Gao, Tianyu Li, and Nadia Figueroa

   International Conference on Intelligent Robots and Systems (IROS), 2025.

   Project Website Abs arXiv PDF Video

   We propose an object-centric recovery policy framework to address the challenges of out-of-distribution (OOD) scenarios in visuomotor policy learning. Previous behavior cloning (BC) methods rely heavily on a large amount of labeled data coverage, failing in unfamiliar spatial states. Without relying on extra data collection, our approach learns a recovery policy constructed by an inverse policy inferred from object keypoint manifold gradient in the original training data. The recovery policy serves as a simple add-on to any base visuomotor BC policy, agnostic to a specific method, guiding the system back towards the training distribution to ensure task success even in OOD situations. We demonstrate the effectiveness of our object-centric framework in both simulation and real robot experiments, achieving an improvement of 77.7% over the base policy in OOD.

3. 

   (On-Going) Eureka for Manipulation: Real-World Dexterous Agent via Large-Scale Reinforcement Learning

   Training a skilled manipulation agent with RL in simulation that can zero-shot transfer to the real world is hard. The question is: does this get any easier when we add LLM in the loop and utilize ginormous levels of computing power, such as hundreds of Nvidia’s latest generation of data-center GPUs?, 2025.

4. 

   Novel Environment Transfer of Visuomotor Policy Via Object-Centric Domain-Randomization

   Proposed GDN-ACT, a novel, scalable approach that enables zero-shot generalization of visuomotor policies across unseen environments, using a pre-trained state-space mapping for object localization, May 2025.

   PDF

2024

1. 

   (On-Going) Stable Visuomotor Policy from a Single Demo: Elastic Action Synthesis Data Augmentation

   We propose a methodology that uses our in-house Elastic-Motion-Policy, enabling the training of visuomotor policies with full spatial generalization from only a single demonstration, May 2024.

2023

1. 

   Modular Gait Optimization: From Unit Moves to Multi-Step Trajectory in Bipedal Systems

   Proposed the Gait Modularization and Optimization Technique (GMOT), which leverages modular unit gaits as initialization for Hybrid Direct Collocation (HDC), reducing sensitivity to constraints and enhancing computational stability across various gaits, including walking, running, and hopping, Dec 2023.

   PDF Code

2. 

   Miniature City Autonomous Driving Platform Development with Real-Time Vision-Based Lane-Following

   Developed the drive stack for Washington University’s inaugural miniature city autonomous driving platform by developing the vision-based lane-following pipeline, May 2023.

   Code