Vidhi Jain

email: {first name}{last name} at cmu dot edu

Hi, I am a PhD student at the Robotics Institute (RI), part of the School of Computer Science at Carnegie Mellon University (CMU), where I am advised by Yonatan Bisk. I am also a student researcher at Google DeepMind Robotics. Before this, I worked at Meta AI and Microsoft Research India.

I am interested in learning algorithms for interactive and adaptable embodied AI. My long term vision is to develop robots that can perform multiple tasks around the home and learn new skills from their users. My research focuses at the intersection of language, vision and actions to enhance real-time perception, motion control and dialog in robots.

Bio |  CV  |  GitHub  |  Google Scholar  |  LinkedIn  |  Twitter

profile photo


September 2023,

Spatial Language Attention Policies accepted at CoRL'23

How to use few examples to learn manipulation skills? SLAP is a new approach that learns to attend to spatial language to learn manipulation skills.
June 2023,

Started as Student Researcher at Google DeepMind

I am excited to start as a student researcher at Google DeepMind, Mountain View. I will be working with Debidatta Dwibedi on end-to-end video conditioned policy learning for robotics.
June 2023,

HomeRobot Challenge at NeurIPS'23

Check out the HomeRobot, a large-scale sim-to-real mobile manipulation challenge at @NeurIPSConf 2023! More details about the challenge here. You can submit to EvalAI here. Our paper (accepted at CoRL 2023) shows RL and heuristic policies for sim to real transfer and identifies the challenges in the domain.
October 2022,

Blogpost on AI residents at Meta

I am working with Akshara Rai and Yixin Lin on preference-based task planning for dishwasher loading task. Read about the work of 2021-2022 AI residents at Meta here.

September 2022,

Transformers Task Planners accepted at CoRL'22

Our work on learning preferences for canonical dish loading task using Transformers Task Planner is accepted at CoRL 2022. Read more here.

Show more ...
March 2020,

Heidelberg Laureate Forum (HLF)

Selected among 224 young researchers to meet laureates in the mathematics and computer science (postponed to Sep 2021); Participated in Virtual HLF 2020.
August 2019,

Research Fellow at Microsoft Research

I explored deep learning theory for generative models at Microsoft Research India, advised by Amit Deshpande and Navin Goyal.
June 2019,

J. N. Tata Endowment Scholarship for Higher Studies, 2019

Awarded interest-free loan and travel scholarship for higher studies.
May 2019,

K. C. Mahindra Scholarships for Post-Graduate Studies Abroad, 2019

62 students were awarded with interest-free loan for higher studies.
July 2018,

Bachelor Thesis at Mila, University de Montreal

I worked on my bachelor thesis Investigating the viability of Generative Models for Novelty Detection, with Aaron Courville.
July 2017,

Mitacs Globalink Research Internship

I was a Mitacs Globalink Research Intern at Simon Fraser University, Burnaby, Canada. I worked with Prof. Oliver Schulte on bayesian optimization algorithms for machine learning. Find our code here.
March 2017,

Citi Women Leader Award (CWLA) Scholarship

Awarded one year of study scholarship (Top 3 among 1200 candidates selected nationwide).
cwla group pic


project image

Spatial Language Attention Policies for Efficient Robot Learning

Priyam Parasher, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Yonatan Bisk and Chris Paxton."
7th Annual Conference on Robot Learning (CoRL), 2023

webpage | arXiv | code | reviews | bibTeX

project image

HomeRobot: Open-Vocabulary Mobile Manipulation

Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, Zsolt Kira, Manolis Savva, Angel X Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, Chris Paxton.
7th Annual Conference on Robot Learning (CoRL), 2023

webpage | arXiv | code | reviews | bibTeX

project image

Transformers are Adaptable Task Planners

Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk and Akshara Rai.
6th Annual Conference on Robot Learning (CoRL), 2022

webpage | arXiv | video | code | reviews | bibTeX

project image

MAEA: Multimodal Attribution in Embodied AI

Vidhi Jain, Jayant Sravan Tamarapalli, Sahiti Yerramilli, and Yonatan Bisk.
NeurIPS Workshop on Trustworthy Embodied AI, 2022

webpage | arXiv | video | reviews | bibTeX

project image

Towards Explainable Embodied AI

Vidhi Jain
Masters thesis, 2021

pdf | bibTeX

project image

Learning to capture spatial semantic priors for indoor navigation

Vidhi Jain, Shishir Patil, Prakhar Agarwal and Katia Sycara.
NeurIPS Object Representations for Learning and Reasoning (ORLR) , 2020

pdf | webpage | arXiv | video | code | bibTeX

project image

Predicting strategies in simulated search and rescue tasks

Vidhi Jain, Rohit Jena, Huao Li, Tejus Gupta, Dana Hughes, Michael Lewis and Katia Sycara.
NeurIPS AI for Humanitarian Assistance and Disaster Response (AIADR) , 2020

arXiv | video | slides | bibTeX

project image

Learning to navigate in unseen cluttered environments

Vidhi Jain, Ganesh Iyer and Katia Sycara.
NeurIPS Women in Machine Learning workshop (WiML), 2020

pdf | poster | bibTeX

project image

Coping with sample inefficiency in deep reinforcement learning

Vidhi Jain, Simin Liu, and Ganesh Iyer.
ICML Women in Machine Learning Un-Workshop (WiML), 2020

pdf | slides | bibTeX

project image

Investigating the viability of Generative Models for Novelty Detection

Vidhi Jain
Bachelors thesis, 2018

pdf | bibTeX

project image

Symptomatic Diagnosis and Prognosis of Psychiatric Disorders through Personal Gadgets

Vidhi Jain, Simin Liu, and Ganesh Iyer.
ACM CHI Extended Abstracts (CHI EA '17), 2017

pdf | webpage | slides | poster | bibTeX

project image

Model Selection Scores for Multi-Relational Bayesian Networks

Sajjad Gholami, Oliver Schulte, Vidhi Jain, Qiang Zhao.
IJCAI Declarative Learning Based Programming (DeLBP), 2017

pdf | code | bibTeX

project image

Empowering API Consumer Community: Collaborative Annotation of Web API Documentation for Semantically Structured Format

Vidhi Jain and Matthias Frank
Grace Hopper Conference India (GHCI), 2016

pdf | poster |


December 2022,

MAEA: Multimodal Attribution in Embodied AI

Presented our work on Multimodal Attribution in Embodied AI at NeurIPS 2022 at Trustworthy Embodied AI workshop. Watch the talk here.
March 2021,

AI Symposium organized by SAiDL & APPCAIR

Invited as Early Career Speaker to discuss AI research and suggestions to get started in it. Video

December 2020,

Learning Embeddings that Capture Spatial Semantics for Indoor Navigation.

Presented our work on Predicting Human Strategies in Simulated Search and Rescue at NeurIPS 2020 AI+HADR workshop. Watch the talk here.
December 2020,

Predicting Human Strategies in Simulated Search and Rescue

Presented our work on Predicting Human Strategies in Simulated Search and Rescue at NeurIPS 2020 AI+HADR workshop. Watch the talk here.
July 2017,

NDTV telecast on Innovation by Young India

Invited speaker for panel discussion on national news NDTV India to present project Automated Psychiatrist. Watch here@5:07.
Show more ...
April 2019,

Pyladies Bangalore

gave a tutorial on Deep Learning with PyTorch. Resources & Event Poster
April 2019,

Speaker at IIIT-Bangalore ACM Student Chapter

Invited for a information session on internships and research. Video
April 2019,

One in Asankhya Project

Invited for a discussion for one-in-asankhya project. Blogpost & Video


logo image

Master of Science in Robotics (MSR)

Advisor: Katia Sycara (social machine intelligence)  
Thesis: Towards Explainable Embodied AI
logo image

Bachelors of Engineering (Honors) in Computer Science

Advisor: Aaron Courville, Mila (off-campus thesis)  
Thesis: Investigating viability of generative models for out-of-distribution detection.

Design and source code from Leonid Keselman's Jekyll fork and Jon Barron's website