I want a robot assistant that can work with me to accomplish everyday tasks around the home: tidying up the living room, preparing the table for dinner, etc. This broad capability involves the use of natural language for task specification, embodied multi-agent planning, and robust skill execution. Toward this end, I will present our simulation benchmark called PARTNR: Planning And Reasoning Tasks in humaN-Robot collaboration. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs as planners for these tasks, and reveal limitations in coordination, task tracking, and failure recovery. When paired with real human partners, these LLMs are less efficient than human-human collaboration by 1.5x and less efficient than a single human by 1.1x. Through the PARTNR benchmark, many directions of research can be pursued, for example multi-agent planning, 3D scene understanding, skill coordination, Sim2Real transfer, and even HRI studies.
Jacob Krantz is a research scientist at the Fundamental AI Research lab at Meta (FAIR), where he works on systems of embodied intelligence with applications toward socially intelligent robotics. In the past he co-organized several community challenges relating to semantically driven navigation (VLN, ImageNav). He received his PhD in Computer Science from ¾«¶«Ó°ÊÓin 2023 under Dr. Stefan Lee and his BS in Computer Science with a minor in Physics from Gonzaga University in 2019.