In collaboration with: 

Smith, Jason B, Nazanin Tabatabaei, Karan Khosla

 

AI Holodeck, an application to generate virtual 3D scenes from natural language input using environment-specific knowledge we gathered through an annotated dataset. The AI Holodeck application gathers common-sense knowledge through two layers. First, semantic annotation of image datasets provides the system with objects implicitly related to those explicitly defined by the user. Second, a CLIP-guided search extracts objects and their spatial relationships from reference images related to the user input. This second layer also allows the system to display the images searched for by CLIP as visual explanations of the system's scene generation method.

​

Here you can find our publication on AI Holodeck.