Imagine picking up an item in virtual reality and watching it appear in your real-world space seemingly by magic. That’s exactly what scientists at Princeton University are working on with their new mixed-reality experiment.
The Princeton research team has introduced a system called Reality Promises: Virtual-Physical Decoupling Illusions in Mixed Reality via Invisible Mobile Robots. The idea is simple but groundbreaking: when users select objects in a VR environment, real-world robots hidden from sight deliver the physical equivalent to the same location.
“By decoupling virtual modes of manipulation from the physical mode of manipulation, Reality Promises create the illusion of manipulating the physical scene instantaneously, using magical forces or fantastical creatures,” the researchers explained.
Assistant computer science professor Parastoo Abtahi emphasized the significance of this innovation, saying it could “fundamentally change” how humans interact with their surroundings, since “we are no longer limited by the laws of physical reality in our interactions.”
A demonstration video shows users wearing VR headsets virtually moving objects—a plant, coconut water, and a tube of chips—around a room. As the countdown timer in VR completed, invisible robots delivered the corresponding physical items to the exact same spots.
In one striking example, a digital bee appeared to carry a tube of chips through the air. In truth, a robot discreetly delivered the real object. Project leader Mohamed Kari highlighted that removing visual clutter from the VR interface, including the robot itself, strengthens the illusion. “Even the robot itself,” Kari said, is omitted so users feel the transfer happened “as if by magic.”
Abtahi confirmed the authenticity of the results: “Yes, the robot is actually delivering the chip in the last clip, but also the plant and the coconut water in the other clips. No footage is theoretical.”
Building this futuristic illusion required extensive digital reconstruction. The team created a “digital twin” of the workspace using about 200 images and several video clips from an iPhone 14 Pro. These were processed with 3D Gaussian splatting, a technique for generating realistic 3D environments. As Abtahi explained, “We train the splat in Jawset Postshot (v0.5.146), using the MCMC profile, and cap the splat count at 25k splats for scenes and 3k–5k for objects.”
For the physical component, they used a commercially available Stretch 3 robot paired with a Meta Quest 3 VR headset. Users guided the robot with hand gestures in VR, allowing it to fetch and reposition objects in real life.
While the illusion is already compelling, the researchers admit the current version leaves behind a faint outline of the robot. Their next step is to improve the technology so that the robot becomes “truly invisible.” According to Abtahi, this will involve boosting the visual fidelity of the VR scene, raising splat counts, and refining color blending.
The team also hopes to expand the robot’s capabilities beyond simple pick-and-place actions to tasks like pressing buttons or pouring liquids. Remote telepresence, collaborative work, and immersive entertainment are among the long-term applications envisioned.
The paper will be formally presented at the ACM Symposium on User Interface Software and Technology this September in Busan, South Korea.
