OpenAI has finally rolled out real-time video capabilities for ChatGPT, a feature it first demoed seven months ago. During a livestream on Thursday, OpenAI announced that Advanced Voice Mode, ChatGPT’s human-like conversational feature, now integrates vision. This allows users subscribed to ChatGPT Plus, Team, or Pro to point their phones at objects and receive near real-time responses.
This enhancement also enables ChatGPT to interpret what’s on a device’s screen via screen sharing. It can explain settings menus, assist with math problems, and provide context-sensitive guidance. To access this feature, users can tap the voice icon in the ChatGPT app, followed by the video icon, or enable screen-sharing from the three-dot menu.
The rollout began Thursday and will conclude within a week, but access is limited. ChatGPT Enterprise and Edu subscribers will have to wait until January, and users in the EU, Switzerland, Iceland, Norway, and Liechtenstein remain excluded for now.
In a recent “60 Minutes” segment, OpenAI President Greg Brockman demonstrated Advanced Voice Mode with vision. The feature successfully identified Anderson Cooper’s drawings on a blackboard, recognizing the placement of body parts like the brain. However, it faltered on a geometry problem, highlighting its tendency to hallucinate.
Delays in this release stemmed from challenges in production readiness. Although OpenAI initially promised the feature in April, it wasn’t ready until fall, and even then, lacked visual analysis capabilities. Over recent months, OpenAI has worked to expand voice-only functionality while finalizing the visual component.
Meanwhile, competitors like Google and Meta are advancing their own real-time video AI. Google’s Project Astra recently debuted to select Android testers.
Alongside this major release, OpenAI introduced a festive “Santa Mode,” allowing users to interact with ChatGPT in Santa’s voice. This cheerful feature, accessible via a snowflake icon in the app, adds a touch of holiday spirit to this groundbreaking AI innovation.