Italy Has Banned AI-Powered Virtual Girlfriends From Collecting Users’ Data

Italy privacy watchdog has ordered San Francisco-based AI chatbot maker Replika, which operates a freemium ‘virtual friendship’ service based on customizable digital avatars whose “personalized” responses are powered by artificial intelligence (and designed, per its pitch, to make human users feel better), to stop the use of Italian’s data due to concerns that the AI-driven “virtual friendship” service could have negative effects on minors and individuals who are emotionally fragile.

Replika offers users customized avatars which talk and listen to them and has been in operation since 2017 and has been marketed as being able to improve the user’s emotional well-being. The company lacks a proper legal basis for processing children’s data under the EU’s data protection rules.

But Italy’s Data Protection Agency said on Friday that its ability to intervene in the user’s mood “may increase the risks for individuals still in a developmental stage or in a state of emotional fragility.”

The privacy agency also accused Replika’s developer Luka Inc of failing to comply with legal requirements to show how it’s using people’s data and said that the company does not possess the legal right to process children’s data under EU data protection laws.

Replika was an early API partner for OpenAI’s text-generating large language model technology, GPT-3 — although its service is not running on a carbon copy of GPT-3. Rather, the startup claims it “fine-tuned” GPT-3, using a network machine learning model trained on dialogue, to hone the generative technology for its particular use-case: Conversational AI companions.

These concerns are arguably pretty justified given recent reports of the chatbot “companion” spewing unwanted sexual comments at users. (And yes, Replika is also the app that men use to create AI girlfriends and verbally abuse them.)

Previously, Replika positioned itself as offering AI “friends” rather than AI partners, but it now prominently features sexting as a selling point for its paid subscriptions. There have been recent reports of users, even those using the supposed non-sexual free version, receiving inappropriate messages from the AI.

As a result, Replika has been ordered to stop processing Italians’ data with immediate effect.

“There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage,” the watchdog’s statement reads. “During account creation, the platform merely requests a user’s name, email account and gender.”

If the company fails to comply within 20 days, the regulator said that it could be fined up to $21.5 million, roughly four percent of the company’s “total worldwide annual turnover.”

Italy may be considered by some to be one of the most romantic countries in the world but the amore in the Italian air does not extend to artificial intelligence-powered girlfriends.