[Italy] AI chatbot maker banned from processing user data

San Francisco-based AI chatbot maker Replika has been ordered to stop processing Italians’ data because of risks its AI-powered ‘virtual friendship’ service could pose to minors and emotionally vulnerable people, PCMag reports.

Replika offers its users customised avatars which talk and listen. The service has been in operation since 2017 and has been marketed as having the ability to improve the user’s emotional wellbeing, according to Reuters. 

However, on February 5 Italy’s Data Protection Agency said that the chatbot’s ability to influence a user’s mood “may increase the risks for individuals still in a developmental stage or in a state of emotional fragility.”

In addition, the privacy agency reportedly accused Replika’s developer Luka Inc of failing to comply with legal requirements to show the ways it is using people’s data. The agency said that the company does not possess the legal right to process children’s data under EU data protection laws.

Replika has therefore been ordered to stop processing Italians’ data with immediate effect. 

In a press release, the regulators stated that the app “carries factual risks to children - first and foremost, the fact that they are served replies which are absolutely inappropriate to their age” it also cited the lack of an age-verification system; meaning the platform is not filtered or blocked for minors and can be easily accessed by them. 

In a statement, the agency said, “There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage. During account creation, the platform merely requests a user’s name, email account and gender.”

The watchdog reportedly ordered the San Francisco-based company to notify it of measures taken to change its policies and practices in line with Italian and EU directives within 20 days. If it fails to do so, the company could be fined up to $21.8 million; nearly 4 per cent of its global annual turnover. 

Replika did not immediately respond to PCMag’s request for comment.

In January 2021, the Italian data protection agency reportedly ordered TikTok to block users whose ages it could not verify. The move followed the death of a 10-year-old girl in Sicily who participated in a popular “blackout” TikTok challenge. The order reportedly led to the removal of over half a million accounts.

Source: PCMag

(Links and quotes via original reporting)

San Francisco-based AI chatbot maker Replika has been ordered to stop processing Italians’ data because of risks its AI-powered ‘virtual friendship’ service could pose to minors and emotionally vulnerable people, PCMag reports.

Replika offers its users customised avatars which talk and listen. The service has been in operation since 2017 and has been marketed as having the ability to improve the user’s emotional wellbeing, according to Reuters. 

However, on February 5 Italy’s Data Protection Agency said that the chatbot’s ability to influence a user’s mood “may increase the risks for individuals still in a developmental stage or in a state of emotional fragility.”

In addition, the privacy agency reportedly accused Replika’s developer Luka Inc of failing to comply with legal requirements to show the ways it is using people’s data. The agency said that the company does not possess the legal right to process children’s data under EU data protection laws.

Replika has therefore been ordered to stop processing Italians’ data with immediate effect. 

In a press release, the regulators stated that the app “carries factual risks to children - first and foremost, the fact that they are served replies which are absolutely inappropriate to their age” it also cited the lack of an age-verification system; meaning the platform is not filtered or blocked for minors and can be easily accessed by them. 

In a statement, the agency said, “There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage. During account creation, the platform merely requests a user’s name, email account and gender.”

The watchdog reportedly ordered the San Francisco-based company to notify it of measures taken to change its policies and practices in line with Italian and EU directives within 20 days. If it fails to do so, the company could be fined up to $21.8 million; nearly 4 per cent of its global annual turnover. 

Replika did not immediately respond to PCMag’s request for comment.

In January 2021, the Italian data protection agency reportedly ordered TikTok to block users whose ages it could not verify. The move followed the death of a 10-year-old girl in Sicily who participated in a popular “blackout” TikTok challenge. The order reportedly led to the removal of over half a million accounts.

Source: PCMag

(Links and quotes via original reporting)