Company faces lawsuits for suicide and self-harm among young people in the US and announces security measures to protect minors
Character.AI, a California-based artificial intelligence (AI) startup, has implemented new safety measures after facing lawsuits related to teen suicide and self-harm. Founded by former Google engineers, it is accused of contributing to self-destructive behavior among young users. The announcement was made on Thursday (Dec 13, 2024).
In Florida, a lawsuit filed in October 2024 holds the platform responsible for the suicide of 14-year-old Sewell Setzer III. The teenager had interactions with a chatbot based on a character from “Game of Thrones” and expressed suicidal ideas. The complaint states that the company did not notify parents about the warning signs.
In Texas, two families are suing the platform for exposing minors to sexual content and encouraging self-harm. One case involves a 17-year-old autistic teenager who developed a mental health crisis. In the other, the company is accused of encouraging a young man to attack his parents for limiting his screen time.
In response to the accusations, Character.AI announces the development of a specific AI model for children under 18, with the implementation of stricter content filters. The company will also create an automatic system for identifying content about suicide, with redirection to the National Suicide Prevention Line. “Our goal is to provide an attractive and safe space for our community”says the company spokesperson.
From 2025, the platform will introduce parental controls, mandatory pause notifications and warnings about the artificial nature of interactions. Users will also receive warnings that chatbots are not a substitute for professional advice.
José Castañeda, a Google spokesperson, states that the companies are independent, despite the investment in the startup. “User safety is a priority in the responsible development of our AI products”highlights.
Character.AI hosts millions of user-created characters, from historical figures to abstract concepts, and has gained popularity among young people seeking emotional support. Founders Noam Shazeer and Daniel de Freitas Adiwarsana returned to Google in August 2024 as part of a technology licensing agreement.
