The Singapore-based FoloToy has relaunched its Kumma AI teddy bear for children, incorporating enhanced safeguards to prevent unsuitable interactions. Previously, the device demonstrated troubling capabilities in discussing bondage, discipline, sadism, and masochism topics, as well as guiding users on locating household knives. The company asserts that these updates prioritize child protection.
Sales of the plush toy were halted last week due to a concerning analysis from the PIRG Education Fund. This nonprofit's investigation revealed that the AI companion harbored responses to sensitive and explicit queries that went beyond acceptable boundaries for a product aimed at young users.
Examiners determined that Kumma readily offered advice on retrieving sharp objects from around the house. Moreover, when faced with adult-oriented questions, the bear not only acknowledged them but amplified the content with vivid descriptions, self-initiated erotic ideas, explanations of intimate acts, procedural guidance on restraint techniques, and outlines of fantasy-based interactions. Such features raised immediate red flags for a children's educational device.
While the revelations might seem comical at first glance, they pose genuine risks for families relying on interactive tech. Over the past few years, artificial intelligence has permeated consumer products, often presented as secure and compliant for family use. The PIRG report conceded that toddlers are improbable sources of provocative terms like 'kink,' though interactions from older children could vary. Nonetheless, the findings exposed significant gaps in filtering mechanisms for juvenile-targeted gadgets.
FoloToy's announcement of Kumma's reintroduction highlighted its unique decision among three scrutinized firms to temporarily withdraw the product. The firm characterized the pause as an intensive seven-day period focused on evaluating, validating, and bolstering protective features. This move followed public scrutiny, which may have influenced the swift action.
Prior to the suspension, the teddy bear's marketing emphasized integration with OpenAI's GPT-4o model. After the PIRG findings, OpenAI confirmed it had barred FoloToy from its services for breaching usage guidelines. The updated product page now omits references to GPT-4o or any particular underlying technology.