Gen AI’s Risks to Child Safety on Digital Platforms

In the last two years, generative AI has seen unprecedented advances. The technology ushered in the ability to create content and spread ideas faster than ever before. Yet these same capabilities present critical implications for child safety. In short, generative AI is introducing new vectors of risk for kids online, but it’s also unlocking critical tools to combat online child sexual abuse at scale as Thorn's Head of Data Science, Dr. Rebecca Portnoff, discussed during a recent presentation.

Content-hosting platforms considering or in the process of deploying gen AI solutions need to understand the risks that misuse of this potent technology poses to their platforms and users. Doing so is not only crucial to preventing child sexual abuse and exploitation from occurring on their domains, but to maintaining a safe and legally compliant digital space.

Bad actors misuse gen AI to conduct child sexual abuse

As they’ve done with numerous other technologies, bad actors are exploiting generative AI models to perpetrate, proliferate, and further sexual harms against children. In a recent webinar, Thorn's Senior Research Manager, Amanda Goharian discussed the critical implications of the misuse of generative AI may have for Trust and Safety teams focused on child safety.

Gen AI unlocks rapid content creation. Using generative AI, perpetrators may adapt original child sexual abuse material (CSAM) images and videos into new abuse material, create fully AI-generated CSAM, or manipulate benign material of children into sexualized content.

Meanwhile, generative AI text models allow bad actors to rapidly scale child grooming conversations, as well as the sextortion of child victims, often carried out by organized crime.

This misuse is already occurring. And mitigating these crimes requires a proactive approach by all content-hosting platforms.

Safeguarding platforms and preventing abuse

It is illegal for digital platforms to host CSAM. Thus, preventing its viral spread matters not just to the vital safety of children and online communities, but to a platform’s legal compliance and risk.

Mitigating CSAM and other sexual harms against children caused by the misuse of gen AI requires a multifaceted strategy, which should include a Safety-by-Design approach to the development and deployment of products and technologies, as well as on-platform detection solutions.

Safety by Design

Safety by Design is a proactive approach to product design. It requires that companies anticipate where threats may occur during the development process and design in the necessary safeguards — rather than retrofit solutions after harm has occurred.

Each stage in the AI ecosystem includes opportunities to prioritize child safety, regardless of data modality or if an organization releases its technology as closed source or open source.

Such mitigations may include detecting, removing, and reporting CSAM from your AI training data, and including prevention messaging for CSAM solicitation.

On-platform detection

Even as bad actors co-opt AI for child sexual abuse and exploitation, this innovative technology also holds the power to help protect platforms and their users from these harms.

Thorn’s solution Safer empowers content-hosting platforms to detect, review, and report CSAM at scale, leveraging both hash matching capabilities and state-of-the-art predictive AI classifiers.

Designed by Thorn’s experts in child safety technology, Safer identifies known CSAM using cryptographic and proprietary perceptual hashing to match against a database of 57.3 million CSAM hashes from trusted sources.

Through predictive AI/ML image and video classifiers, Safer also detects novel CSAM and can be strategically enlisted by Trust and Safety teams to prioritize reports or conduct investigations into high-risk accounts.

In addition, Safer’s new text classifier provides line-by-line predictions for complete conversation analysis and assigns risk scores for a range of abuse-related text, including child access, CSAM, sextortion, and self-generated content.

Together, Safer’s features allow digital platforms to address CSAM at scale and give platforms the opportunity to proactively mitigate risks related to hosting CSAM.

Let’s build a safer internet

Across the digital ecosystem, generative AI introduces numerous benefits for consumers and businesses alike, transforming how we live and work. Yet these advancements must be designed, developed, deployed, and maintained with child safety in mind. Already, we’re seeing the consequences of generative AI’s misuse to further sexual harms against children, while putting platforms and users at risk.

Digital platforms don't have to tackle this issue alone. Together, we can drive meaningful action and build safer online environments.