What is AI-generated CSAM (AIG-CSAM)?

AI-generated child sexual abuse material, or AIG-CSAM, includes images or videos depicting child sexual abuse that are created or altered using artificial intelligence.

CSAM is any content that depicts sexually explicit activities involving a child. Visual depictions can include photographs, videos, live-streaming, and digital or computer-generated images, like AI-generated content, that are indistinguishable from an actual minor. The term child sexual abuse material is preferred by investigators and organizations because it highlights the harm in the content. “Child pornography” implies consent, which doesn’t accurately describe sexual abuse and exploitation involving children.

AIG-CSAM still represents a small share of overall CSAM, but advances in generative AI are changing how quickly abusive content can be created and how realistic it can appear. It also affects how easily existing images of children can be manipulated. For trust and safety teams, this technology creates ongoing challenges in detection and review.

Within AIG-CSAM, the circumstances can vary considerably. Some material depicts entirely synthetic subjects. Other content manipulates benign photographs of real children, incorporates the likeness of an identifiable person, or alters existing abuse imagery to create new material.

What are AI-generated CSAM and deepfakes?

  • AI-generated CSAM (AIG-CSAM): Images or videos depicting child sexual abuse that are created or altered using artificial intelligence. The content could depict an identifiable individual (deepfake), or a fully synthetic depiction of a person. If an identifiable individual is the subject, that individual could be an identified or unidentified victim of hands-on abuse or could be a child that has never experienced hands-on abuse. You might also encounter the term AI-manipulated CSAM (AIM-CSAM), which captures media that has some element altered using AI technology, ranging from photorealistic to cartoon.
  • Deepfake nudes: Created with AI, deepfake nudes are synthetic media creations that depict real people in sexually suggestive or explicit situations or activities.

Why is AIG-CSAM a distinct child safety threat?

Continued advancements in generative AI make it possible to create increasingly photorealistic sexual imagery of children quickly and at scale, with little to no technical knowledge. While AIG-CSAM currently represents a smaller portion of the overall CSAM ecosystem, its rapid growth threatens to overwhelm current child protection systems.

Victim identification efforts represent an immediate concern. Investigators already work through enormous volumes of digital material searching for new evidence connected to children who may be experiencing ongoing abuse. As synthetic content adds more novel material to that process, determining which files point to a real child in immediate danger is increasingly more difficult. Because AIG-CSAM could depict a real child suffering sexual abuse, it’s not enough for investigators to simply detect if content is AI-generated and move on.

Generative AI also expands the potential for victimization and revictimization. Benign images of a child can be transformed into abusive material with a few clicks. Bad actors and other young people may use this technology to produce AIG-CSAM and other sexualizing content from non-explicit imagery of minors or of wholly fictional minors. Imagery connected to known victims can be used to generate new depictions of their abuse. Deepfaked imagery can also become part of harassment, bullying, sextortion, or other forms of exploitation.

The ease of creating AIG-CSAM, and the ability to do so without the victim’s involvement or knowledge, can perpetuate the misconception of this content being “harmless”. The accessibility of generative tools lowers social and technical barriers to sexualizing children. Among young people in Thorn research who acknowledged creating deepfake nudes of others, 70% said they downloaded the app they used through their device’s app store.

Generative AI can also support broader offender behavior by making it easier to create customized abusive material, such as adding themselves to existing imagery. It can also help exchange information about producing or obtaining CSAM. Generative AI models can provide bad actors with instructions for hands-on sexual abuse of a child, information on coercion, details on destroying evidence and manipulating artifacts of abuse, or advice on how to manipulate victims to stay silent.

How has AIG-CSAM created new challenges for online platforms?

Hash matching, which is the standard first line of defense against CSAM, is not effective at detecting material when new images are created. Generative AI makes creating new images that aren’t already in a hash list much easier. Hashes serve as digital fingerprints that allow platforms to compare uploaded files against libraries of previously identified CSAM. That remains highly effective for known material.

The challenge with AIG-CSAM is that each newly generated file is likely novel. Before it’s identified, reviewed, and hashed, it would have no existing fingerprint for a hash-matching system to find. This creates a triage problem. Novel CSAM traditionally receives significant attention because it may document previously unidentified abuse and point investigators toward a child who needs help. A growing volume of AI-generated material can make it more difficult to distinguish those cases quickly.

Platforms may therefore need additional signals that help reviewers understand both the nature and likely origin of the material they detect.

How can platforms prevent generative AI from being misused for AIG-CSAM?

Platforms can help prevent generative AI from being misused for AIG-CSAM by taking a proactive approach to system design. In 2024, Thorn and All Tech is Human forged “Safety by Design” principles which have been adopted by Amazon, Anthropic, Civitai, Google, Invoke, Meta, Metaphysic, Microsoft, Mistral AI, OpenAI, and Stability AI, and released a whitepaper with specific mitigations for preventing AIG-CSAM. This has been further extended through the IEEE Recommended Practice 3462-2026, which provides a standardized framework for Safety by Design.

Safety by Design mitigations guard against the creation and spread of AIG-CSAM with a proactive approach that requires companies to anticipate where threats may occur during the development process — and design necessary safeguards in advance instead of after harm has occurred.

Develop, build and train generative AI models that proactively address child safety risks.

Responsibly source your training datasets, and safeguard them from CSAM and CSEM: Detect and remove CSAM and CSEM from your training data, and report any confirmed CSAM to the relevant authorities.

Incorporate feedback loops and iterative stress-testing strategies in your development process: Continuous learning and testing to understand a model’s capabilities to produce abusive content is key in effectively combating the adversarial misuse of these models downstream.

Employ content provenance with adversarial misuse in mind: Content provenance solutions that can be used to reliably discern whether content is AI-generated will be crucial to effectively respond to AIG-CSAM.

Deploy and distribute models after they have been trained and evaluated for child safety

Safeguard your generative AI products and services from abusive content and conduct: Combat and respond to abusive content (CSAM, AIG-CSAM and CSEM) throughout your generative AI systems, and incorporate prevention efforts. Users’ voices are key: incorporate user reporting or feedback options to empower these users to build freely on your platforms.

Responsibly host your models: Responsibly host your first-party generative models, assessing them via red teaming or phased deployment for their potential to generate AIG-CSAM and CSEM. Also responsibly host third-party models in a way that minimizes the hosting of models that generate AIG-CSAM.

Encourage developer ownership in safety by design: Provide information about your models, including a child safety section detailing steps taken to avoid the downstream misuse of the model to further sexual harms against children. Support the developer ecosystem in their efforts to address child safety risks.

Maintain model and platform safety by continuing to actively understand and respond to child safety risks.

Prevent your services from scaling access to harmful tools: Remove models and services that allow bad actors to produce AIG-CSAM from your platform. This includes “nudifying” tools.

Invest in research and future technology solutions: Effectively combating the misuse of generative AI to further child sexual abuse will require continued research to stay up to date with new harm vectors and threats. Invest in relevant research and technology development to address the use of generative AI for online child sexual abuse and exploitation.

Fight CSAM, AIG-CSAM and CSEM on your platforms: Fight CSAM online and prevent your platforms from being used to create, store, solicit or distribute this material. As new threat vectors emerge, meet this moment.

How can platforms detect and triage AIG-CSAM?

Even with safeguards built through the development process, a layered detection approach is critical to help trust and safety teams detect and triage AIG-CSAM.

Hash matching remains important once AIG-CSAM has been detected. Once known, that material can be identified in future uploads using the same matching techniques used for other known CSAM.

Machine learning classifiers analyze the visual characteristics of an image or video to help detect previously unseen content that may contain CSAM. These classifiers evaluate the content itself rather than relying on a previous hash match.

AI detection has limitations, and generative models change quickly. Classifiers may not be as accurate when classifying content from models that they weren’t trained on. Content origin standards such as C2PA can provide useful information when provenance data is present and intact. In child-safety settings, however, platforms shouldn’t assume that someone creating abusive material will keep that information and should rely instead on tools like indelible watermarks that are purpose-built for generative AI applications.

How should platforms respond to AIG-CSAM?

Responding to AIG-CSAM requires a layered approach that addresses the full risk lifecycle.

Prevent misuse

Whenever possible, platform response should be built into the system from inception. Safety by Design principles and mitigations can help lower the chances of misuse and spot risks.

Detect known and novel content

Detection systems should account for both known and previously unseen material. Hash-matching systems can be paired with predictive classifiers to detect previously identified and novel suspected content that may require attention.

Prioritize content for review

After content is found, trust and safety teams need enough detail to know which material needs urgent review, especially when there is a lot to process. Tools like Context Labels can add information to help teams prioritize suspected high-risk content.

Maintain human review

Technology can help detect suspected CSAM, while trained human reviewers check the content and apply moderation rules. Platforms should set up clear policies for escalation and reporting. This way, human reviewers know when content needs further investigation and can report it to the right authorities.

Iterate as threats change

Child safety systems should keep improving as technology changes. Generative AI and misuse methods can change fast. Established safety processes require regular testing and reviews to evaluate how effective they are.

Building a broader response to AIG-CSAM

Platform response is one part of a greater child-safety effort. Through its Safety by Design initiative, Thorn advocates for AI developers and technology companies to consider child safety throughout model development, deployment, and maintenance. This includes technical safeguards, stronger industry standards, responsible model distribution, and continued investment in detecting abuse.

Protecting children from AI-enabled abuse will require cooperation across technology companies, child-safety organizations, policymakers, law enforcement, researchers, families, and communities. AIG-CSAM changes the tools available to people who seek to harm children, which means the systems built to prevent and respond to that harm must continue changing as well.

How Safer helps platforms strengthen CSAM detection and response

Safer is purpose-built to help trust and safety teams detect and respond to CSAM at scale. Safer’s tools combine multiple detection methods, so platforms can address both known and previously unseen material.

Safer Match uses hash matching to identify known CSAM, while Safer Predict uses machine learning to detect suspected novel CSAM for further review. Context Labels can add additional useful information about suspected CSAM to help teams prioritize higher-risk material and make more informed moderation decisions.

These capabilities can fit into existing trust and safety workflows with a variety of deployment options.

FAQs about AIG-CSAM

What does AIG-CSAM stand for?

AIG-CSAM stands for AI-generated child sexual abuse material. In this guide, the term includes CSAM created or altered using artificial intelligence, including wholly generated images and videos, and manipulated content involving real people.

What is the difference between wholly generated AIG-CSAM and AI-manipulated CSAM?

Wholly generated AIG-CSAM is created entirely by an AI model rather than beginning with an existing base image. AI-manipulated CSAM uses AI to alter existing media, such as by sexualizing a benign image or changing aspects of existing abuse material.

Is AIG-CSAM illegal?

According to US DOJ, AIG-CSAM depicting identifiable children or produced by models trained on CSAM is federally illegal under 18 U.S.C. §1466A.

State laws vary across regions. In some areas, additional work is underway to ensure local statutes fully cover AIG-CSAM. Platforms should work with legal counsel to understand the obligations that apply to their services.

Beyond legal repercussions, the emotional and psychological impact on victims of AIG-CSAM can be profound, affecting their sense of safety and well-being in the digital and real world.

How is AIG-CSAM different from deepfake nudes?

Deepfake CSAM is one form of AIG-CSAM that uses the likeness of an identifiable person to create or manipulate sexual imagery. AIG-CSAM also includes wholly synthetic content that does not depict an identifiable real child or modified images of existing CSAM.

Notably, while all deepfake nudes of minors qualify as AIG-CSAM, not all AIG-CSAM consists of deepfake nudes (e.g., AIG-CSAM that does not depict a child with a real identity).

Can hash matching detect AIG-CSAM?

Hash-matching cannot detect novel material. It can only match material that has already been identified and added to an appropriate hash library. Newly generated material will not have an existing hash when it first appears, so platforms need additional methods for detecting novel content.

How do you report suspected AI-generated CSAM?

Reporting requirements depend on the organization and jurisdiction. In the United States, electronic service providers are legally required to report apparent CSAM to the National Center for Missing & Exploited Children’s CyberTipline when they become aware of it. Members of the public can also report suspected online child sexual exploitation through the CyberTipline.