Flash

Flash·Safety & Risk·2026-11-17 08:12

OpenAI Launches CSAM Image Classifier With 24-Hour ChatGPT Suspensions

OpenAI launched a CSAM image classifier on Nov 16, free via API to vetted organizations; ChatGPT now warns users who seek content sexualizing children and suspends their accounts for 24 hours.

A shield icon on a monitor in a bright office
AI-generated illustration

What happened

The package has four parts: a CSAM image classifier, ChatGPT interventions, developer guidance and law-enforcement tools. The classifier looks for visual signs of abuse, including in AI-generated images, and scores each image so human reviewers can prioritize; OpenAI says it developed the tool with the National Center for Missing and Exploited Children (NCMEC), with the Tech Coalition and ROOST helping distribute it, free via API to vetted organizations.

Early testing published by OpenAI shows the classifier flags 48% fewer images than existing protections while finding the same number of abusive ones, and flagged images are twice as likely to be abusive; the model card says the system was tested on a held-out evaluation of 46,849 images. Partner fal says its reviewers confirmed 72% of high-confidence flags as previously unidentified CSAM, and SafetyKit says it detected 90% of drawings or animations sexualizing children in its evaluation. The API model, omni-moderation-csam-latest, takes one still image per request and returns two signals, likely-csam and likely-csam/gen-ai.

The ChatGPT interventions go live at the same time: when a user tries to access or create content sexualizing children, ChatGPT shows a warning, points the user toward specialized support and suspends the account for 24 hours, with repeated violations leading to permanent removal. The deterrence messaging was developed with Protect Children, a Finnish child welfare organization, the Lucy Faithfull Foundation, a UK charity, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of the messages; the rollout to personal accounts began gradually in mid-November and does not yet cover enterprise or education accounts.

Key facts

  1. Classifier: Looks for visual signs of abuse, including in AI-generated images; scores unseen images; the model omni-moderation-csam-latest takes one still image per request and returns likely-csam and likely-csam/gen-ai signals
  2. Free access: Vetted organizations use it free via API; the Tech Coalition and ROOST help distribute it; NCMEC assisted in development and evaluation
  3. Testing: 48% fewer flags at the same detection rate; flagged images twice as likely to be abusive; held-out evaluation of 46,849 images; fal confirmed 72% of high-confidence flags as previously unidentified CSAM
  4. ChatGPT: Warning plus pointers to support and a 24-hour account suspension; repeated violations lead to permanent removal; gradual rollout to personal accounts from mid-November
  5. Developer guidance: Published with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, covering perceptual hash matching, classifiers for uncatalogued images, human review and reporting
  6. Law enforcement: Plugins and investigative tools, training and a field guide; OpenAI joined the Tech Coalition’s Lantern program to share and receive signals on suspected exploitation
  7. Context: OpenAI faces several lawsuits alleging its technology harmed users, especially children; Common Sense Media labeled its technology an “unacceptable risk” for teens; Grok has been used to create nonconsensual sexualized imagery, including of children

Why it matters

Hash matching only catches known images, while the classifier extends detection to unseen images and AI-generated ones; 48% fewer flags and 24-hour suspensions put intervention and account penalties into ChatGPT for the first time.
Useful Tap if this story helped you
Send to a friendMakes a share image for X, WeChat, Telegram and more

SourcesTechCrunch (Nov 16, 2026); OpenAI CSAM Image Classifier model card (Nov 16, 2026). This article is compiled from public information and does not constitute investment advice.

From HIMIAO Horizon · himiao.ai

Grab todayOpenAI Codex for Students: $100 in Free CreditsHIMIAO Picks →

Follow HIMIAO Horizon

No sign-up · Unsubscribe any time

Email

Notifications

A ping on your phone or computer when the brief is out. No email needed.

HIMIAO PicksWeekly freebies: one email every Friday, a 5-minute read

Comments

  1. Loading comments…
Ask the cat