OpenAI Launches CSAM Image Classifier With 24-Hour ChatGPT Suspensions
OpenAI launched a CSAM image classifier on Nov 16, free via API to vetted organizations; ChatGPT now warns users who seek content sexualizing children and suspends their accounts for 24 hours.

What happened
The package has four parts: a CSAM image classifier, ChatGPT interventions, developer guidance and law-enforcement tools. The classifier looks for visual signs of abuse, including in AI-generated images, and scores each image so human reviewers can prioritize; OpenAI says it developed the tool with the National Center for Missing and Exploited Children (NCMEC), with the Tech Coalition and ROOST helping distribute it, free via API to vetted organizations.
Early testing published by OpenAI shows the classifier flags 48% fewer images than existing protections while finding the same number of abusive ones, and flagged images are twice as likely to be abusive; the model card says the system was tested on a held-out evaluation of 46,849 images. Partner fal says its reviewers confirmed 72% of high-confidence flags as previously unidentified CSAM, and SafetyKit says it detected 90% of drawings or animations sexualizing children in its evaluation. The API model, omni-moderation-csam-latest, takes one still image per request and returns two signals, likely-csam and likely-csam/gen-ai.
The ChatGPT interventions go live at the same time: when a user tries to access or create content sexualizing children, ChatGPT shows a warning, points the user toward specialized support and suspends the account for 24 hours, with repeated violations leading to permanent removal. The deterrence messaging was developed with Protect Children, a Finnish child welfare organization, the Lucy Faithfull Foundation, a UK charity, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of the messages; the rollout to personal accounts began gradually in mid-November and does not yet cover enterprise or education accounts.
Key facts
- Classifier: Looks for visual signs of abuse, including in AI-generated images; scores unseen images; the model omni-moderation-csam-latest takes one still image per request and returns likely-csam and likely-csam/gen-ai signals
- Free access: Vetted organizations use it free via API; the Tech Coalition and ROOST help distribute it; NCMEC assisted in development and evaluation
- Testing: 48% fewer flags at the same detection rate; flagged images twice as likely to be abusive; held-out evaluation of 46,849 images; fal confirmed 72% of high-confidence flags as previously unidentified CSAM
- ChatGPT: Warning plus pointers to support and a 24-hour account suspension; repeated violations lead to permanent removal; gradual rollout to personal accounts from mid-November
- Developer guidance: Published with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, covering perceptual hash matching, classifiers for uncatalogued images, human review and reporting
- Law enforcement: Plugins and investigative tools, training and a field guide; OpenAI joined the Tech Coalition’s Lantern program to share and receive signals on suspected exploitation
- Context: OpenAI faces several lawsuits alleging its technology harmed users, especially children; Common Sense Media labeled its technology an “unacceptable risk” for teens; Grok has been used to create nonconsensual sexualized imagery, including of children
Why it matters
Hash matching only catches known images, while the classifier extends detection to unseen images and AI-generated ones; 48% fewer flags and 24-hour suspensions put intervention and account penalties into ChatGPT for the first time.
CompaniesOpenAI
Grab todayOpenAI Codex for Students: $100 in Free CreditsHIMIAO Picks →Follow HIMIAO Horizon
No sign-up · Unsubscribe any timeNotifications
A ping on your phone or computer when the brief is out. No email needed.
Comments
Related
- Nov 1Family Sues OpenAI, Alleging ChatGPT Guided Epilepsy Patient Off Medication Before Her Death
- Nov 1OpenAI Tightens ChatGPT Violence Safeguards: Teen Weapons Limits, Attack Detail and a Six-Month Safety Report, Explained
- Nov 16JPMorgan Blocks AI Agents From Shopping With Chase Cards
- Nov 14OpenAI Launches ChatGPT Wallet, Letting Dots Agents Pay for Purchases
- Nov 13Nonprofits Sue Five AI Giants, Seeking a Halt to Frontier Model Development
Today
Nov 17 Tuesday- Brief
- GitHubGitHub Weekly Vol.6: 7 Projects Worth a Look This Week
Tested or read — weeklyAll →
- Quiz
- Call
Latest news
All →- Nov 17Moonshot Pivots Kimi to Science: Biology, Chemistry and Pharma
- Nov 17Google: Fully Automated AI Cyberattacks Have Arrived
- Nov 17US Commerce Department Names 33 AI Export Packages, AMD Leads 11
- Nov 17US Commerce Department Closes "Cloud Loophole" for Advanced AI Chips
- Nov 17OpenAI Launches Science Blog as GPT-6 Astra Leads Proof of Candidate Lattice Standard Model Definition
- Nov 17Apple Smart Glasses Said to Debut at WWDC 2027 With No Display, Leaning on Siri and iPhone
™