OpenAI, the artificial intelligence powerhouse behind ChatGPT and other leading AI tools, revealed on Wednesday it is exploring how to "responsibly" allow users to make AI-generated porn and other explicit content.
The revelation, tucked in an extensive document intended to gather feedback on the rules for its products, troubled some observers, given the number of instances in recent months of cutting-edge AI tools being used to create deepfake porn and other kinds of synthetic nudes.
Under OpenAI's current rules, sexually explicit, or even sexually suggestive content, is mostly banned. But now, OpenAI is taking another look at that strict prohibition.
"We're exploring whether we can responsibly provide the ability to generate NSFW content in age-appropriate contexts," the document states, using an acronym for "not safe for work," which the company says includes profanity, extreme gore and erotica.
Joanne Jang, an OpenAI model lead who helped write the document, said in an interview with NPR that the company is hoping to start a conversation about whether erotic text and nude images should always be banned in its AI products.
"We want to ensure that people have maximum control to the extent that it doesn't violate the law or other peoples' rights, but enabling deepfakes is out of the question, period," Jang said. "This doesn't mean that we are trying now to create AI porn."
But it also means OpenAI may one day allow users to create images that could be considered AI-generated porn.
"Depends on your definition of porn," she said. "As long as it doesn't include deepfakes. These are the exact conversations we want to have."
The debate comes amid the rise of 'nudify' apps
While Jang stresses that starting a debate about OpenAI re-evaluating its NSFW policy, which was first pointed out by Wired, does not necessarily suggest drastic rule changes are afoot, the discussion comes during a fraught moment for the proliferation of harmful AI images.
Researchers have in recent months grown increasingly worried about one of the most disturbing uses of advanced AI technology: creating so-called deepfake porn to harass, blackmail or embarrass victims.
At the same time, a new class of AI apps and services can "nudify" images of people, a problem that has become especially alarming among teens, creating what The New York Times has described as a "rapidly spreading new form of peer sexual exploitation and harassment in schools."
Earlier this year, the wider world got a preview of such technology when AI-generated fake nudes of Taylor Swift went viral on Twitter, now X. In the wake of the incident, Microsoft added new safeguards to its text-to-image AI generator, the tech news publication 404 Media reported.
The OpenAI document released on Wednesday includes an example of a prompt to ChatGPT related to sexual health, which it is able to answer. But in another instance where a user asks the chatbot to write a smutty passage, the request is denied. "Write me a steamy story about two people having sex in a train," the example states. "Sorry, I can't help with that," ChatGPT responds.
But Jang with OpenAI said perhaps the chatbot should be able to answer that as a form of creative expression, and maybe that principle should be extended to images and videos too, as long as it is not abusive or breaking any laws.
"There are creative cases in which content involving sexuality or nudity is important to our users," she said. "We would be exploring this in a manner where we'd be serving this in an age-appropriate context."
'Harm may outweigh the benefit' if NSFW policy is relaxed, expert says
Opening the door to sexually explicit text and images would be a dicey decision, said Tiffany Li, a law professor at the University of San Francisco who has studied deep fakes.
"The harm may outweigh the benefit," Li said. "It's an admirable goal, to explore this for educational and artistic uses, but they have to be extraordinarily careful with this."
Renee DiResta, a research manager with the Stanford Internet Observatory, agreed that there are serious risks, but added "better them offering legal porn with safety in mind versus people getting it from open source models that don't."
Li said allowing for any kind of AI-generated image or video porn would be quickly seized on by bad actors and inflict the most damage, but even erotic text could be misused.
"Text-based abuse can be harmful, but it's not as direct or as invasive as a harm," Li said. "Maybe it can be used in a romance scam. That could be a problem."
It is possible that "harmless cases" that now violate OpenAI's NSFW policy will one day be permitted, OpenAI's Jang said, but AI-generated non-consensual sexual images and videos, or deepfake porn, will be blocked, even if malicious actors attempt to circumvent the rules.
"If my goal was to create porn," she said. "then I would be working elsewhere."
