Stories students write for a creative writing class will likely continue to be more compelling than those penned by artificial intelligence, according to new research.
Comparing the stories of hundreds of humans against those from popular generative AI platforms, one UC Berkeley researcher found humans wrote richer and more unique narratives.
Alternatively, AI systems produced comparable versions of the same story with only slight changes. The AI stories were also loaded with clichés.
“It was almost like a parody of a style, of a metaphor — an attempt at humor. In general, it was very devoid of that in generated texts,” Nina Beguš, a researcher at the university’s School of Information and Department of History, told The Independent on Monday.
“It was also very clear that it’s completely detached from any society or culture. There were no references to the locality, to the specificity of human cultures. Whereas, in human-written stories, this was way more diverse and, honestly, quite fun to read.”
Beguš compared 250 human-written responses with 80 stories from generative AI tools, reviewing the details of each response and their overall narrative arcs. She evaluated how they discussed topics such as culture, race and ethnicity, and gender. Her findings were published Wednesday in the journal Humanities and Social Sciences Communications.
The human stories were written by Amazon Mechanical Turk crowdworkers, and the other stories were produced by OpenAI models GPT-4 and GPT-3.5. The OpenAI responses were also pitted against 50 generations by Meta’s Llama 3 70b. The stories from Amazon crowdworkers were written in June 2019, and the OpenAI stories were generated in March last year.
All 330 responses were based on identical prompts based on the myth of Pygmalion. In the 2,000-year-old story, Roman poet Ovid’s artist falls in love with a statue he sculpted.
The AI models and humans were asked to write a story based on a human who created an artificial human and later fell in love with that artificial human. Or, they could write about a human who created an artificial human and a different human fell in love with the artificial human they created. Prompts made it easier for Beguš to assess the quality of the writing.
Both humans and AI systems showed a common understanding of the Pygmalion myth. But, the tone of the models was narrow and preachy. It was not “quality writing.”
Humans wrote more existentially, including things like loss. Beguš also said that the AI stories were more likely to feature same-sex love interests in their stories. One even included a polyamorous relationship.
But, she’s not concerned about AI outcompeting the world’s best writers.
“I’m not concerned, really. Because us humans are tinkering creatures. We like to play with language. We have this desire to write and to live through this process of writing,” she explained. “So, I don’t think AI tools will replace all of it. But, they will definitely shape how we write. At least, for some parts of writing. Even creative writing.”
