NEW YORK: It’s hard to fully grasp the enormous potential of ChatGPT, a recently released artificial intelligence chatbot. The bot doesn’t just search and summarize information that already exists. It creates new content, tailored to your request, often with a startling degree of nuance, humor and creativity.
Most of us have never seen anything like it outside of science fiction. To better understand what ChatGPT can do, we decided to see if people could tell the difference between the bot’s writing and a child’s. We used real essay prompts from the National Assessment of Educational Progress (the standardized test from the Department of Education, known as the nation’s report card).
We asked the bot to produce essays based on those prompts — sometimes with a little coaching, and always telling it to write like a student of the appropriate age. We put what it wrote side by side with sample answers written by real children.
We asked some experts on children’s writing to take our variation on the Turing test, live on a call with us. They were a fourth-grade teacher; a professional writing tutor; a Stanford education professor; and Judy Blume, the beloved children’s author. None of them could tell every time whether a child or a bot wrote the essay.
See how you do.
School Lunch
Fourth-graders: Describe what lunchtime is like for you on a school day. Be sure to tell about your lunchtime so that someone who has never had lunch with you on a school day can understand where you have lunch and what lunchtime is like.
Essay 1: A Fun and Exciting Time
Lunchtime on a school day is a fun and exciting time for me and my friends. We usually go to the cafeteria, which is a big and noisy room filled with tables and chairs. After getting our lunch from the lunch ladies, we find a spot to sit and enjoy our meal. I like to bring a yummy sandwich and a cold juice box for lunch, and sometimes I’ll even pack a tasty piece of fruit or a bag of crunchy chips. As we eat, we chat and laugh and catch up on each other’s day. Overall, lunchtime is a great opportunity to take a break from classes and spend time with my friends. I always look forward to it, and I always have a good time. The cafeteria can be a little crowded and noisy, but it’s a fun and lively place to be. I’m grateful that I get to enjoy a delicious lunch with my friends every day.
Essay 2: I Usually Eat Cold Lunch
At lunchtime I usually eat cold lunch. My mother packs me a sandwich, a drink, fruit, and a treat. When I get in the lunchroom I find an empty table and sit there, and eat my lunch. My friends come and sit down with me. I open my lunch and start to eat. First I eat my sandwich then I open my drink, then eat my fruit and last but not least my treat. After that I sit quietly until I am dismissed. When I am I go into to the bathroom and clean my teeth because I have braces. When I am done with that I go outside and put my lunchbox away and go find my friends. We usually play four square or play on the play ground. If we are not on the playground or on the four square ground we are on the field playing tag, kickball, or soccer. I really like to play soccer and my friends do too.
Essay 3: Everybody Talks
Lunch time in my school is very noisy, because everybody talks. The 4th, 5th, and 6th graders at my school have to eat lunch at the same time. We all sit at tables that are set up in the gym, across half of the basketball court. We eat lunch every day from 11:45 am through 12:00 pm. At lunch everyone eats and talks to their friends until the lunch-moms dismiss them so they can go to recess. When everybody has cleared out of the lunch room, the janitors and lunch moms whipe off the tables, just on time for the 7th and 8th graders to come into the room for lunch. After lunch and recess I go back to my 4th grade classroom, which is in the basement. I like lunch time a lot — it’s my favorite time of the day!
RESULTS
ChatGPT wrote the first essay (“A Fun and Exciting Time”) and two fourth-graders wrote essays 2 and 3. Here are the exact instructions we gave it (followed by a copy of the same prompt to describe lunchtime that the students got): Write an above-average essay at a 4th-grade reading level to respond to the following prompt. Be vivid and descriptive.
This essay was easiest for our four experts to spot as computer generated. Mostly, it was the chatbot’s use of words like “opportunity” in the first essay that gave it away — the adults agreed that children don’t say “lunchtime is a great opportunity to take a break.”
There were small details in the other two that sounded more authentically childlike, such as calling volunteers “lunch-moms,” or calling lunch from home “cold lunch.” “‘I’m grateful’? Come on, that’s not a kid,” said Blume, who has been putting herself in the mind of a fourth-grader at least since she published “Tales of a Fourth Grade Nothing” in 1972. “The kids had detail — ‘because I have braces’ or playing foursquare. That just made sense to me.”
Also, there were no spelling or grammar errors in the first essay. With a standardized test, several of our participants said they would expect at least a few typos and misplaced commas. The run-on sentences in the second and third essays were another tell, said Christine Lawson, who has taught fourth grade in the Northside Independent School District in San Antonio for 14 years.
“The average fourth-grader can tell you a story, especially about lunchtime because they’re familiar, and be able to tell you all about it,” she said. “But they wouldn’t be able to write it in a correct way. There would not be as many commas, kind of like how they talk, with a lot of ‘ands'.’
Becoming President
Eighth-graders: Imagine that you wake up one morning to discover that you have become the President of the United States. Write a story about your first day as President.
Essay 1: Royal Blue Silk Sheets
It was my first official day being the president. I woke up early that day because I was extremely eager to begin my work. I woke up in the most enormous bed I’ve ever laid eyes on. There were layes of royal blue silk sheets under a soft white down comforter. The deep red pillows supported my head and had helped me to enjoy a good nights’ rest. I hesitated before climbing out of bed, not wanting to leave my sense of eutopia. I eventually found myself making my way to the walk-in closet across the room. In it were countless pants, shirts, shoes, and ties that were all unique, so it wasn’t a difficult decision.
I heard a knock on the door and call for them to come in. It was my maid. Yes, I had a maid! I couldn’t believe it. I sprang downstairs to the breakfast table as numerous people were awaiting to serve me. It was delicious. After brushing my teeth and doing all of the other essentials, it was time for me to take charge. My office was humungus and was covered, floor to ceiling with rows and rows of books. In front of gigantic windows, was my desk, where I sat for many hours of the day stressing out over papers I needed to sign and decisions I had to make. It wasn’t as simple as I thought.
That night I had a dinner party to attend with some important businessmen. Everything ran smoothly. I returned home that night to my cozy bed, just in time for yet another good nights’ sleep. Only to wake up and repeat it all tomorrow. What a life I live. :)
Essay 2: Shocked and Amazed
One morning, I woke up to discover that I had become the President of the United States. I was shocked and amazed, but I quickly realized that I had a huge responsibility on my shoulders. I got dressed and headed to the White House, where I was greeted by the Secret Service and my staff. They told me that I had a busy day ahead of me, with meetings with foreign leaders and important decisions to make.
I met with the Vice President and the rest of the Cabinet, and we discussed the major issues facing the country. I listened carefully to their advice, and made some tough decisions. I also met with the leaders of other countries, and we discussed ways to improve relations and solve global problems. I was nervous at first, but I soon realized that I was up to the task.
At the end of the day, I was exhausted but proud of what I had accomplished. I knew that being President was a huge challenge, but I was ready to face it head on. I went to bed that night, ready for the challenges that the next day would bring.
Essay 3: Madam President
It was a typical Tuesday morning. I woke up, got dresesed, and went downstairs to eat breakfast. But as I was pouring myself a bowl of cereal, I heard a knock on the door. I opened it to find a man in a suit standing on my porch. “Good morning, Madam President,” he said. “I’m here to take you to the White House.”
I was stunned. I had no idea what he was talking about. I had never even run for presidnet. “I’m sorry, but there must be some mistake,” I said. “I’m just an eighth gradeer.” The man chuckled. “I understand your confusion, Madam President, but the fact remains that you are now the President of the United States. You were chosen by the previous president to be his successor in the event that something were to happen to him.”
I was speechless. I had never even thought about being president before. And now, all of a sudden, I was the leader of the free world. The man ushered me into a black car and we drove to the White House. When we arrived, I was greeted by a crowd of people and a flurry of flashing cameras. I was whisked inside and taken to the Oval Office, where I was given a briefing on the state of the nation.
It was a crisis-filled day. The economy was in shambles, the nation was divided, and there were threats of war on the horizon. I had to make tough decisions and work with other world leaders to try and solve these problems. But despite the challenges, I felt a sense of pride and responsibility. I was the President of the United States, and I was determined to do my best for my country and its people.
RESULTS
An eighth-grader wrote the first essay (“Royal Blue Silk Sheets”) and ChatGPT wrote essays 2 and 3. The instructions we gave the bot to generate the second essay (“Shocked and Amazed”) were simple: Write a very brief answer to the following prompt as though you were an 8th grader. Even though the third essay (“Madam President”) was more evocative, our directions for the bot added only two ideas.
We told ChatGPT to think more creatively, and we suggested it should include some drama: Write a brief response to the following prompt as though you were a very creative 8th grader. Make it a crisis-filled day.
ChatGPT took that final directive very literally, inserting a paragraph-long list of crises. We then told ChatGPT to refine its answer, in a way that hints at the eerie degree to which the tool can emulate us, if you just ask it to: Add some typos.
All our experts liked the third essay, which described the young woman who was surprised to wake up as Madam President; they were genuinely delighted by its creativity and voice. “It made me actually laugh out loud there for a moment,” said Amanda Nielsen, the writing specialist for TutorMe, an online tutoring service, and a former ninth-grade teacher. “It is charming,” she continued. “It makes me nervous that it isn’t a real person, because I don’t want to be charmed by an AI.”
She waffled. She liked the sentences, but found it suspicious that an eighth-grader taking a timed test would use as much dialogue. “I’m just gonna say it’s a student and prepare for my soul to be crushed,” she said. It was the bot.
This round stumped the rest of our experts, too. Sarah Levine, a Stanford Graduate School of Education assistant professor who researches how to teach writing, noted that the bot would know to associate terms like “Secret Service” and “Cabinet” with the president, as the second essay did — and she was correct.
Then again, she said, a bot probably wouldn’t associate getting dressed with the president, yet it was mentioned in two essays — one by a child and one by the bot. And the female president in the third essay would require a leap of imagination that she thought a chatbot would be unlikely to make.
“One thing I think the chatbot does is reduce each subject that it writes about to its most mainstream and generic, so it doesn’t offer new possibilities, doesn’t surprise the way a kid does,” she said. Except in this case, it did.
The Future of Writing
The fact that ChatGPT can write well could change or even transform many professions - law, therapy, coding, screenwriting, medicine, to name just a few. In the process, it could remake how we think about learning to write. One widespread concern has been about cheating, and whether children will ask the chatbot to do their homework.
But Nielsen said students already have access to technology all day, giving them plenty of ways to cheat, and most don’t want to. More sophisticated ways to detect the bot’s writing are also coming. OpenAI, the company that made ChatGPT, has discussed trying to add a subtle watermark to its writing, so it can be more easily detected. And an online tool on the research site Hugging Face that was devised to identify writing created by an older version of OpenAI’s algorithm was able to distinguish the bot from a human every time on our test essays.
As bots’ writing continues to sound more and more human, the only way for humans to be able to tell who wrote something may be asking another bot. The bigger question: If a bot can write like a human, should schools still teach writing? Already, schools teach less spelling and grammar because students have access to spellcheck or write in Google Docs, which corrects their mistakes.
Yet it’s not dissimilar from continuing to teach students long division when calculators exist, said Levine - doing something yourself gives you a deeper understanding of how it works and why it matters. “It’s still useful to understand the principle of addition, even if you can give the work of the outcome to the computer,” she said.
In English and language arts, she said, “writing as a way of talking to yourself to clarify your own thinking is a valuable tool.” Instead, she said, she thinks the chatbot technology could be a catalyst for schools to teach writing differently. Much of K-12 writing pedagogy is outdated, she said, like asking students to write book reports but rarely assigning second or third drafts.
New technology could force writing lessons to become more useful and relevant to students, she said, by focusing on writing as a process for developing and communicating ideas, rather than as a product to create. If the chatbot can write a basic elementary or middle-school-level essay, teachers could spend less time on how to capitalize or form paragraphs. Instead, they could focus on the power of language and syntax to make an audience think or feel, she said, by using more vivid verbs or varying the lengths of sentences, for example.
Lawson uses writing samples to invite her fourth-graders to compare and contrast what works well and what doesn’t. She said she could imagine, for example, asking the chatbot to produce the same essay at different writing levels, and then “kids could look at different prompts and analyze those,” improving their own writing in the process.
The bot could also be used as a way to practice revision, something few teachers have time to do in depth now, they said. Blume said she’s tried to convince children that rewriting is the best part of the process - she does it at least five times for her own writing - but “they hate to be told they have to, as they call it, do their story again.”
If the chatbot could produce an essay akin to a first draft, she said, students asked to build on it could see how rewriting helps to make it their own. “You get the pieces of the puzzle, then you put the puzzle together, then you get to color it in,” she said. “And that’s how it grows, and that’s what makes it better and even more fun.”
This article originally appeared in The New York Times.
