Fourteen-year-old Zeynep Demirbas has found that ChatGPT-4o is less accurate than a mental health-focused AI model and a simpler machine-learning system at detecting stress in human written text.
Get breaking news anytime, anywhere. Download the TOI app now!
Zeynep, an eighth-grade student at Transit Middle School in East Amherst, New York, tested four different models using more than 3,500 Reddit posts. According to Society for Science, the posts had already been labelled by humans as showing stress or no stress.
Her project, titled “ Evaluating the reliability of Large Language Models for stress detection ”, has earned her a place among the finalists in the 2025 Thermo Fisher Scientific Junior Innovators Challenge. The competition recognises young students working on science-based projects.
Zeynep became interested in the project after speaking with a family friend who is a psychologist. The psychologist told her that some health insurance companies were exploring large language models (LLMs) as cheaper, 24/7 alternatives to human therapists. Zeynep wondered whether AI systems could actually be trusted to identify stress.
Testing AI models
To test the models, Zeynep used a dataset called Dreaddit. It contains 3,553 Reddit posts that human raters had labelled according to whether they contained signs of stress.
She gave the data to four different models including Bidirectional Encoder Representations from Transformers (BERT), MentalBERT, Random Forest and ChatGPT-4o. MentalBERT is a version of BERT designed for mental health-related language, while Random Forest is a basic machine-learning technique that uses multiple decision trees to make predictions.
Zeynep asked each model to identify which posts showed stress. She then used a measure called an F1-score to compare their performance. The score considers both how accurately a model identifies stress and how often it misses stress or wrongly labels a post as showing stress.
MentalBERT performed the best in her testing, with a score of about 82 percent. BERT followed with about 79 percent. ChatGPT-4o scored about 74 percent. It also performed worse than the Random Forest model, which was included as a simpler baseline for comparison.
The result surprised Zeynep because Random Forest is a much simpler machine-learning method and does not understand language and context in the same way an LLM does. “ChatGPT performing badly was ‘really surprising,’” Zeynep said.
She found it particularly interesting that a simpler model could outperform an LLM with millions of parameters. “Random-forest is ‘supposed to be a very simple and old technique. So I just put it in as a baseline,’” Zeynep said, as quoted by Science News Explores. “That was very interesting; how something so small and simple was able to beat an LLM like ChatGPT that used millions of parameters and had so much coding go into it,” she added.
