Personally, I'd rather not feed all of my anxieties into a black box of AI modelling like ChatGPT—that's what my therapist, private Discord server, and locked social media accounts are for. Joking aside, an alarming amount of folks are turning to LLMs rather than fellow humans when it comes to puzzling out personal problems. For these users, OpenAI is attempting to ensure ChatGPT won't lead them up the garden path into thornier emotional territory.
According to the latest update blog post, if a user asks ChatGPT "Should I break up with my boyfriend?" for instance, "ChatGPT shouldn't give you an answer." As young people are apparently becoming increasingly reliant on ChatGPT for emotional support—at least, so says OpenAI CEO Sam Altman—this example is hardly out of left field.
