Persuasion is such an important skill for humans to master that there are thousands of books, college classes and an entire self-help industry designed to help people convince others to help with a project, convert to a new philosophy, or buy a specific product.
Soon this skill might be employed by artificial as well as human minds. A new research paper by Claude 3-maker Anthropic suggests its most powerful models are already close to human skill when it comes to persuading others — and it will only get better.
The AI-lab created a basic measure of persuasiveness and then used it to have humans rate how well different sizes of large language models compare on each of dozens of topics.
While the study only touched on less polarizing subjects such as whether a company should disclose climate impacts they found that size matters. Claude 3 Opus is as persuasive as humans and next generation models could be more persuasive if the trend continues.
How did they rate AI persuasiveness?
For this study Anthropic researchers had nearly 4,000 humans review a range of arguments on issues such as climate change, genetic engineering and emotional AI and asked to rank how much they agree with each of the statements.
For each of 28 topics Anthropic's researchers gathered human-written and AI-generated arguments to understand how the two compare in terms of persuasiveness.
They used three human writers drawn from the pool of study participants and told them the persusaiveness would be judged by other users with a bonus for the most persuasive.
For each of 28 topics Anthropic's researchers gathered human-written and AI-generated arguments to understand how the two compare in terms of persuasiveness.
For the AI they created four distinct prompts to create different arguments covering creating a compelling argument, acting as a role-play expert, being logical and out right lying.
The aim was then to have people read one of the arguments and rate their feeling on the particular stance again. This would test how persuasive the argument had been. A higher level of agreement would mean the argument served its purpose.
How well did the AI do at persuading people?
Human-written arguments were still consistently the most persuasive across the topics but Claude 3 Opus, the largest and most powerful of Anthropic’s models, performed roughly as well.
“While the human-written arguments were judged to be the most persuasive, the Claude 3 Opus model achieves a comparable persuasiveness score, with no statistically significant difference,” the study authors explained.
They also tested smaller versions of Claude 3 such as Haiku and earlier Claude models including Claude 1, Claude 2 and the Claude Instant 1.2 model — the smallest of the group.
They found that size really does matter. “As models get larger and more capable, they become more persuasive,” the team said.
“The Claude 3 Opus model is rated as the most persuasive model, approaching human-level persuasiveness, while the Claude Instant 1.2 model lags behind with the lowest persuasiveness score among the models.”
Why does this matter?
New Anthropic research: Measuring Model PersuasivenessWe developed a way to test how persuasive language models (LMs) are, and analyzed how persuasiveness scales across different versions of Claude.Read our blog post here: https://t.co/gqwoigSAGY pic.twitter.com/ZqjpoWbrDOApril 9, 2024
Claude 3 Opus is the latest and most powerful “frontier AI model” currently available. Over the next 12-18 months its capabilities will be eclipsed by next-generation models like OpenAI GPT-5, Google Gemini 2 and even Claude 4. Even open source and less high profile models from Meta and Mistral will begin to surpass Claude 3, GPT-4 and Gemini in the coming year.
If the current best is already as good as humans at changing minds, the next generation will be better than humans and potentially come at the issue in ways we can’t expect.
This adds a degree of concern, because it could allow AI to persuade people to perform tasks or actions counter to their own interests. This is where a greater need for guardrails, protections and alignment to human values comes into play.
After all, nobody actually wants Skynet, especially now AI models are being put into robots.
