THE BBC has been accused of producing a “deeply flawed” report on the level of abuse MPs receive on social media.
The broadcaster has claimed more than 3000 offensive tweets are sent to MPs every day following an eight-month investigation.
Reporters used the AI tool Perspective to analyse three million tweets aimed at Westminster politicians over a six-week period and found around one in 20 could be classed as “toxic”.
But several academics have claimed the tool is unreliable in detecting some forms of abuse, including racist language.
One of the conclusions the report draws, for example, is that ethnic minority MPs aren’t more likely to receive toxic tweets, but journalists seemingly didn’t programme the AI to pick up racial slurs.
Journalist and former politics lecturer Ash Sarkar has insisted she would have failed a student who produced research drawing such “unreliable” conclusions.
She told the National: “The methodology is deeply and, I would argue, fatally flawed and those flaws mean the conclusions they come to are fundamentally unreliable.
“I used to be a lecturer and if one of my students put together a piece of research like this I would fail them because it just does not stand up to basic scrutiny.
“It’s using an AI that’s programmed to flag specific words and one of the conclusions of the report is ethnic minority MPs aren’t more likely to receive toxic tweets when they seemingly didn’t programme the AI to flag racist tweets.”
The BBC put together a report about "toxic" tweets aimed at MPs, using an AI to analyse 3 million tweets. It claimed ethnicity didn't play a role in MPs being more likely to receive abusive tweets, but didn't even programme the AI to flag racist abuse. And the report's still up!
— Ash Sarkar (@AyoCaesar) November 9, 2022
Sarkar said she tested out the application programming interface (API) used by the BBC and found racist comments were not being picked up as toxic at all.
Meanwhile, in the BBC’s table showing the top 100 words used against MPs, words like “stupid” and “idiot” were regarded as being the most commonly used adjectives in toxic tweets.
Sarkar added: “I tested it out and it didn’t even pick up terms like “p*ki”.
Just did a test run on how it measures a compliment, vs how it measures a selection of racist tweets I've personally received. It didn't even process the racist tweets. pic.twitter.com/SgHpPGn6QU
— Ash Sarkar (@AyoCaesar) November 9, 2022
“If you’re saying we’ve drawn a conclusion that ethnic minority MPs don’t receive more toxic replies than anyone else, but you exclude explicitly racist content, then that is an unreliable conclusion to draw.
“When you look at the list of MPs they compiled as being the most targeted as well, 90% are white.
“[Labour MP] Diane Abbott doesn’t make the list and in 2017 she received over half of all abuse aimed at female MPs, so I just don’t think that’s plausible.
“I was plugging in terms like 'go back to your country' and 'Muslims don’t belong here' and white supremacist slogans and none of them even flagged at all on the toxicity rating.
“It’s embarrassing."
Problems with 'toxic' ratings
The research found Tory MP Ben Bradley received the highest percentage of tweets rated as "toxic" between March 15 and April 23, followed by Labour MPs Christian Matheson and James Murray.
In Scotland, the research was riddled with apparent errors. It reported that nine of the top 10 MPs who received the most abuse were SNP MPs.
However, that same BBC spreadsheet reports elsewhere that Tory MP David Mundell has a "toxicity percentage" of 5.4, SNP MP Stephen Flynn had one of 4.9, and Scottish LibDem MP Alistair Carmichael had one of 4.8.
These scores should have placed those three MPs in the top ten of most abused Scottish MPs, but they are omitted from the list with no reasoning given.
Data also showed Boris Johnson received the largest number of tweets considered toxic at 19,000, around 4% of the total he received.
The BBC further provided "full analysis of the top 100 words used against male and female MPs".
Appearing on the list were five names. These were "Nadine", "Boris", "Johnson", "Putin", and "tulipsiddiq" (the Labour MP for Hampstead and Kilburn Tulip Siddiq).
However, despite her name appearing in full on the list of "the top 100 words used against" MPs, Siddiq came 72nd on the BBC's list of most abused MPs.
What are data academics saying?
Other academics have raised concerns that swear words such as “f******” appear to make the toxicity rate spike even if the programme doesn’t know who the tweet is being directed at or the context.
The BBC report on the abuse of politicians is unhappy reading, & certainly more needs to be done. But as we found with our research, AI driven approaches miss important elements of abuse (such as direction) and are inaccurate. Let me show you why👇🧵.https://t.co/vjTOjKnZCb
— Dr Liam Mcloughlin ☕ (@Leelum) November 9, 2022
Dr Liam McLoughlin, who studies political communication and has written before in academic journals about the abuse of MPs on Twitter, said: “The system that they used was trained on Wikipedia data and data from the New York Times, I believe.
“So the system does not understand the UK way of talking, the nuances of UK politics and as a result, it’s got so many flaws in it.
“There are Nazi dog-whistles in the system that it will detect as not being abusive.
“In the tests we ran, it didn’t pick up ‘it’s a shame Hitler didn’t get to your family too’ as abusive, but it did say ‘Channel 4 is f****** rubbish’ was abusive.
“I’m frustrated with the way this has been reported. It’s really dangerous because there are people who have been working on this very nuanced, really important issue for years and they [the BBC] have not referenced it.”
The BBC does say in its in-depth report on the issue that one of the “inherent problems” with textual analysis tools is they “cannot truly determine whether the negative sentiment was aimed directly at the MP in question or whether the MP was simply mentioned in a negative tweet”.
One of the authors, Paul Bradshaw, also said they had been “very careful” with the language in the report so as not to misrepresent what the AI was capable of.
He later admitted on Twitter that "an extra caveat might be good to include" in the BBC's story.
The broadcaster has been approached for further comment.
