Meta has acknowledged problems with its moderation and recommendations system, after a joint investigation from academics and the Wall Street Journal.
The report outlines how Instagram’s recommendation algorithms helped “connect and promote a vast network of accounts openly devoted to the commission and purchase of underage-sex content”.
In response to the report, Meta has set up an internal task force to tackle the problem. “Child exploitation is a horrific crime," Meta told the paper. "We’re continuously investigating ways to actively defend against this behaviour."
The investigation conducted by researchers at Stanford University and the University of Massachusetts Amherst does not suggest that the algorithms surface child-abuse content to those who aren’t actively seeking it. Rather, algorithms promoting paedophile networks work in the same way as other systems: they serve up more content to those already searching for it.
And that’s exactly what happened when researchers set up a test account viewing content shared by paedophile networks: they were recommended more of such accounts to follow.
But just because this is surfaced algorithmically, that doesn’t mean that Instagram is incapable of doing more, as not all of the accounts are subtle or coded.
While some accounts use mildly coded phrases and emojis, researchers discovered accounts using more blatant hashtags such as “#preteensex”, “#pedowhore” and “#pedobait”. They reportedly offer “menus” of illegal content for users to buy or commission.
“That a team of three academics with limited access could find such a huge network should set off alarms at Meta,” said Alex Stamos, formerly chief security officer at the company, but now head of Stanford’s Internet Observatory. “I hope the company reinvests in human investigators.”
Indeed, there are signs that beyond the problems with algorithms surfacing illegal content, Meta’s human moderation side simply isn’t substantial enough to deal with everything flagged by disturbed users trying to do the right thing.
The WSJ report outlines an example where a concerned user reported an Instagram account purporting to be of a girl selling underage sex content with a post captioned “This teen is ready for you pervs.” The company responded with an automated message saying that, thanks to the “high volume of reports we receive”, the account hadn’t been reviewed.
Another reported post featuring a young girl accompanied by a “graphically sexual caption” received a response that the post “does not go against our Community Guidelines”.
Meta accepted that it had received reports that it had failed to act on, the WSJ reported, with a spokesperson explaining that a “software glitch” prevented a “substantial” portion of reports being processed. It also accepted that its moderation staff were failing to adequately enforce the rules.
Both of these flaws have now been addressed, the company said, adding that it has been proactive in taking down accounts that violated its child safety policies, with 490,000 being removed in January alone.
While paedophile networks attempt to infiltrate most social networks, Instagram appears especially vulnerable. The researchers found just 128 accounts offering to sell child-sex content on Twitter despite its relatively small size and well-documented moderation problems – less than a third of the number found on Instagram. Equally, it does “not appear to proliferate” on TikTok.
“You have to put guardrails in place for something that growth-intensive to still be nominally safe, and Instagram hasn’t,” concluded David Thiel, the chief technologist at the Stanford Internet Observatory.
