Anthropic (the company behind the Claude AI tool) has reportedly dismantled classic physical literature in an effort to make the scanning process go faster in a bid to further train their AI models on their material. Here’s an explainer on why and how this trend has come to fruition.
Why and how this trend is happening
Known as “Project Panama," Anthropic is reportedly using that codename to define a massive effort to acquire a vast amount of printed books and convert them into AI training data.
The AI company is said to have spent millions of dollars acquiring millions of physical books in bulk and proceeding to cut off the spines of said books so the individual pages could be more easily fed through high-speed industrial scanners. This process then led to those pages being converted into machine-readable text for machine learning — that information eventually became a part of a larger dataset used to train Claude AI. And due to those original books’ bindings being removed, they were disposed of as they were now considered nothing but trash.
The Washington Post’s report featured a quote from an internal planning document connected to Anthropic’s ultimate goal. “Project Panama is our effort to destructively scan all the books in the world,” that document stated. “We don’t want it to be known that we are working on this.” This past July, Anthropic settled with those same authors at $1.5 billion.
Reuters reported that the court overseeing that case ruled that, while training AI on books is classified as fair use under copyright law, Anthropic violated the copyrights of several authors and their publishers by keeping 7 million pirated books of theirs in a central library. While companies such as OpenAI, Google and Meta have been embroiled in legal trouble regarding the use of books being used for AI training, Anthropic is the only one that’s been exposed thus far for actually dismantling physical books to train its models.
Here’s what Anthropic had to say
A spokesperson for Anthropic tells Tom’s Guide, "Claude is trained on a mix of publicly available web data, commercially acquired datasets, and data we generate ourselves,” the Anthropic spokesperson told us. “Sourcing books is a widely used approach for training large language models across the AI industry. None of our data acquisition programs buy and destroy rare or antiquarian books."
Anthropic also provided more background on how they engage in the practice of using books to train AI:
- Sourcing books is a widely used approach for training LLMs across the AI industry.
- None of our data acquisition programs buy and destroy ‘rare’ or ‘antiquarian’ books. We buy books from regular commercial markets.
- Much of the chatter online seems to reference our recent Bartz settlement ; as a reminder: We settled this case last summer, after the court found that training AI on books is allowed under copyright law [known as ‘fair use’] — a ruling that still stands.
- As Judge Alsup wrote: "Like any reader aspiring to be a writer, Anthropic's LLMs trained upon works not to race ahead and replicate or supplant them — but to turn a hard corner and create something different."
- Bartz specifically concerns claims that Anthropic improperly downloaded materials from two online libraries — the LibGen and PiLiMi datasets.
- Anthropic never commercially released any model trained on the LibGen or PiLiMi datasets, and earned no revenue from any such model.
- We are pleased that more than 91% of authors and publishers covered by the settlement have claimed their share of the payment.
Bottom line
Rare and even out-of-print books may have been obtained by Anthropic for their past efforts, but we might never truly know until years later if the names of those demolished books become widely known.
I’m sure no one wants to live in a world dominated by AI tools that have been trained on reading material that’s not readily available in their original physical form.
