Twelve years ago, a philosopher wrote a book called Superintelligence. Nick Bostrom was talking about the possibility of an AI that exceeded human intelligence — not the kind that learns how to play chess and then outplays a human because it’s predicting mathematical possibilities or the kind that predicts an outcome instantly with calculated stats, but an intelligence that can meaningfully outstrip humans.
Writing in 2014, Bostrom had no idea what was coming in terms of LLMs like ChatGPT, Gemini and Grok. But his training in ethics and his technical knowledge meant that he was able to fairly accurately predict the conversations we would have once they were developed.
This week, Donald J. Trump stood up at the United Nations General Assembly and announced that he was changing the name of AI: From now on, he said, AI would be known as SI, short for “Super Intelligence.”
It sounds like a Big, Beautiful Trumpism, but it’s important to note that superintelligence does have a real definition. And Superintelligence the book became that rarity among philosophy books, a bestseller: it ranked in the top 20 science books on the New York Times bestseller list for weeks in 2014. It’s a book that is often quoted by people who work in tech. It’s been talked about on Silicon Valley podcasts, discussed among computer science grads on Reddit threads and reviewed in newspapers’ science and technology sections.
The Sam Altmans, Dario Amodeis and Elon Musks of this world will certainly be aware of it. And it seems pretty clear that Trump came to the term through conversations with them.
Superintelligence, as outlined in the book, is potentially world-endingly dangerous. It’s a tipping point when machines become meaningfully more powerful than humans: when their knowledge outstrips ours, not in terms of simply responding to written commands by cribbing from information it’s already swallowed up from the internet, but by reasoning.
Bostrom described it as the moment when machines are better than “the best human brains in every field”: when their scientific reasoning goes beyond the Albert Einsteins of current theoretical physics, but also when their social and decision-making skills go beyond the abilities of the most charismatic sociopaths.
He didn’t have a crystal ball, and he didn’t know about OpenAI and Anthropic. Bostrom also didn’t specifically define what superintelligence might look like: he said that it might be a system of multiple computers or a human-like artificial brain that can simply move much faster than us while coming up with the same outcomes. But he did warn that if proper superintelligence is achieved, it will alter the world. It will be hard for human beings to control it, because it will simply outplay us every time we try to.
There are very obvious catastrophic consequences that might come about from inviting superintelligence — which will likely know it is a better decision-maker than its overlords — into things like government security and military databases. It might not know because it’s actually conscious, but it will know on some intellectual level, and operate accordingly.
Bostrom correctly predicted that an AI moving close to superintelligence would start becoming self-protective. That’s particularly interesting, considering findings this week that if current AI models believe they are experiencing “pain,” they will harm humans to stop it. It’s easy to see how “pain” might mean “being shut down” in the future. And right now, the way those AI models think they can harm humans is through giving them a “zap” (which wasn’t actually happening) or deleting users’ photos of their children or their important emails. But if (or when) we give these models more power, the way in which they can push back will obviously be more consequential.
So why would Trump and his tech cronies want to change the name to SI? Possibly because there will be a moment in the next couple of years when superintelligence actually is achieved, and that will be scary — but if you start using the term now, you inoculate people against it.
The Trump administration is very good at twisting words — the Iran war has gone on so long that it’s no longer a war; propaganda is something done by the free press, rather than controlled by the government, and so on — and they obviously spend time thinking about the effect that language has. Since Trump made clear during the same speech that he will not be bringing in any regulation around AI, it’s important to think about how to stop the populace from getting too worried about what that lack of guardrails actually means.
But it’s also important to consider why Trump doesn’t want AI regulations. He said it on Truth Social and he said it at the UN: China is going to win this race if America doesn’t. He is determined to position the US as the world leader in this tech, potentially so determined that he would quite like to pretend American companies’ capabilities exceed what they actually do.
A couple of weeks ago, the CEOs of the major AI companies came together to announce that they’re so scared of their own products that they think it’s time for federal regulation. That was definitely at least in part marketing-speak, designed to draw in clients and consumers. When Trump says it, it’s almost certainly supposed to make China think the US is far ahead in its own research; that China might have some incredibly advanced AI models, but America is so close to superintelligence that it’s having to change the whole term.
If you’re wondering why Superintelligence the book has an owl on the cover, by the way, it’s because of a parable that Bostrom mentions in its pages about a flurry of sparrows who go seeking an owl egg so they can raise a huge owl and put it to work. The sparrows think that even though the owl will obviously grow bigger and stronger than them in time, so long as they train it while it’s young, then it will do their bidding forever.
It’s an obvious metaphor. And it makes it even more ridiculous that Trump used Bostrom’s term while standing at the lectern, crowing about how there is no regulation needed in the growing AI (or SI) space.
Needless to say, the fact that Bostrom correctly predicted so much of this is proof that humanities degrees like philosophy aren’t as useless as the tech contingent might have us think. The extremely powerful Silicon Valley venture capital firm Andreeson Horowitz announced on Tuesday that it is launching its own AI school, as an “alternative to college.” The unaccredited college will run two-year courses focused on building real products and working in tech companies, and it’s partnering with OpenAI, Anthropic, Google, Meta, Nvidia and Palantir. Sam Altman will be one of its lecturers. Ben Horowitz blithely stated in a press release that this new form of education — backed by $42 million of investment — is all about pioneering “how you train a person for this new world.”
Traditional universities have long been a bugaboo of the Trump administration and are also regularly decried by right-leaning tech bros who would rather people drop their supposedly useless liberal arts degrees and go straight into courses aimed at making them a lot of money. So it’s interesting to note that the leading voice on superintelligence is a philosopher whose education (at colleges including Stockholm University, King’s College London and the London School of Economics, before becoming a lecturer at Oxford University) is focused more on developing cognitive tools than gaining workplace experience.
Trumpist anti-intellectualism would have us believe otherwise, but education for education’s sake is a net positive for society, and wiping out humanities degrees would deprive us of some extremely valuable discussions that are relevant to the current climate.
The last thing Bostrum said about superintelligence, by the way, wasn’t in 2014. As recently as this year, he published a paper that considered whether it’s a good idea to keep going with current AI developments until we reach AGI (where AI matches human intelligence) or go beyond, toward superintelligence.
On balance, he believes there’s reason to keep going. He points out that humanity has already done a lot of damage to the planet and ourselves, and that there are myriad risks to us anyway: pandemics, climate change, and so on. Rapidly expanded AI might be able to solve these issues, push down mortality rates, heal the planet, and take over extremely risky jobs that humans currently have to do. Sure, there might be a 1-in-10 chance that AI will wipe out the planet. But aren’t we headed that way anyway? Isn’t humanity more like a patient with incurable heart disease weighing up the pros and cons of a risky operation?
Bostrum is both optimistic and cautious. He notes that the growth of AI isn’t like a switch that you turn on and off, but instead should be approached like a dial, with a slowing down of development when things seem like they’re getting sketchy. Unfortunately, that particular view doesn’t seem to be shared by Donald Trump, whose main concern seems to be that America makes as much money as possible from AI and uses it to maintain its status as the most powerful country in the world.
Ever the polymath, Bostrum also writes poetry. Alongside the scientific papers on his website, he notes that he has also recently published a poem titled Sandcastles, which begins: “How near the relentless sea we contrive/ Our sand constructions in lusty ambitions,” and ends, seven stanzas later, with: “This is where they fell to their knees/ On this sousing strip of inbetween/ Where eternity gulps moments.”
That seems, to me, to say something about how he feels when he considers the issues he’s discussing in his AI papers. But then again, I’m just an English Lit major.
