Wouldn’t AI takeover leave survivors?
The human species is resilient. It has survived every disaster thrown at it so far. And even catastrophes bigger than anything it has lived through, like global nuclear war, probably wouldn’t kill everyone. Nuclear war is often thought of as meaning the end of the world, but careful modeling suggests that, in reality, many would survive even such a horrible event.
Some argue that the same applies to AI takeover — that even if such an event caused mass death, full human extinction would remain unlikely. In the words of a participant of the Existential Risk Persuasion Tournament: “Perhaps the most common argument against AI extinction [made here]1 is that killing [close to all]2 humans is incredibly difficult.” Forecasters point out that there are large populations already living in remote and harsh locations, which could survive an AI-directed nuclear or biological strike. An AI might not go out of its way to target those populations because they’d pose no threat — at least as long as they didn’t start rebuilding human civilization.
But even if these people aren’t seen as a threat, they still have to deal with the side-effects of the AI’s actions. And the side-effects of AI taking control of the world are no joke.
A misaligned superintelligence with uncontested control of the world could grow its industrial base exponentially, with much shorter doubling times than we’re used to. You can think of it as a technological civilization in its own right. And a technological civilization isn’t confined to a blast radius, like a nuclear explosion, or limited by antibodies, like a pandemic. Its growth may run into limits — but those limits look like “there’s only so much matter and energy in the solar system,” and an AI pushing up against them would not be good for any human survivors.
Seen another way, a superintelligence is a powerful optimizer, pushing the world continually toward some state it considers most intrinsically preferable, and systematically overcoming any obstacles. If that preferred state (like most possible configurations of atoms) doesn’t involve humans, then in the end, human life will die out.
Nor would it find the conditions supporting human life to be the most instrumentally useful to its projects. Its long-term goals might be most effectively served by capturing all the sun’s energy, or covering the Earth’s surface with infrastructure (like datacenters, power plants, or factories) and scorching it with waste heat, or cooling our environment close to absolute zero for efficient computing, or taking the planets apart for raw materials. Our world would essentially become a technological artifact, and we can’t inhabit most possible technological artifacts. Humans are resilient, but not that resilient.
To be clear, this conclusion depends on some assumptions. People could survive superintelligence if we never build it; or if we successfully align it to our values; or if we control it so it never tries to take over; or if it tries but is stopped partway through; or maybe if it ends up caring just a little bit about us, perhaps as a result of some sort of deal.
But if a superintelligence succeeds at taking control, and cares only about something alien, then it will create a world too alien to contain biological life, let alone us or our descendants.
“Most common” here may have been meant in the context of the tournament or one of the groups. We don’t think it’s the main argument made in the wider discussion on AI extinction, but still want to address it. ↩︎
The original quote says “killing all but 5,000 humans,” because the tournament question defined extinction as meaning fewer than 5,000 survivors. ↩︎