We like to mix things up on a Saturday morning. Bit of this bit of that. 

So, in the interest of variety, having recently mused over the merits of different apple varieties and the joys of my family’s fabric shaver, I’ve got a different question for you this morning: 

How do you think it all ends? 

Heavy stuff for a Saturday morning, I know. But entertain it for a second. Not just how it ends for you or me, how it ends for all of us. Humans. People. The species. Will there ever be a day when homosapiens become no-mosapiens? 

Probably. I reckon so. Seems likelier than not, right? Depending on how you do the sums, they reckon the intensity of the sun will make it impossible for Earth to hold onto liquid oceans in a billion years or so. It’s a very literal deadline. If we are to survive, presumably well before the point at which the last ocean disappears, we’d have to have expanded our reach further throughout the cosmos. 

And while, sure, maybe an external mass extinction event outside of our control will wipe us out well before then —a meteor to make the dinosaurs’ one look like a mere pebble plopping in a pond— it’s hard not to escape the sense that for all of our brilliance, the end of our species is perhaps just as likely to be brought about with that most human of phenomena: hubris.   

An AI researcher named Evan Hubinger quit Anthropic, one of the world’s frontier AI companies this week in dramatic fashion. He issued a public statement saying the top companies were being too reckless, and in which he warned there is a greater than 10% chance the technology could “kill all humans” within the next decade. 

How? He was asked. 

Who knows? He said. Could be a novel bioweapon. Could be a huge strike on globally important infrastructure. The problem is we’d have little notice, and potentially even less capacity to stop it once it began. 

For anyone who geeks out on this stuff, it was a frightening conclusion to a pretty crazy couple of months. Only last week, with the release of independent reports into the incident, was the public properly appraised of the true scale of OpenAI’s recent Hugging Face hack, the thousands of autonomous AI agents which worked together, independently communicated with each other, roamed the internet and hacked an independent platform outside of human control. 

There's a term you hear a lot with AI now: recursive self-improvement. In simple terms it means the big companies are increasingly using AI to build AI. They can theoretically hit go and then leave it to make itself. But if we are already at a point where their systems are going rogue, where might the recursively improved systems be a few years from now? 

Hubinger’s comments appeared to divide the AI community. Some fellow researchers endorsed his existential concerns. Elon Musk dismissed them. Others suggested that going public and putting a number and a timeline on existential risk was exactly the sort of cynical play a big AI company might make if it was trying to impress investors about the strength of its technology. 

Personally, as much as I find the technology absolutely fascinating, I don’t feel sufficiently qualified to make a judgement. Besides, is a ten percent risk actually any worse than a one percent risk? Or a one-in-one-thousand risk? Still seems pretty risky, given the stakes! 

So all I’d say is in the interests of our continued existence, of seeing our children grow up, of seeing a second Auckland Harbour Crossing completed in the year 2084, I think it’s best we err on the side of imagining what might happen if we’re wrong. 

If we end up catastrophising the technology and we’re wrong, it is all just a marketing play, maybe we’ll inadvertently mint a few more Silicon Valley billionaires. 

But if we dismiss the warnings, the risk, refuse to slow down or regulate or control the technology’s development in some kind of meaningful way, and we’re wrong… 

Well… I’m tempted to put a tweak on that brilliant quote. 

The last thing we shall hear before the universal destruction is an expert on Twitter saying it's impossible.