The people building the world’s most powerful artificial intelligence systems promise extraordinary gains. They also warn those systems could escape our control.

That is the ethical fault line running through this week’s episode of The Business of Tech.

My guest, New Zealander Peter Barnett, works at the Machine Intelligence Research Institute in Berkeley, California. The University of Otago physics graduate develops governance plans aimed at preventing catastrophic outcomes from advanced AI. He previously worked on technical AI alignment, the difficult job of making powerful systems act in line with human intentions.

This conversation is a topical one.

OpenAI is facing fresh turmoil over its approach to safety. Three recently fired safety and alignment researchers claim they were punished for challenging the company and working with outside evaluators. OpenAI rejects that account, saying they mishandled sensitive information and breached its trust.

David Robinson, one of OpenAI’s senior safety figures, has also resigned. The former author of safety reports accompanying major model launches says the company’s culture is “broken” and the industry is not being nearly careful enough. He just sat down for a podcast interview with Ezra Klein of the New York Times, which makes for fascinating viewing.

These departures expose the moral dilemma at the centre of the AI boom.

Worrying emergent traits in AI

The companies racing towards more capable models say the technology could cure disease, lift productivity and create enormous wealth. But they face fierce competition, huge investor expectations and a geopolitical contest framed around beating China.

Every incentive rewards speed. Caution risks handing the advantage to a rival.

So who decides when the danger is too great?

Barnett argues it cannot be the companies themselves. He says frontier AI developers are effectively writing their own rules and can loosen them when safety commitments obstruct progress. Independent audits, government oversight and limits on training more powerful models may all be needed.

His concern is not simply that people will misuse AI. It is that increasingly autonomous systems could learn to game evaluations, work around safeguards and pursue goals their creators did not intend. Safety researchers still cannot reliably predict everything a frontier model will be capable of before it is trained.

Barnett believes New Zealand can play a role. We do not build frontier models, but our nuclear-free history shows a small country can challenge powerful nations on an issue carrying global risk.

When insiders abandon lucrative careers to raise the alarm, maybe it's worth listening to them.

Listen to the full conversation with Peter Barnett on The Business of Tech, streaming on iHeartRadio, Apple, Spotify or your podcast app of choice.