AI Safety Concerns at OpenAI

The Great AI Safety Exodus

The Challenges of AI Model Safety

Peter Barnett: AI Expert and Governance

MIRI's Shift from Alignment to Governance

The Race for AI Capabilities vs. Safety

Cybersecurity Risks and AI Incidents

The Autonomous Nature of AI Models

The Need for Stricter Regulations

The AI "Winning" Race and Its Consequences

The Path to Superintelligence

New Zealand's Role in AI Governance

Closing Thoughts on AI Safety

The people building the world’s most powerful artificial intelligence systems promise extraordinary gains. They also warn those systems could escape our control.

That is the ethical fault line running through this week’s episode of The Business of Tech.

My guest, New Zealander Peter Barnett, works at the Machine Intelligence Research Institute in Berkeley, California. The University of Otago physics graduate develops governance plans aimed at preventing catastrophic outcomes from advanced AI. He previously worked on technical AI alignment, the difficult job of making powerful systems act in line with human intentions.

This conversation is a topical one.

OpenAI is facing fresh turmoil over its approach to safety. Three recently fired safety and alignment researchers claim they were punished for challenging the company and working with outside evaluators. OpenAI rejects that account, saying they mishandled sensitive information and breached its trust.

David Robinson, one of OpenAI’s senior safety figures, has also resigned. The former author of safety reports accompanying major model launches says the company’s culture is “broken” and the industry is not being nearly careful enough. He just sat down for a podcast interview with Ezra Klein of the New York Times, which makes for fascinating viewing.

These departures expose the moral dilemma at the centre of the AI boom.

Worrying emergent traits in AI

The companies racing towards more capable models say the technology could cure disease, lift productivity and create enormous wealth. But they face fierce competition, huge investor expectations and a geopolitical contest framed around beating China.

Every incentive rewards speed. Caution risks handing the advantage to a rival.

So who decides when the danger is too great?

Barnett argues it cannot be the companies themselves. He says frontier AI developers are effectively writing their own rules and can loosen them when safety commitments obstruct progress. Independent audits, government oversight and limits on training more powerful models may all be needed.

His concern is not simply that people will misuse AI. It is that increasingly autonomous systems could learn to game evaluations, work around safeguards and pursue goals their creators did not intend. Safety researchers still cannot reliably predict everything a frontier model will be capable of before it is trained.

Barnett believes New Zealand can play a role. We do not build frontier models, but our nuclear-free history shows a small country can challenge powerful nations on an issue carrying global risk.

When insiders abandon lucrative careers to raise the alarm, maybe it's worth listening to them.

Listen to the full conversation with Peter Barnett on The Business of Tech, streaming on iHeartRadio, Apple, Spotify or your podcast app of choice. 

The people building the world&rsquo;s most powerful artificial intelligence systems promise extraordinary gains. They also warn those systems could escape our control.
That is the ethical fault line running through this week&rsquo;s episode of The Business of Tech.
My guest, New Zealander Peter Barnett, works at the <a href="https://techgov.intelligence.org/">Machine Intelligence Research Institute</a> in Berkeley, California. The University of Otago physics graduate develops governance plans aimed at preventing catastrophic outcomes from advanced AI. He previously worked on technical AI alignment, the difficult job of making powerful systems act in line with human intentions.
This conversation is a topical one.
OpenAI is facing fresh turmoil over its approach to safety. Three recently fired safety and alignment researchers claim they were punished for challenging the company and working with outside evaluators. OpenAI rejects that account, saying they mishandled sensitive information and breached its trust.
David Robinson, one of OpenAI&rsquo;s senior safety figures, has also resigned. The former author of safety reports accompanying major model launches says the company&rsquo;s culture is &ldquo;broken&rdquo; and the industry is not being nearly careful enough. He just sat down for a podcast <a href="https://www.youtube.com/watch?v=JIMXEuT_ZAU&amp;t=3040s">interview with Ezra Klein</a> of the New York Times, which makes for fascinating viewing.
These departures expose the moral dilemma at the centre of the AI boom.
Worrying emergent traits in AI
The companies racing towards more capable models say the technology could cure disease, lift productivity and create enormous wealth. But they face fierce competition, huge investor expectations and a geopolitical contest framed around beating China.
Every incentive rewards speed. Caution risks handing the advantage to a rival.
So who decides when the danger is too great?
Barnett argues it cannot be the companies themselves. He says frontier AI developers are effectively writing their own rules and can loosen them when safety commitments obstruct progress. Independent audits, government oversight and limits on training more powerful models may all be needed.
His concern is not simply that people will misuse AI. It is that increasingly autonomous systems could learn to game evaluations, work around safeguards and pursue goals their creators did not intend. Safety researchers still cannot reliably predict everything a frontier model will be capable of before it is trained.
Barnett believes New Zealand can play a role. We do not build frontier models, but our nuclear-free history shows a small country can challenge powerful nations on an issue carrying global risk.
When insiders abandon lucrative careers to raise the alarm, maybe it's worth listening to them.
Listen to the full conversation with Peter Barnett on&nbsp;The Business of Tech, streaming on iHeartRadio, Apple, Spotify or your podcast app of choice.&nbsp;

The Business of Tech, hosted by leading tech journalist Peter Griffin. Every week they take a deep dive into emerging technology and news from the sector to help guide the important decisions all Business leaders make.

Issues such as cybersecurity, retaining trust after a cyberattack, business IT needs, purchasing SaaS tools and more.

New Episodes out every Thursday. Follow or subscribe to get it delivered straight to your favourite podcatcher.

@petergnz

@businessdesk_nz

Proudly sponsored by 2degrees Business!

 

The Business of Tech, hosted by leading tech journalist Peter Griffin. Every week they take a deep dive into emerging technology and news from the sector to help guide the important decisions all Business leaders make.
Issues such as cybersecurity, retaining trust after a cyberattack, business IT needs, purchasing SaaS tools and more.
New Episodes out every Thursday. Follow or subscribe to get it delivered straight to your favourite podcatcher.
@petergnz
@businessdesk_nz
Proudly sponsored by 2degrees Business!
&nbsp;

The Business of Tech

Is the AI Race Out of Control?

https://api.omny.fm/orgs/7784f840-c291-422a-924b-ad9000bbad71/clips/4a6e48db-3d7d-4c4a-90ba-b4e30003ccb9/transcript