Spotting Wildfires with AI

Published Aug 31, 2023, 4:05 AM

Sonia Kastner is the founder and CEO of Pano. Sonia’s problem is this: How do you use data and machine learning to mitigate the damage caused by climate change?

Pano mounts cameras on remote mountaintop towers, then sends images from the cameras to an AI model trained to spot wildfire smoke. The goal is to alert fire crews early, before the fire spreads.

Pushkin.

So in September of twenty twenty, there was lightning storms that started over six hundred fires over the span of a single day, and it caused a number of enormous mega fires and really overwhelmed the fire department, and smoke flooded over San Francisco and blanketed the sky, and all of us living in San Francisco woke up to a blood red, blade runner sky and the sun never rose that day.

This is Sonya Kessner. She spent a lot of her career managing supply chains for companies like Nest, which makes doorbells and thermostats, and Packs which makes fancy vape pins. But the year before that day the sky turned red, she had founded a company called pano Ai. The company's initial goal was to reduce the damage caused by wildfires.

After that day that the sky turned blood red, we talked to each other and decided we wanted to speed up, and we just got an outpouring of support from all of our friends and family, and everyone encouraged us go out and raise vunt your capital funding, go as quickly as you can. We need folks working on this problem. And we felt optimistic actually that this crisis would lead to urgency in the market, and we actually have seen that. So that's the silver lining of what was a very, very scary and bone chilling wake up call.

I'm Jacob Goldstein and this is What's Your Problem, the show where I talk to people who are trying to make technological progress. Pano, the company Sonya founded, mounts cameras on remote mountaintop towers, then sends panoramic images from those cameras to an AI model that's trained to spot smoke from wildfires. The goal is to alert fire cruise early before the fire spreads. So far, Panel's customers include utility companies and firefighting agencies in several states across the western US and also in Australia. I should mention that I talked to Sonya last month before the fire in Maui. Sonia start a Pano because she wanted to solve a problem that goes beyond wildfires. The problem is this, how do you use data to mitigate the damage caused by climate change?

I will say, even after the sky turned red, raising venture capital was not that easy. For the first year or two, the idea of using technology to adapt to climate change was still a very immature, nascent idea. There was starting to be more focus again on technologies to mitigate climate change, but when we would meet with VCS, we had to start with wait, tell me again. You're saying climate change is already here today. You're saying we should do something about it. You're saying we can do something about it. That's where we started the conversation.

Huh, that's interesting. So you're saying when you were trying to raise money when you started the company not that long ago, three four years ago, the idea of like, oh no, we live in a world where the climate already has changed, and we need to deal with that, and we need to build companies to deal with that. That was a novel idea.

It was often the first time they had heard that pitch. It was the meeting with me. So I had to start there. Climate change is not something in the far distant future, but climate change is here today. I think of myself as a realist, or more of a a pessimist or a realist than others. Where you know, I work in supply chain. We're very good thinking of what can go wrong if you're willing to face the harsh reality of what might be going wrong or what is going wrong? Then you can do something about it. So it's an optimistic take on pestimism, I guess. And so I thought there would be a slew of adaptation companies and I was actually shocked that there were not, And that to me felt like, Okay, there's a void here that needs to be filled.

So find of change causes lots of problems. How do you land on fires? How do you get to starting the company that you start?

You know, we actually at PANO our mission is to bring more data to bear to mitigate all types of climate driven natural disasters floods, hurricanes, moth slides, But starting with wildfires is a natural place to start when you live in California and you and your friends have experienced the devastation firsthand. Your friends have lost their homes and their children have been evacuated from schools, And there was also tremendous hunger from the fire fighting community for new technology. When we went to start researching the idea of bringing technology to wildfires, there was already a huge community raising a rallying cross we need more technology as a force multiplier to tackle wildfires, and they were listing out what they wanted. We want cameras, we want drones, we want satellites, we want AI, we want mobile software. Please send us some tools. And this is exactly what I knew how to build for my career and my colleagues who come from Cisco, from Apple, from NAS from Google. This is exactly what we know how to build. But we looked around the space and there were almost no vendors.

It's like a market calling out for suppliers.

It really was. Yeah, I don't think it happens that often. I mean for our first product, the actual detailed features for the product were written in a report from the California Public Utilities.

Basically saying, would someone build this and sell it to us?

Yes? Exactly, that is exactly what happened.

So do you start the company and build that product that the California whatever was asking for?

That is exactly what they We did. So the thing that's challenging about this business at PANO is we need to have all the capabilities of a Internet of things company like a Nest or a Fitbit. Because we design our own hardware, we manufacture in this factory back here, We design our own software, we design our own artificial intelligence algorithms. That's the same as any Internet of things, say consumer IoT gadget in your home. But on top of that, we also are a company that manufactures and deploys ruggedized equipment in remote locations, and so that makes us look more like a telecom company. We need those capabilities in house as well, because one of the things that the customers really want in our industry is a one stop shop. They want a company that just handles the whole thing.

So just tell me how, just tell me how the system works, what happens.

So at each tower location on a mountaintop, we deploy a PANO station which includes two ultra high definition security cameras, and these cameras are rotating three hundred and sixty degrees every minute, capturing ten frames of high resolution images, and we're uploading them to the cloud over cellular or a wired connection or starlink. And then the data goes to the cloud and it goes to two places. First, the images go through our AI algorithm, which is looking for signs of smoke, and whenever the AI thinks that's the smoke, it adds a bounding box, and those bounding boxes are then reviewed by analysts in our Pano Intelligence Center.

A bounding box just mean the AI basically draws a box around what it thinks is the smoke exactly.

That's exactly right, and so our Pano Intelligence Center will dismiss any false alerts, but when they see that there actually is smoke, that means it's time to trigger an alert. We arrange the panel stations so that incidents can be seen from two stations and we mark both. We have an algorithm that calculates bearing. We actually have a patent on this that allows it to be very accurate, and that creates allaw two longitude.

So just to be clear, when you say bearing, it's like there's a line from each camera and you can figure out where the lines cross, so you can know exactly the latitude and longitude of the site where there is smoke. Yes, because every spot can be seen by two different stations.

That's right. And this triangulation strategy was deployed in fire lookout towers for hundreds of years. They would use a dial and a string to draw the bearing manually in a lookout tower. So we just created the digital version of this. So we've done a human review, We've marked the fire we've created a latitude and longitude. We push out then automated notifications through text and email to all of the emergency managers who have been onboarded to the platform in that area, and they all get alerted to this incident within minutes of the fire starting, and it gives them location information and it gives them a video of the growing smoke.

And I presume it's a subscription. They pay you by the season or something.

Yeah, we do an annual subscription and PANO everything's included that subscription, and Panel maintains the equipment. We maintained the AI, the Panel Intelligence Center, et cetera. The customers are just getting fire intelligence as a service.

So tell me about developing the AI, Like, was there some off the shelf model that you could start with or what where did you start with the I? What did you have to do?

So we do use open source object detection models, like any company in modern computer vision would, but these are not specific to detecting smoke or fire. These are just models that are used to detect objects out of camera data. And then we need to train the model with images of smoke and not smoke, And it turns out that Detecting smoke is a hard computer vision problem. You know. When I first started the company, I mean, I'm a hardware manufacturing person. I thought the AI would be easy, you know, like I saw this Silicon Valley episode hot Dog not a hot Dog. I figured, you know, a couple of months, a couple of months of loading some data into you know, TensorFlow, boom boom, be done. You know, three years in we have a great model and it's still getting better every day. Smoke is a difficult thing to detect because a wildfire, smoke is very rare. It doesn't wildfires don't occur that often, but there are lots of things that look like smoke. There's cloud, there's fog, there's dust, there's barbecues.

What did you have to do to make it work? It was harder than you thought. Like how did you go from it not working to it working?

Well? I could tell you, but I'd have to kill you on that one. But I can't get into too much of the things we tried and what worked and what didn't work and how we how we got to our you know, end result. But you know, what I'll say is that you know the range tools have to do with the data you gather, how you labeled the data, the type of model you use, so certain model techniques, the type of not fire data that you gather as well.

You want to show the model lots of instances where there's dust blowing up from the ground and where there's fog in a way that looks kind of like smoke, and all of the things we can think of that are smoke.

Like, right, right, and you know, one of the keys to developing a great AI program, which we're still continuing to build out now. And actually we just hired a new VP of Engineering who had a five hundred person team at Meta that included both machine learning and software and he has a PhD in computer vision. He spent his entire career in this field. One of the first things he did when he joined us a few months ago was to say, we really need to invest in infrastructure, not just not just running the experiments and building new versions of the model, but an entire end to end pipeline that lets you run these experiments more efficiently, gather the learnings, compare the different results, and and then iterate and iterate faster and faster. And so actually a lot of companies in the in the AI space right now are shifting to more focus on just as much focus on the infrastructure around AI development as into the experiments themselves.

Is that why in Vidia stock is worth trillions of dollars all of a sudden trillion dollars. One of the reasons, Yeah, a strained at all by hardware.

We're not constrained by it, because if you're willing to pay, you can get as much as you need from the cloud providers. But it is, but it is extremely expensive. It is. It is one of our is definitely one of our highest R and D expenses is the cloud computed for running experiments and then actually on an ongoing basis. I mean, we're uploading trillions of pixels and and they have to all be processed every minute, and so one of our highest expenses is running the AI on all this data all the time.

In a minute. Lots of other ways that Pano might use data to mitigate the risk of disasters. Now back to the show. Earlier, you said something to the effect that, like, the big idea for the company was not about wildfires per se, but it was an idea that better data could help mitigate natural disasters.

Is that right, You just gave my elevator pitch for me, Thank you very much. Now, I say, what we heard from customers who work in disaster management is that there is a paucity of data at all phases of disaster response. And those phases are the real time response phase, which is probably what most of us think of when they think of disaster management, evacuation, search and rescue, fire containment, restoring power and internet. But there's three other phases. There's recovery, mitigation, which is determining how you're going to harden your system so that the disasters aren't as damaging the next time. And then preparedness, which is planning making sure you have enough emergency blankets, making sure you know your evacuation roots, and and you have shelters prepared. And all of those phases need more data to face this growing threat.

To be clear, you're not just talking about wildfires here, right. Is there a next kind of disaster you're thinking over, a next phase of disaster. You're thinking over both, like what do you want to do next?

So we are customers often started out as wildfire mitigation teams and then we're asked to add mudslides and flooding and other disasters to their remit because extreme heat, extreme cold, for example, our power utility customers, all of these disasters are incredibly disruptive to their power grid, and so we've been asked to explore our building situational awareness tools that can help them make better decisions both in the real time heat of the moment when they're trying to restore power, restore internet after a hurricane, for example, or after the fact, when they're trying to analyze their entire asset map and decide on how to deploy billions of dollars into hardening their system. The flood maps are one hundred years out of date. How are they going to decide which region of their territory should they bury power lines first or second. There's going to be trillions of dollars deployed over the next couple of decades to help humanity prepare for climate change, to help us harden our cities, harden our transportation infrastructure, harden our power grids. Where's the data to inform those that trillion dollars of spend? These customers are realizing that this data gap is a problem.

Somehow, As you were talking about about disaster management as a data problem. I was thinking of what you were saying earlier about supply chains and supply chains as sort of a worldview, and I couldn't quite nail the link. I couldn't quite articulate the connection. But can you do you feel a connection between those?

Actually you're getting at something which I didn't share. To your question of why I wanted to found this company, I had been in terms of thinking about getting into the field of disaster management, something I had been thinking about for many years because every time I would hear on the news about a disaster and I would envision the skill that it would take to go cope with this disaster. It actually reminded me a lot of supply chain.

Manufacturing in what way.

So when you're running a supply chain, well, nobody notices you. Everything just shows up on time, beautifully, and and it's it's invisible. So you need to make very very meticulous plans of exactly what you're going to build and all the necessary pieces that need to go into place to make sure that you can manufacture that product at the quantities you need on time and get it to where it needs to go. So you're planning meticulously and then all of your plans never goes planned. You you still all your well laid plans still result in fire drill, fire drill, fire drill, fire drill, just a disaster disaster every single day. I mean it's a super high adrenaline job. Where where there's a labor strike, there's a pandemic, there's a a you know, something's held up in customs, the supplier got confused and made the wrong color. Just disaster after disaster after disaster is what happens in supply chain, and you need to recover and think on your feet and figure out how to resolve that issue. And at the end of the day, things result in calm and stability and you save the day. And and you know, disaster management I think has a lot of similarities where emergency managers they spend the off season meticulously planning for how they're going to harden their system, like building fire brakes and safety zones, how they're going to be prepared, like communicating evacuation plans and rehearsing and having drills and ordering emergency blankets. And still when the fire comes, they need to react in real time and make snap spurt the moment decisions, and so I think I have coming from supply chain, I have a lot of empathy for our customers emergency management, and I can imagine how the more data that they can have, the better they can make their decisions. Because being in supply chain. Data is key. You know what, when I was leading supply chain organizations, my goal was to surface data on every on as far upstream and the supply chain as I could go, as early as possible. And if I could look, if I could look, you know, twelve weeks upstream into the supply chain and I saw, oh, they had a labor strike, I know there's going to be a problem, you know, coming at me twelve weeks from now. But when I have twelve weeks to react to it, it's much easier for me to solve that problem that if I only find out about that labor strike when that part just doesn't show up. Data is critical to running supply chain and emergency managers share the same thing. The more data they have, the more they can respond efficiently and safely. There are some similarities. You're right, It's good. There is something that ties it all together.

We'll be back in a minute with the lightning round, then back to the show, So we do a bunch of questions at the end of the show. I didn't realize when I wrote those questions for this interview how much we talk about supply chains in the main part of the interview. I wrote a bunch of supply chain lightning around questions for you. Oh that's okay, Okay, let's do them anyways.

Yeah, my favorite talking that's great.

What do you love about supply chains?

I think what I love about the supply chain is both the planning part and the and the adrenaline part of having to to have the diving catches in the moment.

Diving catch is a great metaphor. Who doesn't love making the diving catch? Yeah? What was the difference between managing the supply chain for high tech vight pens at packs and managing the supply chain for fancy thermostats or cameras at nest?

Those supply chains are pretty similar, to be honest. The reason we get so many awesome coal gadgets so quickly into the market is that the consumer electronics industry has built a really mature supply chain that's made up of building blocks that can be rearranged into very different products. So the building blocks of both of those products are surface mount technology, which is how you make the printed circuit boards, and then the other building blocks are mechanical components that go around those electronics, so plastic injection molded parts, metal formed parts. Our supply chain here PANO is radically different.

However, it's interesting to think of when you describe it, how many things we get are just circuit boards with plastic wrapped around them, right, So many things yep yep.

By the way, I don't want it to seem that simple, because my husband is a leads hardware engineering and at a startup, and he would say, it's not that simple.

Fair. So, now you work in the world of natural disasters, and I'm curious, are you a prepper?

Oh, that is a great question. You know, we do have our earthquake kit. We do have finally filled up our water jugs. I said to my husband, like, it will just be too embarrassing if I founded an emergency management company and then we die because we didn't have our jugs of water in the earthquake.

Vanity and shame as a motivator to not die works right, Why.

Not Honestly, it's really emergency preparedness is really important.

If everything goes well, what problem will you be trying to solve in five years.

I think the response phase is low hanging fruit, and we have a tool that really helps the response phase. But mitigation is just as important, and we need to We need to think through through how many more helicopters do we need, Where do we need to bury power lines, what fuel breaks do we need to cut? What are ways that we can harden our cities and the infrastructure against wildfires? But I also would like to help solve problems related to other disasters, like how do we restore power and internet faster after a disaster? Right? And can data help in that solution? So I'd really love to be working on that problem, and I'd also really love to be helping inform policy makers around rebuilding efforts. Build back better is an expression used in Washington. You can't build back better if you don't have data.

Sonya Kassner is the founder and CEO of PANO. Today's show was produced by Gabriel Hunter Chang and Edith Russolo. It was edited by Sarah Nix and engineered by Amanda k Wong. You can email us at problem at Pushkin dot fm. You can find me on Twitter at Jacob Goldstein. I'm Jacob Goldstein and we'll be back next week with another episode of What's Your Problem.

What's Your Problem?

Every week on What’s Your Problem, entrepreneurs and engineers talk about the future they’re trying  
Social links
Follow podcast
Recent clips
Browse 139 clip(s)