Jeff Adams as a computer scientist. In two thousand and eleven, he was at a really tiny startup, just thirteen employees, called the App. They were working on technology that could transcribe voicemail into text messages. And then one day Jeff's fairy godmother showed up in the form of executives from Amazon. We went to a show and some people from Amazon came and talked to us, and they might be interested in possibly even acquiring us. And I thought, that's absurd. What could Amazon possibly be interested in speech for? Maybe they want you to be able to, you know, call up on the phone and order a book. Jeff figured the talks wouldn't amount too much, but over the course of the next few months they became serious. Naturally, he wanted to know why Amazon was interested. They said, you know, don't ask, it's our business. And I thought, well, it's gonna be my business too, and you might want to know whether whether we can do it um, But they wouldn't. They wouldn't say anything. They said, I'm sorry, were not at liberty to discuss any of this. There was one hint that Amazon would give, so one of the things they did let us know was that the guy who was kind of running this was Greg Hart, who we knew was the right hand person of Jeff Bezos. So they go to all the meetings with them. They are like the confidante, the consilior or whatever. Uh. And so we knew, okay, something this does have visibility at the highest levels, and Bezos must be behind it in in some form. As the deal was being finalized, Jeff's team traveled to Florence, Italy, to a computer speech conference along with several Amazon managers. They even stayed in a villa together, but the Amazon people didn't want to be seen with the voice transcription people during the conference. From day to day in the conference, we had to let we we I rented a van and we drove down into town and to the conference together. I had to let them out around the block, around the corner. It's like being a teenager and trying to hide the person you're dating from your parents. First of all, they didn't want anyone to know that they were from Amazon. They didn't want anyone to see them with us, so we had to avoid them at the conference. We would like, you know, look at each other and kind of smile from across the room or across the courtyard or whatever. But we couldn't sit together, we couldn't talk together, or whatever. That fall the deal was done. Amazon buys the speech company, and Jeff is still living in mystery. We had been employees of Amazon for like a week, but they had still not told us any anything. They said, no, we have to tell you in a closed room. So we all came out to Seattle. We all got in the room together. They closed the door, locked the door, put paper over the window in the door, closed the exterior windows. Uh. It was very secret, very mysterious, and hush, hush. Jeff and his team leaned forward. This was the moment they said, imagine something the size of a coke can that's sitting on your able and we're going to sell this for uh and people will be able to talk to it. And we thought, you know, we can't do this. The technology isn't there yet, it doesn't exist yet. And of course they did do it by inventing the technology, because the product they're talking about becomes Alexa, Amazon's AI virtual assistant, and that twenty co can is the Echo smart speaker. Though it ends up selling for more than twenty Alexa is one of Amazon's first forays into artificial intelligence, and it would bring Amazon into people's living rooms, further weaving it into their lives. You're listening to Foundering. I'm your host, Brad Stone. In this episode, we're going to tell the story of the creation of Alexa. How Jeff Bezos almost singlehandedly conceived the idea for a voice activated computer and drove the device's creation. Alexa helped to solidify Amazon's image as an innovator and in the process challenge conventional notions of privacy. Will tell you more after a quick break. In the last episode, we talked about how Amazon survived the dot com bust, how Jeff Bezos said to colleagues, the only way out of this is to invent her way out. During those next few years, Bezos launched Amazon Prime and the Kindle. But there's another innovation from this era, one that generates huge profits for Amazon and sets the stage for the creation of its pioneering voice assistant. It's Amazon's cloud service called AWS, or Amazon Web Services. It's the most difficult of Bezos's inventions for lay people to understand, because as much as it may already seem like Amazon is everywhere a massive store, a network of warehouses, and the fleet of delivery people. There's one more way in which Amazon is ubiquitous indispensable. Amazon is literally holding up much of the Internet today. AWS is a highly profitable, fifty billion dollar annual business for Amazon. When you watch Netflix, you're often streaming video from Amazon. When you send a Snapchat, you're using Amazon servers. By two thousand and ten, Bezos was asking everyone in the company, what are you doing for AWS? You wanted Amazon to deepen and exploit the early lead it already had in running massive, cutting edge data centers. Around this time, he was at lunch with Greg Hart, the guy described as his consiliary, and the conversation turned how Google was starting to let people search just by talking into their smartphones. Here's Greg and and I said, look at how convenient is And I just did a Google voice search on pizza and your me um. And you know, it's just so much faster than actually typing, you know, with your fingers or your thumbs pizza and your me on your phone waiting for results to come back. Now, Bezos has long been a believer in voice. Here he is all the way back in the year two thousand talking to Charlie Rose, I believe that for mobile commerce, the thing that's going to be the biggest part of that is voice, and I think there's a lot that can happen in the In the short term, it'll be kind of a stilted special purpose language for talking to Amazon dot Com. But in the long term it could even be natural language processing. I think that is a real mind bender. Now. A decade later, it seemed like this early prediction might come true. A few weeks after their fateful lunch, Bazo sent an email to Greg that read, we should build a twenty device with its brains in the cloud that's completely controlled by your voice. The central insight here was that the product could be inexpensive because it's brains resided in Amazon's data centers and could be constantly improved, So if you buy it, the product would be upgrading itself and you wouldn't even know it. A few weeks later, Greg started recruiting people inside the company for the project. He proceeded with total secrecy. The only thing he told colleagues was that it had the potential to be bigger than the Kindle, and at the time that seemed like laughable that that we could create something that could be bigger than Kindle, because Kindle, you know, at that time was now for years into its life, dominating the e reader and e ink market and was changing the way that people thought about meeting. Next, Greg acquired YAP, the speech recognition company where Jeff Adams worked. That team didn't think he knew what he was doing either. Amazon wanted people to be able to talk to a device across a room and haven't understand them. That's called far field speech recognition, and the technology for it didn't exist. Here's Jeff Adams again. The big problem is speech recognition at the time really relied on a close talking microphone. You needed to capture the speech close to the person's mouth. And they were talking about, Oh, I'm going to be in the garage and this thing is gonna coach me through changing my car's oil, and it's gonna be over on the other side of the garage, and I'm gonna be shouting things to it, and it's gonna be shouting instructions back or whatever. And we thought, you can't do that. There's there's too many reflective surfaces in the room. They're gonna mess up the audio. Basically, if you're shouting across the room, that introduces lots of echo and the computer gets confused. So I realized that this far field issue was going to be a problem. I didn't want to say anything in in front of the group. I didn't wanna, you know, um, appear unsupportive. But afterwards I tracked down Greg Hart in the hallway and I said, Greg, we're excited about this, but I think you should know that what you want to do, the technology isn't there yet. It doesn't exist yet. We don't we we don't have that technology to solve this far field speech problem. And he was unflapped. He said, uh, I appreciate that, thank you for telling me, But solve it. We are an Amazon, We've got resources, hire as many people as you need, to take as long as you want, but you know, solve the problem. The team gave itself the code name Doppler, as in the Doppler effect, which describes the way a sound wave moves with respect to a listener. They hotly debated everything from what to name the device, to what it should do, to how to market that to the public. At first, they met with Bezos once a month. He would get deeply involved in the technology. UM, not just the business or the product, but deeply involved in the technology. He I would say, he stayed very close to the project throughout. As a project progressed, those meetings would increase in frequency UM and UH. And by the end we were meeting with him, you know, leading up to launch, I would say, there probably wasn't a day that went by that we didn't have at least one meeting with Jeff UM. And you know, sometimes there were multiple meetings with Jeff in a single day. UM. That is both a blessing and a curse. The blessing came in the form of money. The team could spend whatever they needed to break through the technical obstacles. This is a big project that Jeff is personally invested in. It was his idea originally. He can be very demanding. UM, he can push teams UM. But at the same time, like being able to go through that experience and have his brain on your side is an immensely powerful opportunity and experience if you let it, like if you get really frustrated by it. And at times some of the people on my product team would say, it feels like Jeff is the product manager for Alexa. Bezo set the vision for Alexa. He wanted it to be the Star Trek computer. He pictured a versatile, conversational machine that could respond to any question. You should be able to sit in your living room and ask Alexa anything. It sounds simple, but it's not because they needed a lot of data to train the AI algorithms. For example, for Alexa to tell you the weather, they would have to understand the phrasing of your question and your dialect from across a noisy room, and sort through databases for the right answer. This would take Alexa a step further than Syrior Google Voice, which only worked when you spoke directly into your phone. But the Alexa team just didn't have the data to get the AI smart. The early prototypes worked so badly that even Amazon employees didn't really want to test them. Bassos was getting impatient. He actually walked out of a number of internal meetings and frustration. Then Greg and his colleague struck upon an idea they called Project Amped. Basically, rents houses or apartments in cities all over the US, and we would put devices in those apartments. They were all camouflaged, and they were not just Amazon devices. There were other companies devices there as well, UM, some of which were visible, some of which were not visible. And characteristic of all Amazon projects, secrecy was the priority. Employees were careful to conceal Amazon's identity when they set up the room, and that was all about obfuscation, you know, we didn't want people to understand what company we were working for. UM, and so you could see an xbox, you could see this, you could see that UM. And then we would have all these Alexa devices hidden throughout the room or echo devices, prototype devices, and then they brought in testers, thousands of people paid an hourly wage, coming in at all hours of the day and days of the week to train the machines. Then we would bring in we would recruit participants, have people with different accents, you know, male, female, different ages who would come in and we would ask them to do a mixture of reading scripted things and then also talking in a much more off the cuff all the cart fashion to ask for things that people would be we hoped asking Alexa when we launched. Amazon conducted the data gathering effort with such stealth that neighbors started to get suspicious. House that we rented in Boston that the neighbors thought that because we had a lot of cars showing up and individuals getting out on their own and coming in and spending I think maybe an hour, and so there's a lot of sort of you know, transient in and out traffic, and neighbors thought that maybe there was you know, a drug running ring or something else going on, and so the police actually showed up despite the attention from police. Bezos loved the inventiveness of the data gathering program. When we first took AMPED to him, his response was effectively like, now you're talking, like let's do this, um, Like tell me if you want more money, Greg says. The project AMPED ran in thirteen cities and included over ten people. The devices were placed all over the room, and so we were trying to capture a massive amount of acoustic data about, you know, how noise performs in a room in all kinds of different rooms, in bedrooms and living rooms, in kitchens, in bathrooms. The result Amazon basically solved the far field voice problem. It took six months for this company to solve a problem that had stumped speech scientists for decades. But since the project was top secret, they weren't able to tell anyone, not even the speech science community, about their big technological breakthrough. Here's Ahmed Boozid. He had been working in voice technology for almost twenty years when Amazon tried to recruit him in early two fifteen. I turned him down. I said, you're not showing me anything. You have never done anything in voice, and you're telling me that you are going to any uly to to go from DC and go to Seattle and they'll let you. Just trust us. I know, no, I don't think so. I mean, I've seen the Kindle and it's fine, it's okay, but Kindall is like, you know, it's a toy compared to what you guys are trying to do. So anyway, I said no twice. Eventually they flew Ahmed out to Amazon headquarters to see Alexa in person. So it was, you know, on the table, and so the first thing I did is I asked it for for music, and he plays some Miles Davis and it did I'm like, okay, cool, And then I suppos I said, you know what time is it? Man stopping and told me the time, and then it continued playing Miles datas, which was great. Interacting with Alexa was almost a moving moment for Ahmed. He felt like decades of scientific research had been realized in this little device, and I, you know, I was like, Okay, this is amazing, This is amazing. This is clearly an important moment in technology. And this is what I've been doing all my life, right, solving the problem problems. See, you know, we people who are in the speech were are always saying this expression, you know, speeches around the corner. We've been saying it's since the mid nineties, species around the corner, because we do believe that if you do speech well, a lot of stuff becomes easy to do for people. Ahmed was particularly impressed by two innovations that Amazon made. One that they solved the far field speech issue and secondly a let so it was fast to speed right. The fact that it's just I was, I was just amazed, Like, how the hell does come back within within like two seconds? I mean, if I if I type, you know, if I want to launch a page on my browser, or it takes like sometimes like three four or five seconds. Right, How is how the hell is this thing doing all of these things, getting my voice going to the cloud, processing it, coming back talking back within two seconds. It's like a magic even for someone like me who was been like you should be jaded by by by then, Right, okay, I know exactly how it's happened. I was astonished. Right, how the hell did they do that? We'll be right back. I want to take a moment to address Amazon's obsessive secrecy. Many companies are secretive, but tech companies today have the eyes of the world on them, so they are taking corporate secrecy to a whole other level. Like that story. Jeff Adams, the speech scientist, told that the Amazon executives were so secretive when the acquired a company that they wouldn't even be seen together at a conference. That's typical of Amazon. This intense drive for secrecy comes straight from Jeff Bezos. He wants to tightly control the messaging around Amazon's new products. The idea is that complete secrecy pays off with a surprising, almost magical reveal once the product is launched. That man keeping Alexa under wraps until launch, preserving the details of how it actually worked, and choosing the perfect voice here's Greg Hart, how do you define the characteristics that you want the voice to have? And so there was sort of a brief that we wrote up about the qualities you wanted the person to be knowledgeable. We very quickly early on, decided the first voice would be female. We knew there would be additional voices, but we felt that the first voice should be female in because yeah, so that's the logical question why In part because the we knew that the device would be in the kitchen, and we felt that a female voice would be more open and inviting and warm than a male voice would be in that environment, and more appropriate in that environment. Not because of any sexist things, but just because of the fact that we knew, um that we knew the way that, um, that's the right way to say this. We had seen evidence that people respond differently to male computer voices than to female computer voices, and they respond more positively to female computer voices, and we wanted because the device was in the home, we wanted it to be a device that everybody would respond positively to. Putting a female voice in the kitchen, so to speak, would turn out to be a somewhat controversial choice, and Amazon isn't the only company to do this. Serie. Google Voice and Microsoft Cortana are all women by default, So Amazon moved forward in their search for a voice. They contracted with the same studio that had developed the voice of Sirie for Apple. Okay, I set up your meeting with David tomorrow. Shall I schedule it? Who is voiced by Susan Bennett, a career voice actress. Hi, my name is Susan Bennett, and I have a voice actor and the original voice of Sirie. So to find their Alexa, the studio had half a dozen female voice actresses read for hours. They read entire books and random articles, and finally Greg Hart and Jeff Bezos picked one woman. Her identity hasn't been revealed in all this time. It's amazing to me as an Amazon reporter that Amazon has still been able to keep the voice of Alexa a secret. In I started canvassing voice over actors, asking if they knew the identity of Alexa. No one knew. Finally, I got a tip from someone who had worked with the studio that Amazon contracted with, and they said that Alexa was voiced by an actress and singer from Boulder, Colorado named Nina Raleigh reached out to Raleigh numerous times, she wouldn't confirm that she's the voice of Alexa, but she didn't deny it either. Eventually I confirmed with enough people that I feel confident she's the one. She's Alexa. Here's a clip from her website and add that she did for Time Warner Cable. Thanks for choosing Time Warner. Now that you've ordered your installation, let's set up a time to get a crew over to your place. And here's Alexa. Time Warner Cable, also simply known as Time Warner, was an American cable television company. It was ranked the second largest cable company in the United States. But imagine what it might be like to be Nina Raleigh. Your voice is piped into millions of homes every day, each new iteration of Alexa, each update for Amazon, you have to record something new, Like when Amazon Fresh released a product called the single Burger. She had to be available single Cow Burger, a beef burger made with meat from just a single cow. She's chained to Alexa. It sounds like a life of obscurity and loneliness. Compare that to Siri and Susan Bennett. She's spoken on CNN and countless talk shows. She's even been able to use her fame as the voice of Sirie as leverage for many more opportunities. Greg saw the burden that Amazon secrecy put on the human behind the voice of Alexa. I never met her um and I don't even know that I might have. I'm not have spoken with her once, but I never met her. And it would be interesting to be in her shoes now, because it's on the one hand, it's it's incredible that this thing that you contributed to is now so ubiquitous. On the other hand, I would think it would be, you know, maybe a little bit unsettling. Amazon introduced the Echo in November two, fourteen were re strived from Amazon. I didn't know what it was. With a YouTube video, Alexa play rock music, rock music, alexis stop. We want to try Alexa, what time is it? The time is three. You actually don't have to yell at it. Okay. It uses far Fueld technology so it can hear you from anywhere in the room, So I can just hear you anywhere. Yes, that promise was enough. A responsive computer that can tell the time or the weather, play music, and answer questions from across the room. Tens of thousands of people joined a waiting list to receive a device. Here's Bezos at the Recode Technology conference in two thousand sixteen saying that the future is Alexa. He's speaking to Walt Mossburg. But it has been a dream ever since, you know, people started, you know, in the early days of science fiction to have a computer that you can talk to. So are you deeply committed to this becoming a huge part of your business and what you Absolutely We've been working on it. You know, we worked on it. We have more than a thousand people dedicated us to Alexa in the Echo ecosystem, and it's a and there's so much more to come. Baz also is deploying his playbook. For experiments that produced promising sparks, he poured gasoline on them. Amazon ramped up hiring the Alexa team, balloon to ten thou employees, and Bezos paider on ten million dollars for the company's first ever Super Bowl at starring Alec Baldwin and Missy Elliott. Alex to stop. How you do that? It's my Amazon Echo like extreme music, order things and watch this. I like to turn on the lights. Wow. Inside the company, employee has noticed that he seemed to take real joy in the work here's Ahmed Jeff, you know, the CEO and the guy in charge, uh was I would say he was obsessed with Alexa. How many he had this saying that I am happiest. He used to say this, I'm happy is when I'm working on Alexa UM and so he you know, it was his favorite time of the day. Used to meet with ALEX a team and work and work on ALEX. So he was directly involved in the early days. Basis was frequently asked about privacy. At the two thousand sixteen Code Conference, he promised to be a good steward of the sensitive personal data that Alexa was sure to pick up. This is going to get much deeper into our lives. So so what are what are the privacy I think that if you take the totality of you know, privacy UM and our ability to store large amounts of information to use it in ways that customers actually do want us to use it. So there are benefits. And I think one of the things that you have to do is when you collect and store data, you have to be clear about what you're doing. You have to and not just you know, subsection seventeen, paragraph three clearly as you can see at our privacy policy we were allowed to do that. Would you have to figure out ways to be kind of obviously clear? He stood on stage and promised that the privacy policy would be obviously clear. What he wasn't saying was that Alexa was so smart in part because there were real life humans listening through the machines, helping to craft many of Alexa's answers and to fix its errors by listening to what a subset of users said. The company kept Alexa owners in the dark about how this aspect of their devices worked. Ruthie Hope Slattess was one of those people being paid to listen. Several years ago, she saw an ad from a temp agency seeking someone with an English or journalism degree. They offered twelve dollars an hour to transcribe audio recordings. There was a kind of a generic add on prex list. I don't think it mentioned Amazon at all. She applied, She passed US curity check and a grammar test, and then she was let in on her task. She would listen to conversations picked up by the echoes microphones and type them up and then feed the information back in the Amazon system. She said it seemed like a good job. At first, we heard the customer using it in you know, ordering flour and asking what time it was and asking to be told to joke and so forth, and it seemed pretty cool. I mean a little invasive, for sure, but but pretty cool overall. Ruthie was there to make Alexa's responses seem more nuanced, more intuitive, more human. This was right around the time Amazon started selling the Echo to customers on a limited basis. She assumed at first that the folks she was listening to had signed up to help improve Amazon speech recognition software, and knew that someone might be listening. Who were all of these all of these many many voices and people were they? Various people work for Amazon who agreed to bring this home where they you know? But then if they were, would they really be ordering sex toys, you know, um and talking dirty to it and all of the grotesque things that we occasionally heard. Almost immediately, it was clear to Ruthie that people liked a toy with Alexa in ways that Amazon maybe didn't intend. What can we ask her? How how will she respond if we abuse her, if we talked dirty to her, if we asked her to marry us? If we you know, that sort of thing. Most of the sexual stuff seemed innocuous enough. Percent of it wasn't disturbing. You know, it was sometimes humorous and sometimes you know, kind of silly or weird or entertaining even you know, like somebody trying to figure out what kind of dildo toward her or something like that, you know, but sometimes it did get disturbing if it was a man who was talking to her, like he would talk in an abusive way to a woman, Like you could hear it in his voice, And she wondered if men were being so rude in part because Alexa was female, and being a woman, even an ai woman, meant that Alexa would be on the receiving end of misogyny. Even children did this. There were a lot of kids who would talk abusively to her, and it felt as though they were exercising some sort of anger that they had towards their parents or their teacher or something like that. There was a lot of psychology that I thought was fascinating. Ruthie felt like the way people spoke to Alexa was so private, so intimate. They would never ever speak like this if they knew someone was listening. There in their private home. They think that no one will ever hear the words that are coming out of their mouth. Ever, they do know that they're talking to a robot, but at the same time they're speaking to a row that as those essentient being. Ruthie's job was a secret. It wouldn't become public until years later that people like her were helping to improve the software by analyzing real Alexa recordings. My colleagues first interviewed Ruthie in two thousand nineteen, and after we published a story describing what she and her colleagues did, Amazon acknowledged the listening program. Technically, Alexa's terms of use gave Amazon wide latitude to duke basically whatever it wanted with the recordings, as long as the company was using that audio to improve the software. But many customers fell duped. They thought that when they spoke to their Alexa device, they were only dealing with clever software, and Bezos had gone back on his word his precise promise to be transparent with how he used as customers data. It seemed that while he highly prioritized Amazon's corporate privacy it's secrecy, he was being careless with the privacy of his customers, and in two thousand eight teen, Alexa had a huge privacy mishap. Here's my colleague Prea and Nod describing the infamous incident. What happened there is a family in Portland's said that their echo randomly sent recorded conversations to a contact of their's, a contact of the man in the houses. And what happened at the time was Amazon said this is how it worked. They said that Alexa interpreted something those people were saying in their background conversation as Alexa, so it woke up. Then it misunderstood a different phrase in their background conversation as send that message. It understood a different phrase in their background conversation. After that, the device asked to whom and thought that they responded with a name, So it ended up sending these conversations to one of their contacts through this crazy chain of events. It's like a horror movie. After the couple's friend reached out to them and said, hey, I think you're Alexa has been hacked, it turned out they weren't hacked. This was Alexa mishearing. But at the end of the day, this kind of scenario where Alexa could misinterpret a series of things people were saying in their own home, interpret them as requests, and send these people's recordings to someone else. If this kind of unlikely scenario can happen, and did happen, then who's to say it can't happen again? What else can Alexa mix up? Even so, people were undeterred. Over the next few years, Amazon would sell tens of millions of Alexa devices and inspire imitators from Google and Apple. It's popularity cemented bezos notion of himself as an inventor and the general public's perception of Amazon as an innovator. But Alexa hasn't quite met the goals its creators had for it. Most people only use it as a kitchen timer, a music player, a source for the occasional weather report. It's never been conversational in the way Bezos imagined no one would call at the Star Trek computer, but that hasn't stopped consumers and estimated one in three US households currently have a smart speaker. In the years since Alexa launched, people have invited more items that integrate AI assistance into their homes. That includes stuff like nest thermostats, ring cameras, and Sono speakers. Whenever a story about an Alexa privacy breach gets published, it makes a big splash, but it hardly ever changes anything. Because Alexa marked a turning point, It was another doorway into a more surveiled world and placed Amazon alongside other big tech companies like Google and Facebook at the hot controversial center of the battle over privacy. Finally, Alexa turned to Amazon into a company its customers interacted with almost every single day, not just once a week or a few times a month when they wanted to buy something online. And there was another way Bezos would accomplish the same goal by producing his own TV shows and movies. Bezos is risky expansion into hollyw would that's next time on Foundering The Amazon story. Foundering is hosted by me brad Stone. Sean wen Is. Our executive producer. Pria Nad and Matt Day contributed reporting to this episode. Raymondo is our audio engineer, Molly Nugent as our associate producer, Mark Million and Manner May Robin a Jello and Molly shoots our story editors. Francesca Levi is the head of Bloomberg Podcast. Be sure to subscribe and if you like our show, leave a review. Most importantly, tell your friends see you next time.

Jeff Adams as a computer scientist. In two thousand and eleven, he was at a really tiny startup, just thirteen employees, called the App. They were working on technology that could transcribe voicemail into text messages. And then one day Jeff's fairy godmother showed up in the form of executives from Amazon. We went to a show and some people from Amazon came and talked to us, and they might be interested in possibly even acquiring us. And I thought, that's absurd. What could Amazon possibly be interested in speech for? Maybe they want you to be able to, you know, call up on the phone and order a book. Jeff figured the talks wouldn't amount too much, but over the course of the next few months they became serious. Naturally, he wanted to know why Amazon was interested. They said, you know, don't ask, it's our business. And I thought, well, it's gonna be my business too, and you might want to know whether whether we can do it um, But they wouldn't. They wouldn't say anything. They said, I'm sorry, were not at liberty to discuss any of this. There was one hint that Amazon would give, so one of the things they did let us know was that the guy who was kind of running this was Greg Hart, who we knew was the right hand person of Jeff Bezos. So they go to all the meetings with them. They are like the confidante, the consilior or whatever. Uh. And so we knew, okay, something this does have visibility at the highest levels, and Bezos must be behind it in in some form. As the deal was being finalized, Jeff's team traveled to Florence, Italy, to a computer speech conference along with several Amazon managers. They even stayed in a villa together, but the Amazon people didn't want to be seen with the voice transcription people during the conference. From day to day in the conference, we had to let we we I rented a van and we drove down into town and to the conference together. I had to let them out around the block, around the corner. It's like being a teenager and trying to hide the person you're dating from your parents. First of all, they didn't want anyone to know that they were from Amazon. They didn't want anyone to see them with us, so we had to avoid them at the conference. We would like, you know, look at each other and kind of smile from across the room or across the courtyard or whatever. But we couldn't sit together, we couldn't talk together, or whatever. That fall the deal was done. Amazon buys the speech company, and Jeff is still living in mystery. We had been employees of Amazon for like a week, but they had still not told us any anything. They said, no, we have to tell you in a closed room. So we all came out to Seattle. We all got in the room together. They closed the door, locked the door, put paper over the window in the door, closed the exterior windows. Uh. It was very secret, very mysterious, and hush, hush. Jeff and his team leaned forward. This was the moment they said, imagine something the size of a coke can that's sitting on your able and we're going to sell this for uh and people will be able to talk to it. And we thought, you know, we can't do this. The technology isn't there yet, it doesn't exist yet. And of course they did do it by inventing the technology, because the product they're talking about becomes Alexa, Amazon's AI virtual assistant, and that twenty co can is the Echo smart speaker. Though it ends up selling for more than twenty Alexa is one of Amazon's first forays into artificial intelligence, and it would bring Amazon into people's living rooms, further weaving it into their lives. You're listening to Foundering. I'm your host, Brad Stone. In this episode, we're going to tell the story of the creation of Alexa. How Jeff Bezos almost singlehandedly conceived the idea for a voice activated computer and drove the device's creation. Alexa helped to solidify Amazon's image as an innovator and in the process challenge conventional notions of privacy. Will tell you more after a quick break. In the last episode, we talked about how Amazon survived the dot com bust, how Jeff Bezos said to colleagues, the only way out of this is to invent her way out. During those next few years, Bezos launched Amazon Prime and the Kindle. But there's another innovation from this era, one that generates huge profits for Amazon and sets the stage for the creation of its pioneering voice assistant. It's Amazon's cloud service called AWS, or Amazon Web Services. It's the most difficult of Bezos's inventions for lay people to understand, because as much as it may already seem like Amazon is everywhere a massive store, a network of warehouses, and the fleet of delivery people. There's one more way in which Amazon is ubiquitous indispensable. Amazon is literally holding up much of the Internet today. AWS is a highly profitable, fifty billion dollar annual business for Amazon. When you watch Netflix, you're often streaming video from Amazon. When you send a Snapchat, you're using Amazon servers. By two thousand and ten, Bezos was asking everyone in the company, what are you doing for AWS? You wanted Amazon to deepen and exploit the early lead it already had in running massive, cutting edge data centers. Around this time, he was at lunch with Greg Hart, the guy described as his consiliary, and the conversation turned how Google was starting to let people search just by talking into their smartphones. Here's Greg and and I said, look at how convenient is And I just did a Google voice search on pizza and your me um. And you know, it's just so much faster than actually typing, you know, with your fingers or your thumbs pizza and your me on your phone waiting for results to come back. Now, Bezos has long been a believer in voice. Here he is all the way back in the year two thousand talking to Charlie Rose, I believe that for mobile commerce, the thing that's going to be the biggest part of that is voice, and I think there's a lot that can happen in the In the short term, it'll be kind of a stilted special purpose language for talking to Amazon dot Com. But in the long term it could even be natural language processing. I think that is a real mind bender. Now. A decade later, it seemed like this early prediction might come true. A few weeks after their fateful lunch, Bazo sent an email to Greg that read, we should build a twenty device with its brains in the cloud that's completely controlled by your voice. The central insight here was that the product could be inexpensive because it's brains resided in Amazon's data centers and could be constantly improved, So if you buy it, the product would be upgrading itself and you wouldn't even know it. A few weeks later, Greg started recruiting people inside the company for the project. He proceeded with total secrecy. The only thing he told colleagues was that it had the potential to be bigger than the Kindle, and at the time that seemed like laughable that that we could create something that could be bigger than Kindle, because Kindle, you know, at that time was now for years into its life, dominating the e reader and e ink market and was changing the way that people thought about meeting. Next, Greg acquired YAP, the speech recognition company where Jeff Adams worked. That team didn't think he knew what he was doing either. Amazon wanted people to be able to talk to a device across a room and haven't understand them. That's called far field speech recognition, and the technology for it didn't exist. Here's Jeff Adams again. The big problem is speech recognition at the time really relied on a close talking microphone. You needed to capture the speech close to the person's mouth. And they were talking about, Oh, I'm going to be in the garage and this thing is gonna coach me through changing my car's oil, and it's gonna be over on the other side of the garage, and I'm gonna be shouting things to it, and it's gonna be shouting instructions back or whatever. And we thought, you can't do that. There's there's too many reflective surfaces in the room. They're gonna mess up the audio. Basically, if you're shouting across the room, that introduces lots of echo and the computer gets confused. So I realized that this far field issue was going to be a problem. I didn't want to say anything in in front of the group. I didn't wanna, you know, um, appear unsupportive. But afterwards I tracked down Greg Hart in the hallway and I said, Greg, we're excited about this, but I think you should know that what you want to do, the technology isn't there yet. It doesn't exist yet. We don't we we don't have that technology to solve this far field speech problem. And he was unflapped. He said, uh, I appreciate that, thank you for telling me, But solve it. We are an Amazon, We've got resources, hire as many people as you need, to take as long as you want, but you know, solve the problem. The team gave itself the code name Doppler, as in the Doppler effect, which describes the way a sound wave moves with respect to a listener. They hotly debated everything from what to name the device, to what it should do, to how to market that to the public. At first, they met with Bezos once a month. He would get deeply involved in the technology. UM, not just the business or the product, but deeply involved in the technology. He I would say, he stayed very close to the project throughout. As a project progressed, those meetings would increase in frequency UM and UH. And by the end we were meeting with him, you know, leading up to launch, I would say, there probably wasn't a day that went by that we didn't have at least one meeting with Jeff UM. And you know, sometimes there were multiple meetings with Jeff in a single day. UM. That is both a blessing and a curse. The blessing came in the form of money. The team could spend whatever they needed to break through the technical obstacles. This is a big project that Jeff is personally invested in. It was his idea originally. He can be very demanding. UM, he can push teams UM. But at the same time, like being able to go through that experience and have his brain on your side is an immensely powerful opportunity and experience if you let it, like if you get really frustrated by it. And at times some of the people on my product team would say, it feels like Jeff is the product manager for Alexa. Bezo set the vision for Alexa. He wanted it to be the Star Trek computer. He pictured a versatile, conversational machine that could respond to any question. You should be able to sit in your living room and ask Alexa anything. It sounds simple, but it's not because they needed a lot of data to train the AI algorithms. For example, for Alexa to tell you the weather, they would have to understand the phrasing of your question and your dialect from across a noisy room, and sort through databases for the right answer. This would take Alexa a step further than Syrior Google Voice, which only worked when you spoke directly into your phone. But the Alexa team just didn't have the data to get the AI smart. The early prototypes worked so badly that even Amazon employees didn't really want to test them. Bassos was getting impatient. He actually walked out of a number of internal meetings and frustration. Then Greg and his colleague struck upon an idea they called Project Amped. Basically, rents houses or apartments in cities all over the US, and we would put devices in those apartments. They were all camouflaged, and they were not just Amazon devices. There were other companies devices there as well, UM, some of which were visible, some of which were not visible. And characteristic of all Amazon projects, secrecy was the priority. Employees were careful to conceal Amazon's identity when they set up the room, and that was all about obfuscation, you know, we didn't want people to understand what company we were working for. UM, and so you could see an xbox, you could see this, you could see that UM. And then we would have all these Alexa devices hidden throughout the room or echo devices, prototype devices, and then they brought in testers, thousands of people paid an hourly wage, coming in at all hours of the day and days of the week to train the machines. Then we would bring in we would recruit participants, have people with different accents, you know, male, female, different ages who would come in and we would ask them to do a mixture of reading scripted things and then also talking in a much more off the cuff all the cart fashion to ask for things that people would be we hoped asking Alexa when we launched. Amazon conducted the data gathering effort with such stealth that neighbors started to get suspicious. House that we rented in Boston that the neighbors thought that because we had a lot of cars showing up and individuals getting out on their own and coming in and spending I think maybe an hour, and so there's a lot of sort of you know, transient in and out traffic, and neighbors thought that maybe there was you know, a drug running ring or something else going on, and so the police actually showed up despite the attention from police. Bezos loved the inventiveness of the data gathering program. When we first took AMPED to him, his response was effectively like, now you're talking, like let's do this, um, Like tell me if you want more money, Greg says. The project AMPED ran in thirteen cities and included over ten people. The devices were placed all over the room, and so we were trying to capture a massive amount of acoustic data about, you know, how noise performs in a room in all kinds of different rooms, in bedrooms and living rooms, in kitchens, in bathrooms. The result Amazon basically solved the far field voice problem. It took six months for this company to solve a problem that had stumped speech scientists for decades. But since the project was top secret, they weren't able to tell anyone, not even the speech science community, about their big technological breakthrough. Here's Ahmed Boozid. He had been working in voice technology for almost twenty years when Amazon tried to recruit him in early two fifteen. I turned him down. I said, you're not showing me anything. You have never done anything in voice, and you're telling me that you are going to any uly to to go from DC and go to Seattle and they'll let you. Just trust us. I know, no, I don't think so. I mean, I've seen the Kindle and it's fine, it's okay, but Kindall is like, you know, it's a toy compared to what you guys are trying to do. So anyway, I said no twice. Eventually they flew Ahmed out to Amazon headquarters to see Alexa in person. So it was, you know, on the table, and so the first thing I did is I asked it for for music, and he plays some Miles Davis and it did I'm like, okay, cool, And then I suppos I said, you know what time is it? Man stopping and told me the time, and then it continued playing Miles datas, which was great. Interacting with Alexa was almost a moving moment for Ahmed. He felt like decades of scientific research had been realized in this little device, and I, you know, I was like, Okay, this is amazing, This is amazing. This is clearly an important moment in technology. And this is what I've been doing all my life, right, solving the problem problems. See, you know, we people who are in the speech were are always saying this expression, you know, speeches around the corner. We've been saying it's since the mid nineties, species around the corner, because we do believe that if you do speech well, a lot of stuff becomes easy to do for people. Ahmed was particularly impressed by two innovations that Amazon made. One that they solved the far field speech issue and secondly a let so it was fast to speed right. The fact that it's just I was, I was just amazed, Like, how the hell does come back within within like two seconds? I mean, if I if I type, you know, if I want to launch a page on my browser, or it takes like sometimes like three four or five seconds. Right, How is how the hell is this thing doing all of these things, getting my voice going to the cloud, processing it, coming back talking back within two seconds. It's like a magic even for someone like me who was been like you should be jaded by by by then, Right, okay, I know exactly how it's happened. I was astonished. Right, how the hell did they do that? We'll be right back. I want to take a moment to address Amazon's obsessive secrecy. Many companies are secretive, but tech companies today have the eyes of the world on them, so they are taking corporate secrecy to a whole other level. Like that story. Jeff Adams, the speech scientist, told that the Amazon executives were so secretive when the acquired a company that they wouldn't even be seen together at a conference. That's typical of Amazon. This intense drive for secrecy comes straight from Jeff Bezos. He wants to tightly control the messaging around Amazon's new products. The idea is that complete secrecy pays off with a surprising, almost magical reveal once the product is launched. That man keeping Alexa under wraps until launch, preserving the details of how it actually worked, and choosing the perfect voice here's Greg Hart, how do you define the characteristics that you want the voice to have? And so there was sort of a brief that we wrote up about the qualities you wanted the person to be knowledgeable. We very quickly early on, decided the first voice would be female. We knew there would be additional voices, but we felt that the first voice should be female in because yeah, so that's the logical question why In part because the we knew that the device would be in the kitchen, and we felt that a female voice would be more open and inviting and warm than a male voice would be in that environment, and more appropriate in that environment. Not because of any sexist things, but just because of the fact that we knew, um that we knew the way that, um, that's the right way to say this. We had seen evidence that people respond differently to male computer voices than to female computer voices, and they respond more positively to female computer voices, and we wanted because the device was in the home, we wanted it to be a device that everybody would respond positively to. Putting a female voice in the kitchen, so to speak, would turn out to be a somewhat controversial choice, and Amazon isn't the only company to do this. Serie. Google Voice and Microsoft Cortana are all women by default, So Amazon moved forward in their search for a voice. They contracted with the same studio that had developed the voice of Sirie for Apple. Okay, I set up your meeting with David tomorrow. Shall I schedule it? Who is voiced by Susan Bennett, a career voice actress. Hi, my name is Susan Bennett, and I have a voice actor and the original voice of Sirie. So to find their Alexa, the studio had half a dozen female voice actresses read for hours. They read entire books and random articles, and finally Greg Hart and Jeff Bezos picked one woman. Her identity hasn't been revealed in all this time. It's amazing to me as an Amazon reporter that Amazon has still been able to keep the voice of Alexa a secret. In I started canvassing voice over actors, asking if they knew the identity of Alexa. No one knew. Finally, I got a tip from someone who had worked with the studio that Amazon contracted with, and they said that Alexa was voiced by an actress and singer from Boulder, Colorado named Nina Raleigh reached out to Raleigh numerous times, she wouldn't confirm that she's the voice of Alexa, but she didn't deny it either. Eventually I confirmed with enough people that I feel confident she's the one. She's Alexa. Here's a clip from her website and add that she did for Time Warner Cable. Thanks for choosing Time Warner. Now that you've ordered your installation, let's set up a time to get a crew over to your place. And here's Alexa. Time Warner Cable, also simply known as Time Warner, was an American cable television company. It was ranked the second largest cable company in the United States. But imagine what it might be like to be Nina Raleigh. Your voice is piped into millions of homes every day, each new iteration of Alexa, each update for Amazon, you have to record something new, Like when Amazon Fresh released a product called the single Burger. She had to be available single Cow Burger, a beef burger made with meat from just a single cow. She's chained to Alexa. It sounds like a life of obscurity and loneliness. Compare that to Siri and Susan Bennett. She's spoken on CNN and countless talk shows. She's even been able to use her fame as the voice of Sirie as leverage for many more opportunities. Greg saw the burden that Amazon secrecy put on the human behind the voice of Alexa. I never met her um and I don't even know that I might have. I'm not have spoken with her once, but I never met her. And it would be interesting to be in her shoes now, because it's on the one hand, it's it's incredible that this thing that you contributed to is now so ubiquitous. On the other hand, I would think it would be, you know, maybe a little bit unsettling. Amazon introduced the Echo in November two, fourteen were re strived from Amazon. I didn't know what it was. With a YouTube video, Alexa play rock music, rock music, alexis stop. We want to try Alexa, what time is it? The time is three. You actually don't have to yell at it. Okay. It uses far Fueld technology so it can hear you from anywhere in the room, So I can just hear you anywhere. Yes, that promise was enough. A responsive computer that can tell the time or the weather, play music, and answer questions from across the room. Tens of thousands of people joined a waiting list to receive a device. Here's Bezos at the Recode Technology conference in two thousand sixteen saying that the future is Alexa. He's speaking to Walt Mossburg. But it has been a dream ever since, you know, people started, you know, in the early days of science fiction to have a computer that you can talk to. So are you deeply committed to this becoming a huge part of your business and what you Absolutely We've been working on it. You know, we worked on it. We have more than a thousand people dedicated us to Alexa in the Echo ecosystem, and it's a and there's so much more to come. Baz also is deploying his playbook. For experiments that produced promising sparks, he poured gasoline on them. Amazon ramped up hiring the Alexa team, balloon to ten thou employees, and Bezos paider on ten million dollars for the company's first ever Super Bowl at starring Alec Baldwin and Missy Elliott. Alex to stop. How you do that? It's my Amazon Echo like extreme music, order things and watch this. I like to turn on the lights. Wow. Inside the company, employee has noticed that he seemed to take real joy in the work here's Ahmed Jeff, you know, the CEO and the guy in charge, uh was I would say he was obsessed with Alexa. How many he had this saying that I am happiest. He used to say this, I'm happy is when I'm working on Alexa UM and so he you know, it was his favorite time of the day. Used to meet with ALEX a team and work and work on ALEX. So he was directly involved in the early days. Basis was frequently asked about privacy. At the two thousand sixteen Code Conference, he promised to be a good steward of the sensitive personal data that Alexa was sure to pick up. This is going to get much deeper into our lives. So so what are what are the privacy I think that if you take the totality of you know, privacy UM and our ability to store large amounts of information to use it in ways that customers actually do want us to use it. So there are benefits. And I think one of the things that you have to do is when you collect and store data, you have to be clear about what you're doing. You have to and not just you know, subsection seventeen, paragraph three clearly as you can see at our privacy policy we were allowed to do that. Would you have to figure out ways to be kind of obviously clear? He stood on stage and promised that the privacy policy would be obviously clear. What he wasn't saying was that Alexa was so smart in part because there were real life humans listening through the machines, helping to craft many of Alexa's answers and to fix its errors by listening to what a subset of users said. The company kept Alexa owners in the dark about how this aspect of their devices worked. Ruthie Hope Slattess was one of those people being paid to listen. Several years ago, she saw an ad from a temp agency seeking someone with an English or journalism degree. They offered twelve dollars an hour to transcribe audio recordings. There was a kind of a generic add on prex list. I don't think it mentioned Amazon at all. She applied, She passed US curity check and a grammar test, and then she was let in on her task. She would listen to conversations picked up by the echoes microphones and type them up and then feed the information back in the Amazon system. She said it seemed like a good job. At first, we heard the customer using it in you know, ordering flour and asking what time it was and asking to be told to joke and so forth, and it seemed pretty cool. I mean a little invasive, for sure, but but pretty cool overall. Ruthie was there to make Alexa's responses seem more nuanced, more intuitive, more human. This was right around the time Amazon started selling the Echo to customers on a limited basis. She assumed at first that the folks she was listening to had signed up to help improve Amazon speech recognition software, and knew that someone might be listening. Who were all of these all of these many many voices and people were they? Various people work for Amazon who agreed to bring this home where they you know? But then if they were, would they really be ordering sex toys, you know, um and talking dirty to it and all of the grotesque things that we occasionally heard. Almost immediately, it was clear to Ruthie that people liked a toy with Alexa in ways that Amazon maybe didn't intend. What can we ask her? How how will she respond if we abuse her, if we talked dirty to her, if we asked her to marry us? If we you know, that sort of thing. Most of the sexual stuff seemed innocuous enough. Percent of it wasn't disturbing. You know, it was sometimes humorous and sometimes you know, kind of silly or weird or entertaining even you know, like somebody trying to figure out what kind of dildo toward her or something like that, you know, but sometimes it did get disturbing if it was a man who was talking to her, like he would talk in an abusive way to a woman, Like you could hear it in his voice, And she wondered if men were being so rude in part because Alexa was female, and being a woman, even an ai woman, meant that Alexa would be on the receiving end of misogyny. Even children did this. There were a lot of kids who would talk abusively to her, and it felt as though they were exercising some sort of anger that they had towards their parents or their teacher or something like that. There was a lot of psychology that I thought was fascinating. Ruthie felt like the way people spoke to Alexa was so private, so intimate. They would never ever speak like this if they knew someone was listening. There in their private home. They think that no one will ever hear the words that are coming out of their mouth. Ever, they do know that they're talking to a robot, but at the same time they're speaking to a row that as those essentient being. Ruthie's job was a secret. It wouldn't become public until years later that people like her were helping to improve the software by analyzing real Alexa recordings. My colleagues first interviewed Ruthie in two thousand nineteen, and after we published a story describing what she and her colleagues did, Amazon acknowledged the listening program. Technically, Alexa's terms of use gave Amazon wide latitude to duke basically whatever it wanted with the recordings, as long as the company was using that audio to improve the software. But many customers fell duped. They thought that when they spoke to their Alexa device, they were only dealing with clever software, and Bezos had gone back on his word his precise promise to be transparent with how he used as customers data. It seemed that while he highly prioritized Amazon's corporate privacy it's secrecy, he was being careless with the privacy of his customers, and in two thousand eight teen, Alexa had a huge privacy mishap. Here's my colleague Prea and Nod describing the infamous incident. What happened there is a family in Portland's said that their echo randomly sent recorded conversations to a contact of their's, a contact of the man in the houses. And what happened at the time was Amazon said this is how it worked. They said that Alexa interpreted something those people were saying in their background conversation as Alexa, so it woke up. Then it misunderstood a different phrase in their background conversation as send that message. It understood a different phrase in their background conversation. After that, the device asked to whom and thought that they responded with a name, So it ended up sending these conversations to one of their contacts through this crazy chain of events. It's like a horror movie. After the couple's friend reached out to them and said, hey, I think you're Alexa has been hacked, it turned out they weren't hacked. This was Alexa mishearing. But at the end of the day, this kind of scenario where Alexa could misinterpret a series of things people were saying in their own home, interpret them as requests, and send these people's recordings to someone else. If this kind of unlikely scenario can happen, and did happen, then who's to say it can't happen again? What else can Alexa mix up? Even so, people were undeterred. Over the next few years, Amazon would sell tens of millions of Alexa devices and inspire imitators from Google and Apple. It's popularity cemented bezos notion of himself as an inventor and the general public's perception of Amazon as an innovator. But Alexa hasn't quite met the goals its creators had for it. Most people only use it as a kitchen timer, a music player, a source for the occasional weather report. It's never been conversational in the way Bezos imagined no one would call at the Star Trek computer, but that hasn't stopped consumers and estimated one in three US households currently have a smart speaker. In the years since Alexa launched, people have invited more items that integrate AI assistance into their homes. That includes stuff like nest thermostats, ring cameras, and Sono speakers. Whenever a story about an Alexa privacy breach gets published, it makes a big splash, but it hardly ever changes anything. Because Alexa marked a turning point, It was another doorway into a more surveiled world and placed Amazon alongside other big tech companies like Google and Facebook at the hot controversial center of the battle over privacy. Finally, Alexa turned to Amazon into a company its customers interacted with almost every single day, not just once a week or a few times a month when they wanted to buy something online. And there was another way Bezos would accomplish the same goal by producing his own TV shows and movies. Bezos is risky expansion into hollyw would that's next time on Foundering The Amazon story. Foundering is hosted by me brad Stone. Sean wen Is. Our executive producer. Pria Nad and Matt Day contributed reporting to this episode. Raymondo is our audio engineer, Molly Nugent as our associate producer, Mark Million and Manner May Robin a Jello and Molly shoots our story editors. Francesca Levi is the head of Bloomberg Podcast. Be sure to subscribe and if you like our show, leave a review. Most importantly, tell your friends see you next time.