In this episode, we'll be focusing on Project Debater, which is an AI system designed to process evidence and persuasive arguments and text so that it can ultimately understand and participate in human debate. To get to the heart of this effort, we're going to share two interviews we recorded with leaders at IBM. The first is with Noam slow Name, who is a distinguished engineer at IBM Research and founder of Project Debater, and the second chat will be with matdou Coachar, who is Vice President Offering Management for IBM Data and AI. So today's episode is going to be the third of four episodes in this series that Robert and I are releasing here on the Stuff to Blow Your Mind feed. If you'd like to hear more episodes, you can check out the ones labeled smart Talks that we've released over the past few weeks, and you can also listen to the first four episodes of smart Talks, which were released not on our show but in the feed. For the podcast Text Stuff. You can find them on the I Heart Radio app or wherever you get your podcast. Just look up text Stuff and click on the episode has labeled Smart Talks, and of course stay tuned for the one remaining episode in the series, which is going to be published in our feed in a couple of weeks. And now straight onto our conversation with no One Slowly, no One, thanks so much for joining us today. Can you start by introducing yourself and talking about your role at IBM? Sure, thank you for hosting me. So I'm no One Slownym. I'm a distinguished engineer at IBMI Research. I did my PhD in the Hebew University quite a few years ago walking on machine learning, staff and artificial intelligence, and then I did a past doc at Princeton University and I joined the IBM research in two thousand and seven, and uh in two thousand and eleven, I suggested the project that I guess we're going to talk about today, and of course that project was Project Debat, right do you do? You want to mention a little bit about the origins of that. In IBM research, we have this interesting tradition of grand challenges in artificial intelligence. Back in the nineties, idem introduced the Blue that was able to defeat Gary customers in chess, and in two thousand eleven id AM introduced Watson that was able to defeat the all time winners of the TV trivia game Jeopardy. And just a few days after this event, an email was sent to all the thousands of researchers in i DM across the globe, myself included, asking us what should be the next grand challenge for IDM research and uh I was intrigued by that, so I offered my office mate at the time to brainstone together, and this is what we did. We set in the office in Tel Aviv and we raised many different ideas that probably I should not share with you today, but at some point towards the end of the hour, well I suggested this notion of developing a machine that we'll be able to debate humans, and that this is how we will demonstrate the technology for a full life debate between this envisioned system and an expert human debate. And we submitted that the only guidance that we got from the management was really to submit the proposals in a single side so they will not be swamped with too many details. And we were able to helpfully follow these guidelines and we submitted a single slide. This was fair Boy in two thousand eleven, and this started a fairly long, and the thought review process that lasted for a year, and in February two thousand and twelve, this proposal was selected as the next Man Challenge for IBM research and we started to walk a few months later with a small team that gradually expanded, and we walked on that intensively for I would say six and a half yels dedicated solewly to dismission of developing a machine that will be able to debate humans. And eventually we demonstrated this system in a in a full life debate. It was a little bit more than a year ago, and it was a debate between this system now being called the project debate and one of the legendary debates in the history of university debate competitions, and it still Harris Naam. It was in San Francisco, and and it was a full life debate, surprisingly reminiscent to division that we had back in the office in Tel Aviv quite a few fields earlier in that single side. So the topic of debate brings with it a few different connotations, um, you know, and therefore the idea of AI entering the frame might might be a bit confusing for for some you know, we might imagine a computer designed to defeat play or or perhaps a robot that can shout louder and a televised US presidential debate to Daddy, and can you walk us through what Project Debater is and perhaps what it isn't. Yes, absolutely so. So first of all, it is worth explaining what we mean, indeed by a debate between an AI system like Project Debata and a human opponent. So the debate starts with with a motion in the debate jargon that defines what we're going to debate. And in the event in San Francisco, the topic was whether or not the government should subsidize the schools. Uh. There are many considerations around how this topic is being selected which we can skip, but the only thing we should really emphasize is that this topic is selected from a list of topics that were never included in the training of the system, So the system was never able to train on this particular topic. It was trying to debate a new topic from from the perspective of the machine. And then we are on the side of the governments of Project Debta is supporting the motion and how the issues on the opposition, and we have a full minutes opening speeches for each side and full minutely bottom speeches and two minutes closing statements. So all you know, we are talking about a little more than twenty to twenty five minutes of a discussion that we hope we will be a meaningful discussion between Project Debata and and and a human plish in these particularly so to clarify for people who might not be familiar with competitive debating. So competitive debating does not involve what people might be more familiar with, which is like passionately arguing your actual point of view. It involves having a position selected for you that you then must get up and defend in front of the judges. Correct, yes, this is called act and and this is indeed important to emphasize because you do not know in advance what is going to be your side. And and even if you know in advance that you are going to be on the side of the government, we should bear in mind the motion could have been phrased we should not subsidize previously, and then you should actually contest that. So you do not know in advance what is going to be your stance to the topic. This is true for Project Debata and also for the for the human opponent, and you have only ten to fifteen minutes to PerPell. You don't know the topic in advance. This is again true for project debata and for the human opponent, and uh, your goal is really to to persuade the audience. And this actually touches on an interesting question of how do you do you measure who won the debate? Because in chess and in other games this is very clear and and really part of the problem with with with debate in general and with developing artificial intelligence that is capable of debating in particular now is that it is very hard to to be fine who actually won the debate. Yeah, I know. There are a couple of different metrics. So of course one would just be like, what is the percentage of the audience that is convinced to either side? But that can be problematic because people come in with their own opinions already formed on an issue. So one metric I've seen is how much the percentages change. They ask people before and afterward what their positions are, and then after word they say, okay, which side has one over more people? Whatever the starting percentages were is, And I assume you all had a metric like that precisely so, so this is exactly the point, because if you simply ask people who is more convinced, you need somehow to take into account the opinions to begin with, and and the it is done exactly as as you described it. And all this event was in collaboration with with Intelligence as well, which is really I think the leading platform in the US for organizing such a high profile competitive debate. It was hosted, the moderator was the moderator Form Intelligence as well, John Dunvan, and and the voting was done exactly as you described and as being done with the show of Intelligence Square. That is, the audience is voting before the debate starts, and they vote again after the debate ends, and you win if you were able to move more people to to your side. Now I think a lot of people might be wondering, how on earth would you even begin to organize a persuasive argument from an AI point of view? Could you walk us through the technical specifics of how Project Debater would put together an argument. Yes, so we were asking ourselves the same question actually when when we started this project. And I think this is part of the of the nature of such a grand challenge that you do not really know how exactly you are going to to approach the problem. But we did what computer scientists often do, and this is to take this big and somewhat amorphic problem and break it into more modular and hopefully more tangible tasks. And so in general, the debated system had uh two major sources of information. One of them is the massive collection of around four hundred million newspaper articles, and when the debate starts, the system was using various AI artificial intelligence engines in order to try and pinpoint short pieces of text within this massive collection. We're talking about ten billion sentences, so we were trying to automatically pinpoint these short pieces of text that should satisfy three criteria. They should be relevant to the topic, they should be argumentative in nature, they should argue something about the topic, and they should support our side of the debate. And this is quite a formidable challenge. But assuming that you are capable of finding these short pieces of tax, the system is then using other AI capabilities in order to try and glue them together into a meaningful narrative. So this is one major source of information for the system. The second important source of information for the system was a unique collection of more principled arguments that were actually written by by humans, and we are talking about thousands of more principled arguments. And the role of the system was when the debate starts, was really to navigate within this collection and find the most relevant principled arguments and use them in the right timing. So so, to make this more concrete what we mean by a principal argument, imagine that we are debating whether or not to ban organ trade or whether or not to ban the sale of alcohol. In both cases, the opposition may argue that if you ban something, you are at the risk of the emergence of a black market. So a black market is a principled argument that can be used almost in the same way in many different contexts. So one may naively assume that this is kind of a simple keyword matching thing. If we ban something, then the opposition is going to use the black market argument, and we should be prepared for that. But obviously this is far from true. So imagine a debate about banning breastfeeding in public. Obviously there is little risk for a black market in this contract. Or imagine a debate about banning internet cookies. We're not going to tee a black market of internet cookies if we band these. So the system really needs to develop a more subtle understanding after human language in order to be able to identify the most relevant principle argument and need use them doing a debate. And and this is, by the way, just what all this description is before listening to the opponent. This is just what we're going to say on our side. And and the most the most challenging part is really too uh to listen to the opponent. And it's some kind of a battle to the arguments generated by the opponment raised by the And we do that you using uh an arsenal of technique that most of them rely on the same principle. We start by listening to the world articulated by the opponment, and for that we simply use what's on speech recognition capabilities out of the box. But of course we need to go to beyond the world, and we need to understand the gist of the arguments of the opponent. And in order to do that we try using various smackloads to anticipate in advance what kind of arguments the opposition mind you and then listen to determine whether he did the opposition was making these arguments and then responded cold yeah. That calls to mind the question of the difference between, say, what's a sound argument versus what's a persuasive argument? I mean, we know from reality that often the most persuasive appeals and debates rely on just straightforwardly false claims and logical fallacies, or even on little emotional cues that have little to do with the matter at hand. I was thinking about how in live debates, if you can get a laugh at your opponent's expense, that's worth you know, a dozen sound arguments or claims. So to what degree can AI understand these sorts of persuasive appeals that that go beyond just like what kind of evidence you can bring and the appeals based on style you're right in in in in debate and in the methods. We know already from the ancient weeks that that we have free elaps, we have logos, and we have ethos, and we have afforts, and humans are using a mixture of these pilas when they are debating one another. And just as a quick clarification, logos, pathos and ethos are the types of appeals that were identified in the study of classical rhetoric. Where logos is appeals based on our logical arguments and evidence, Pathos is the appeal to the emotions or the passions, and ethos is an appeal based on the credibility or authority of the speaker. I mean, as you know broadly understood and and the technology that we developed, and and by the way, it should be stated that there is a rapidly emerging community of scientists across the globe that are investigating this kind of topic. It is all under the umbrella of this emerging field, yeah, referred to as a computational argumentation. And when we started in two thousand and twelve, there was a handful of teams pursuing that, and we see a very dramatic increase in the result in these areas of the last few years is very I think from exacting and as I mentioned, the technology that we developed a most focused on logos, and you can see in the debate between proper Debate and Hali. By the way, this this debate is is fully available on YouTube, and you can see that indeed a woman is better in making in using path as and perhaps in using ethos and it is harder for the machine. And indeed most of the research being done by by the by the relevant research communities around logos, but there are already attempt trying to model and to capture additional aspect of path of and ethos in all the further enhanced this kind of technology. So another question I have is debater has to source claims and facts and arguments from existing written work produced by humans, which of course we know can be full of all sorts of flaws. Is there any way at this point for it to to have an analytical function to tell a say, factually true claim or a logically valid argument from just something that is wrong or dubious but repeated a lot in writing, or are we not there yet? This is a very kindly important and difficult problem, and that is receiving going attempting over over the previous teams and go to tackle that. This is certainly not bullet bof and and the problem is is quite complex because one may say, you know, okay, fine, maybe I should only take my argument from highly credibally so and by boxy I can assume that that these arguments are our valid. But this is not necessarily the case. Right. You can see you can lead an opinion article in a highly respectable newspaper which is actually quoting a false argument that was made as well, and if you're not careful enough, you you might be your system is going to pull this argument without understanding that something is happening. So we try to develop and we actually part of Project Debate included some kind of filtering mechanism in order to to filter out these kind of cases. And the way we did that was really once a specific claim was affected and by the way to being ordered, the claim is not a full sentence. A claim is often only a part of a tentence. Even if you were able to detect sentence that contains a claim relevant one that supportal side out of the billions of sentences in the popos, you still need to find the coret boundaries after claim within the sentence, and you have hundreds of options and only all of them is correct. So this is just going back why this this problem is it so talenting? But until you do that and found this claim and asked what is the stance of this claim, and if the stance is supporting your side, you can still ask what is the stance of the full sentence? And if the stance of the full sentences in the opposite direction, you may suspect that something is going on. And perhaps this this claim is quoted in order to contradict and not because it is true. And then perhaps it is there it is safer to avoid using it. But but this is just one safety mechanism, and and the problem that you raise is actually a much more beneval one, and and I think many teams are working on that, and we try to address that as well. And I think it has many interesting dimensions because it is not even just about the validity of the argument. Often, when when you show people to arguments, they will agree that one of them is better than the other. But what are the underlying mechanisms that I'd ask to the one argument over the other, And how do you train an artificial as in system to make the distinction. This is kind of another example of the problems that welcome to them. I have a question about what could come out of AI research like this, because I would say, from my personal perspective, I think studying rhetoric and debate is extremely important, but not necessarily because getting into debates is a good way to figure out what's true and establish you know, the right thing to do. I think one of the most important reasons to study rhetoric and debate is so that you can understand how other people's arguments and persuasive appeals are operating on you, or are designed to operate you. A clear understanding of rhetoric can be a kind of suit of armor for going into you know, the world and seeing how political actors and business actors and advertising and all that is trying to affect you. Do you see project debate or serving any kind of educational purpose like this in the world today. So there are several levels by which I can I can answer that. The first one is that this kind of technology is is definitely relevant and we believe highly valuable in the context of education. You can imagine using the technology in order to build better arguments and more of all, to perform a more analytical and perhaps more objective analysis off complex and controversial topics. This is one aspect. There is another aspect, but often when we debate is other humans. There are many layouts that that are involved in this discussion. In this debate. What all of them are related? To the facts and to the arguments that we are raising. Perhaps we have history with that Belton, Perhaps we have history with ourselves that actually impact our on part and decisions. Perhaps other people are listening and this actually improvides contact, uh that impact what is happening. And we are curious about this option of the dating with the machine in the privacy of your office. Maybe this is a different form of a discussion that to some extent is perhaps all free of of external biases and maybe will enable treat some people to identify situations where they have a blind book and to better listen to the other side. So I think in this case the whole of the technology could be quite instrumental and positive. The false business applications that are also very interesting from the IBM perspective and uh, and this is another another dimension, another level by which we can consider the technology as exacuable. Again, big thanks to No One slow name for taking time to chat with us. And now we're going to go straight into our second talk on the subject with Madu Matt. Thanks so much for joining us today. Could you start off by introducing yourself and talking about your role at IBM. Yeah, absolutely, and really nice to meet you. Robert and Joe uh maduco Chi, vice President Offering Management in Data and AI IBM, And the role of offering management is really all about laying down the strategy and then delivering and executing towards such strategy. And I'm based out of San Jose, Sunny, California, excellent. So just to kick things off here, um, you know we're gonna be talking a lot about AI here, and it makes sense to to to really get into what we mean when we're talking about AI for business. How does AI serve business compared to the way it serves consumers. That's a great question to get started on. UM so redeveloped a thesis a couple of years ago about really how AI for business would be different from consumer AI. Think of consumer AI, which we all know work with our smartphones, smart speakers, social media, photos, everything what it comes. But when it comes for AI for business, it's really very very different. AI for business is all about automation, optimization and making better predictions, and it requires really a very different set of technical capabilities, like you would have to understand how to deal with language, have to deal with what does automation means, and then be able to have the explainability and trust up AI. UM. So that's sort of the big difference between commercial AI and AI for business. So we know that one of the big AI projects at IBM is Watson. Could you tell us about Watson and explain how Watson fits into the broader picture of recent advancements in AI. Sure you you might have heard of Watson, and our audience might have heard of Watson, which came out when we first did our UH in Jeopardy and people remember Watson from there. But fast forward, a lot of work done around Watson. Think of Watson as our definition of IBM AI. We evolved a lot um since then, and our strategic intent always has been to have what's an available anywhere meaning available on any cloud. UH. We have focused on Watson. UH we call with Watson meaning it's embedded in almost all your applications. So for example, UM, I use the world a lot for AI for AI. What does that mean? Like, how do we embed AI in our data sciences and in our data data platforms and such. The other parts of evolution has been you know, as I said earlier, from our AI for business is all about automation. How do we UH evolve into the workflow AI that matters for our clients and our our society. So the workflows could definition could be you know, customer care, uh in I t asset management, in your regulatory or compliance, in supply chain or in your planning and budgeting. Right, these are how you can really embed AI and that is where Watson has really evolved into. And we have also been delivering now Watson an AI capability in a in our integrated single platform we call cloud Pacer data. So a long way. We came from Jeopardy Days and then you just heard from nome where we landed with Debater. So speaking of Debater, what capabilities has IBM commercialized from Project Debater into Watson? So that's a great question. Um, A lot of commercialization has happened. We have uh pretty good rich set of products like what's an assistant, what's on discovery, what's on knowledge, language understanding? And I know the are just works, but let me just give a bit of a background on what what's an assistant is? What's an assistant is? Our conversational AI platform really helps provide customer fast, straightforward answer, accurate answers UM across any application, device or cloud right, UM and our discovery is all about enterprise search and AI search technology that truly retrieves specific answers to your questions while you're analyzing trends and relationships in the enterprise data. So we've been looking at debater and some of the key technologies. Let me give you an example of few UM like sentiment analysis. Uh, let me pose a problem statement, what does that really mean? So, for example, today Watson does not understand idioms or sentiment shifters, and neither does any other competitor operates out there also, So think of elements which include hardly helpful, over the moon, cold feet, UM all years. You know, how do you make that analysis and figure figure this out? What is the real context behind this? So what we have done with that is that now what's on leverages this debat technology and looks at these idioms and sentiment shifters and does the analysis starting with better understanding of this sentiment and analysis is one of the most widely used API s for us UM. This already exists today in our product portfolio. What's coming into the future is UM. It's around all around documents. So let me put a perspective around a problem statement. There are many regulatory documents such as contracts or security filings which contains important clauses that have really really serious business implications for example, payment terms, obligations made to regulatory bodies, or warranties. Such humans can spend countless hours reading and extracting the information so they remain compliant. Although we can provide some of the out of the box models for contracts and invoices and such, but it creates UM but client may still need to create their own element classifications of business classes. So the solution has been with our debaters birth based classification technology into these products so we can learn with few one samples to do new classification of elements. Business documents could include contracts, invoices, and procurement contracts. The end of the day, it really really excelerates the outcomes what the businesses would be looking for. UM. Other technology is around summarization. So the problem statement here is like when you're looking for information customer or employee who may have aggregate research from different sources, clicking through multiple links and pages and finding exactly what they need can be very very difficult, right. It can take months, weeks and months sometimes to your years. So with Watson and Debater technology, we can analyze variety of these sources and provide a summary or brief of the ideas and the information which is contained within UM that's coming up. We're going to be leveraging this technology in our Watson discovery portfolio in second half. The other interesting UM issues we see today is like in our traditional UH rule based systems for contact centers, it categorizes large fraction of calls in a very generic bucket like it says, you know, like not uncommon to see more than maybe of calling a call center for a bank, which says, hey, this this call was just made for generalized checking, and it prevents the company from creating any robust self service. So with Debater technology, now we can leverage advanced topic clustering, which enables users to cluster this incoming data in a meaningful topics of related information and automatically this can be analyzed. So think of discovery of a content minor which will be enhanced with this type of a technology to extract better topics from very large data sets and then make the topic extraction more business user friendly. So a lot of stuff. I give a lot of examples, but sort of the gist of all this is, Look, it's going to impact businesses real outcomes, right, It's going to save them time, is going to automate the process, it's going to remove a lot of human error which comes with it, and really speak towards the productivity. Is going to speak towards the clients UM and P as their own promoter scores and such, and so that's really the gist of what we're looking to drive out of the debater technology. If I'm understanding this correctly, this is interesting that it's interesting that this kind of functionality would come out of an AI debate tool, because debate and persuasion that will seem like the kinds of things that would be inherently the most difficult to master with AI, because you've got all these elements of style and subtlety things that are really difficult to quantify to make into two understandable data. But out of the debater technology, it sounds like you're saying that you're actually getting a lot of derivative technologies that are good at dealing with algorithmic types of text like legal documents. Am I getting this right? Like that you could have a piece of software that works like a lawyer. Uh, and it can explain this contract to you when it's going over your head. And this kind of thing is possible now because of how formulaic and algorithmic legal documents tend to be. Would that be a correct understanding? Yeah, no, totally And if I may, UM give you one of the client example, especially as you started talking about legal UM, Legal Nations platform actually provides this in house legal teams and outside console the ability to respond to their lawsuits UM and draft their initial round up discovery requests literally less than two minutes right um and which shaved off about ten hours of attorney times on each of these lawsuits. So the real direct outcomes of usage of this technology. So you've been talking about big business applications, but I also wonder about applications directly for the consumer. Where, for example, because you have a program that ingests legal documents, so you you feed it some contract you're thinking about signing, and then you say, I have a question because I'm not a lawyer, I don't understand what I would be bound to do under this agreement. And then you could feed the contract in and pose questions to your AI legal assistant in natural language. Can you see a future like that. We do, and we already have a product like what'son Assistant, which is for customer care. It feeds on a lot of you know, pre train models, like especially now in COVID nineteen right, Uh, a situation where our government offices and our healthcare are getting in dated by calls. Right, So leveraging this UM what'son Assistant in front is really helping them deflect a lot of those phone calls and get the accurate answers in hands of the consumers. So you know, this is what we are focusing on around customer care. But yeah, in the future, I mean this similar technology and leveraging UM the from debater, we can actually go into any domain. Right, we have the right framework and we have the right technology to go pursue those different domains. I guess this sets us up for a bigger question, which is what is the overall role of natural language processing in the landscape of AI today and also which are the elements of natural language processing that we've really gotten a lot better at and which are the ones that are still a major challenge. Yeah, great question. As we all know, right, language have existed. I don't know a hundred thousand plus years. You know, started as speech probably people started to talk and the writing came perhaps much later. Um, and we write in ways we don't talk also, right, it's a lot more descriptive and more reflective. And so now with things where we can compute at larger with open data sets and transfer learnings, n LP natural language processing really really is the inflection point, right, And some of the examples I shared earlier around the sentiment analysis and summarization and clustering, these are all such critical aspects of taking LP, not just natural language processing, but natural language understanding, natural language generations is all going to come through all of that. And we really think with the Debater technology it really puts us in a in a leader quadrant here a lot more work to be done, but the the end goal is yes, we can continue to research on these things, but how quickly we commercialize it and how quick quickly we help our clients and users to see the outcomes what are needed here and make them a lot more productive. So how many languages does Project Debater and Watson together, how many do they understand support today? We started with obviously English, we are expanding now to French, Spanish, German in in the second half of this year, and then very soon will expand to Dutch, French, Arabic, Chinese both traditional and simplified, and Italian. UM And obviously we are choosing these based on where we are seeing most of our growth and an adoption. What are some additional examples of how these commercialized capabilities can be used by clients? Great question, um. I gave you an example earlier on legal missions. The other one, which is very close to my heart is um RBS with Watson. Watson RBS built Cora, which is it? Which is their digital assistant that helps better serve their customers through first time problem resolutions. Cora is trained with the were one thousand responses to more than two customer queries. However, if she doesn't know an answer or she sends that customer is getting angry or frustrated, she will transfer it to a live agent. Now, with improved sentiment analysis from Debater, as I mentioned earlier, we hope that clients like RBS will be able to better serve their customers by having digital assistance that better understand the subtleties of the of the sentiments of the clients. So for example, the phrase over the moon might be interpreted as literally about the planetary satellite and not as excited or elated. Right. So this is what with Project Debater core AI built into IBM Watson, it can understand these idioms helping clients like RBS to better serve their customers. The other example switching into financial like Credit Mutual, they had over five thousand branches and they receive more than three fifty thousand online inquiries a day and the volume is growing at least twenty three percent a year. So now with Watson infused email analyzer, they can help deflect and address of the three daily emails received by banks client advisors. So the implementation of the topic clustering from Debater, we believe now clients with similar needs that Credit Mutual will enable more self service by identifying clusters are commonly as topics and can be converted into self service content. Right. So to me, the examples like this are just amazing because I can totally then connect the dots between technology, the usage and the outcome, right, a win win situation. We've got multiple other examples as well, Roberts, and we're going to continue to be focusing on how do we really not just commercialize it, but I believe in AI is really meant to improve our society as well, right, make us more productive and do better things, especially the world we are living in with COVID and other things which are happening around us. Right, Um, the goodness of AI needs to be there, so very critical overall, what do you see as the best possible role for AI, not just as a tool for business, but as a society. What could it do for us in the best case scenario? Yeah, I mean that's a great question, right um. To me fundamentally, I mean there are many examples, but one most critical which comes to my mind is how AI can really help us detect bias? Right, A lot of our data sets and it has been built by humans with unbiased goes into those data. Right, AI can really start separating that help us detect bias and and make our products better, makes our society better. So that to me is the would be sort of the holy grail if I can achieve that. All right, So there you have it. Thanks once again to know I'm slow name and Maduka char for taking time out of their busy days to chat with us about this topic. For more information on smart Talks, visit IBM dot com slash smart Talks, and if you'd like to learn more about n LP, you can go to IBM dot com slash Watson, Slash Natural dash Language, dash Processing. And if you would like to learn more about our show, well, you can find us wherever you get your podcasts and wherever that happens to be. Just make sure you rate, review and subscribe. Huge thanks as always to our excellent audio producers eth Nicholas Johnson. If you would like to get in touch with us with feedback on this episode or any other, to suggest a topic for the future, or just to say hello, you can email us at contact at stuff to Blow your Mind dot com. Stuff to Blow Your Mind is production of I Heart Radio. For more podcasts for my Heart Radio, visit the i Heart Radio app, Apple Podcasts, or wherever you listening to your favorite shows,

In this episode, we'll be focusing on Project Debater, which is an AI system designed to process evidence and persuasive arguments and text so that it can ultimately understand and participate in human debate. To get to the heart of this effort, we're going to share two interviews we recorded with leaders at IBM. The first is with Noam slow Name, who is a distinguished engineer at IBM Research and founder of Project Debater, and the second chat will be with matdou Coachar, who is Vice President Offering Management for IBM Data and AI. So today's episode is going to be the third of four episodes in this series that Robert and I are releasing here on the Stuff to Blow Your Mind feed. If you'd like to hear more episodes, you can check out the ones labeled smart Talks that we've released over the past few weeks, and you can also listen to the first four episodes of smart Talks, which were released not on our show but in the feed. For the podcast Text Stuff. You can find them on the I Heart Radio app or wherever you get your podcast. Just look up text Stuff and click on the episode has labeled Smart Talks, and of course stay tuned for the one remaining episode in the series, which is going to be published in our feed in a couple of weeks. And now straight onto our conversation with no One Slowly, no One, thanks so much for joining us today. Can you start by introducing yourself and talking about your role at IBM? Sure, thank you for hosting me. So I'm no One Slownym. I'm a distinguished engineer at IBMI Research. I did my PhD in the Hebew University quite a few years ago walking on machine learning, staff and artificial intelligence, and then I did a past doc at Princeton University and I joined the IBM research in two thousand and seven, and uh in two thousand and eleven, I suggested the project that I guess we're going to talk about today, and of course that project was Project Debat, right do you do? You want to mention a little bit about the origins of that. In IBM research, we have this interesting tradition of grand challenges in artificial intelligence. Back in the nineties, idem introduced the Blue that was able to defeat Gary customers in chess, and in two thousand eleven id AM introduced Watson that was able to defeat the all time winners of the TV trivia game Jeopardy. And just a few days after this event, an email was sent to all the thousands of researchers in i DM across the globe, myself included, asking us what should be the next grand challenge for IDM research and uh I was intrigued by that, so I offered my office mate at the time to brainstone together, and this is what we did. We set in the office in Tel Aviv and we raised many different ideas that probably I should not share with you today, but at some point towards the end of the hour, well I suggested this notion of developing a machine that we'll be able to debate humans, and that this is how we will demonstrate the technology for a full life debate between this envisioned system and an expert human debate. And we submitted that the only guidance that we got from the management was really to submit the proposals in a single side so they will not be swamped with too many details. And we were able to helpfully follow these guidelines and we submitted a single slide. This was fair Boy in two thousand eleven, and this started a fairly long, and the thought review process that lasted for a year, and in February two thousand and twelve, this proposal was selected as the next Man Challenge for IBM research and we started to walk a few months later with a small team that gradually expanded, and we walked on that intensively for I would say six and a half yels dedicated solewly to dismission of developing a machine that will be able to debate humans. And eventually we demonstrated this system in a in a full life debate. It was a little bit more than a year ago, and it was a debate between this system now being called the project debate and one of the legendary debates in the history of university debate competitions, and it still Harris Naam. It was in San Francisco, and and it was a full life debate, surprisingly reminiscent to division that we had back in the office in Tel Aviv quite a few fields earlier in that single side. So the topic of debate brings with it a few different connotations, um, you know, and therefore the idea of AI entering the frame might might be a bit confusing for for some you know, we might imagine a computer designed to defeat play or or perhaps a robot that can shout louder and a televised US presidential debate to Daddy, and can you walk us through what Project Debater is and perhaps what it isn't. Yes, absolutely so. So first of all, it is worth explaining what we mean, indeed by a debate between an AI system like Project Debata and a human opponent. So the debate starts with with a motion in the debate jargon that defines what we're going to debate. And in the event in San Francisco, the topic was whether or not the government should subsidize the schools. Uh. There are many considerations around how this topic is being selected which we can skip, but the only thing we should really emphasize is that this topic is selected from a list of topics that were never included in the training of the system, So the system was never able to train on this particular topic. It was trying to debate a new topic from from the perspective of the machine. And then we are on the side of the governments of Project Debta is supporting the motion and how the issues on the opposition, and we have a full minutes opening speeches for each side and full minutely bottom speeches and two minutes closing statements. So all you know, we are talking about a little more than twenty to twenty five minutes of a discussion that we hope we will be a meaningful discussion between Project Debata and and and a human plish in these particularly so to clarify for people who might not be familiar with competitive debating. So competitive debating does not involve what people might be more familiar with, which is like passionately arguing your actual point of view. It involves having a position selected for you that you then must get up and defend in front of the judges. Correct, yes, this is called act and and this is indeed important to emphasize because you do not know in advance what is going to be your side. And and even if you know in advance that you are going to be on the side of the government, we should bear in mind the motion could have been phrased we should not subsidize previously, and then you should actually contest that. So you do not know in advance what is going to be your stance to the topic. This is true for Project Debata and also for the for the human opponent, and you have only ten to fifteen minutes to PerPell. You don't know the topic in advance. This is again true for project debata and for the human opponent, and uh, your goal is really to to persuade the audience. And this actually touches on an interesting question of how do you do you measure who won the debate? Because in chess and in other games this is very clear and and really part of the problem with with with debate in general and with developing artificial intelligence that is capable of debating in particular now is that it is very hard to to be fine who actually won the debate. Yeah, I know. There are a couple of different metrics. So of course one would just be like, what is the percentage of the audience that is convinced to either side? But that can be problematic because people come in with their own opinions already formed on an issue. So one metric I've seen is how much the percentages change. They ask people before and afterward what their positions are, and then after word they say, okay, which side has one over more people? Whatever the starting percentages were is, And I assume you all had a metric like that precisely so, so this is exactly the point, because if you simply ask people who is more convinced, you need somehow to take into account the opinions to begin with, and and the it is done exactly as as you described it. And all this event was in collaboration with with Intelligence as well, which is really I think the leading platform in the US for organizing such a high profile competitive debate. It was hosted, the moderator was the moderator Form Intelligence as well, John Dunvan, and and the voting was done exactly as you described and as being done with the show of Intelligence Square. That is, the audience is voting before the debate starts, and they vote again after the debate ends, and you win if you were able to move more people to to your side. Now I think a lot of people might be wondering, how on earth would you even begin to organize a persuasive argument from an AI point of view? Could you walk us through the technical specifics of how Project Debater would put together an argument. Yes, so we were asking ourselves the same question actually when when we started this project. And I think this is part of the of the nature of such a grand challenge that you do not really know how exactly you are going to to approach the problem. But we did what computer scientists often do, and this is to take this big and somewhat amorphic problem and break it into more modular and hopefully more tangible tasks. And so in general, the debated system had uh two major sources of information. One of them is the massive collection of around four hundred million newspaper articles, and when the debate starts, the system was using various AI artificial intelligence engines in order to try and pinpoint short pieces of text within this massive collection. We're talking about ten billion sentences, so we were trying to automatically pinpoint these short pieces of text that should satisfy three criteria. They should be relevant to the topic, they should be argumentative in nature, they should argue something about the topic, and they should support our side of the debate. And this is quite a formidable challenge. But assuming that you are capable of finding these short pieces of tax, the system is then using other AI capabilities in order to try and glue them together into a meaningful narrative. So this is one major source of information for the system. The second important source of information for the system was a unique collection of more principled arguments that were actually written by by humans, and we are talking about thousands of more principled arguments. And the role of the system was when the debate starts, was really to navigate within this collection and find the most relevant principled arguments and use them in the right timing. So so, to make this more concrete what we mean by a principal argument, imagine that we are debating whether or not to ban organ trade or whether or not to ban the sale of alcohol. In both cases, the opposition may argue that if you ban something, you are at the risk of the emergence of a black market. So a black market is a principled argument that can be used almost in the same way in many different contexts. So one may naively assume that this is kind of a simple keyword matching thing. If we ban something, then the opposition is going to use the black market argument, and we should be prepared for that. But obviously this is far from true. So imagine a debate about banning breastfeeding in public. Obviously there is little risk for a black market in this contract. Or imagine a debate about banning internet cookies. We're not going to tee a black market of internet cookies if we band these. So the system really needs to develop a more subtle understanding after human language in order to be able to identify the most relevant principle argument and need use them doing a debate. And and this is, by the way, just what all this description is before listening to the opponent. This is just what we're going to say on our side. And and the most the most challenging part is really too uh to listen to the opponent. And it's some kind of a battle to the arguments generated by the opponment raised by the And we do that you using uh an arsenal of technique that most of them rely on the same principle. We start by listening to the world articulated by the opponment, and for that we simply use what's on speech recognition capabilities out of the box. But of course we need to go to beyond the world, and we need to understand the gist of the arguments of the opponent. And in order to do that we try using various smackloads to anticipate in advance what kind of arguments the opposition mind you and then listen to determine whether he did the opposition was making these arguments and then responded cold yeah. That calls to mind the question of the difference between, say, what's a sound argument versus what's a persuasive argument? I mean, we know from reality that often the most persuasive appeals and debates rely on just straightforwardly false claims and logical fallacies, or even on little emotional cues that have little to do with the matter at hand. I was thinking about how in live debates, if you can get a laugh at your opponent's expense, that's worth you know, a dozen sound arguments or claims. So to what degree can AI understand these sorts of persuasive appeals that that go beyond just like what kind of evidence you can bring and the appeals based on style you're right in in in in debate and in the methods. We know already from the ancient weeks that that we have free elaps, we have logos, and we have ethos, and we have afforts, and humans are using a mixture of these pilas when they are debating one another. And just as a quick clarification, logos, pathos and ethos are the types of appeals that were identified in the study of classical rhetoric. Where logos is appeals based on our logical arguments and evidence, Pathos is the appeal to the emotions or the passions, and ethos is an appeal based on the credibility or authority of the speaker. I mean, as you know broadly understood and and the technology that we developed, and and by the way, it should be stated that there is a rapidly emerging community of scientists across the globe that are investigating this kind of topic. It is all under the umbrella of this emerging field, yeah, referred to as a computational argumentation. And when we started in two thousand and twelve, there was a handful of teams pursuing that, and we see a very dramatic increase in the result in these areas of the last few years is very I think from exacting and as I mentioned, the technology that we developed a most focused on logos, and you can see in the debate between proper Debate and Hali. By the way, this this debate is is fully available on YouTube, and you can see that indeed a woman is better in making in using path as and perhaps in using ethos and it is harder for the machine. And indeed most of the research being done by by the by the relevant research communities around logos, but there are already attempt trying to model and to capture additional aspect of path of and ethos in all the further enhanced this kind of technology. So another question I have is debater has to source claims and facts and arguments from existing written work produced by humans, which of course we know can be full of all sorts of flaws. Is there any way at this point for it to to have an analytical function to tell a say, factually true claim or a logically valid argument from just something that is wrong or dubious but repeated a lot in writing, or are we not there yet? This is a very kindly important and difficult problem, and that is receiving going attempting over over the previous teams and go to tackle that. This is certainly not bullet bof and and the problem is is quite complex because one may say, you know, okay, fine, maybe I should only take my argument from highly credibally so and by boxy I can assume that that these arguments are our valid. But this is not necessarily the case. Right. You can see you can lead an opinion article in a highly respectable newspaper which is actually quoting a false argument that was made as well, and if you're not careful enough, you you might be your system is going to pull this argument without understanding that something is happening. So we try to develop and we actually part of Project Debate included some kind of filtering mechanism in order to to filter out these kind of cases. And the way we did that was really once a specific claim was affected and by the way to being ordered, the claim is not a full sentence. A claim is often only a part of a tentence. Even if you were able to detect sentence that contains a claim relevant one that supportal side out of the billions of sentences in the popos, you still need to find the coret boundaries after claim within the sentence, and you have hundreds of options and only all of them is correct. So this is just going back why this this problem is it so talenting? But until you do that and found this claim and asked what is the stance of this claim, and if the stance is supporting your side, you can still ask what is the stance of the full sentence? And if the stance of the full sentences in the opposite direction, you may suspect that something is going on. And perhaps this this claim is quoted in order to contradict and not because it is true. And then perhaps it is there it is safer to avoid using it. But but this is just one safety mechanism, and and the problem that you raise is actually a much more beneval one, and and I think many teams are working on that, and we try to address that as well. And I think it has many interesting dimensions because it is not even just about the validity of the argument. Often, when when you show people to arguments, they will agree that one of them is better than the other. But what are the underlying mechanisms that I'd ask to the one argument over the other, And how do you train an artificial as in system to make the distinction. This is kind of another example of the problems that welcome to them. I have a question about what could come out of AI research like this, because I would say, from my personal perspective, I think studying rhetoric and debate is extremely important, but not necessarily because getting into debates is a good way to figure out what's true and establish you know, the right thing to do. I think one of the most important reasons to study rhetoric and debate is so that you can understand how other people's arguments and persuasive appeals are operating on you, or are designed to operate you. A clear understanding of rhetoric can be a kind of suit of armor for going into you know, the world and seeing how political actors and business actors and advertising and all that is trying to affect you. Do you see project debate or serving any kind of educational purpose like this in the world today. So there are several levels by which I can I can answer that. The first one is that this kind of technology is is definitely relevant and we believe highly valuable in the context of education. You can imagine using the technology in order to build better arguments and more of all, to perform a more analytical and perhaps more objective analysis off complex and controversial topics. This is one aspect. There is another aspect, but often when we debate is other humans. There are many layouts that that are involved in this discussion. In this debate. What all of them are related? To the facts and to the arguments that we are raising. Perhaps we have history with that Belton, Perhaps we have history with ourselves that actually impact our on part and decisions. Perhaps other people are listening and this actually improvides contact, uh that impact what is happening. And we are curious about this option of the dating with the machine in the privacy of your office. Maybe this is a different form of a discussion that to some extent is perhaps all free of of external biases and maybe will enable treat some people to identify situations where they have a blind book and to better listen to the other side. So I think in this case the whole of the technology could be quite instrumental and positive. The false business applications that are also very interesting from the IBM perspective and uh, and this is another another dimension, another level by which we can consider the technology as exacuable. Again, big thanks to No One slow name for taking time to chat with us. And now we're going to go straight into our second talk on the subject with Madu Matt. Thanks so much for joining us today. Could you start off by introducing yourself and talking about your role at IBM. Yeah, absolutely, and really nice to meet you. Robert and Joe uh maduco Chi, vice President Offering Management in Data and AI IBM, And the role of offering management is really all about laying down the strategy and then delivering and executing towards such strategy. And I'm based out of San Jose, Sunny, California, excellent. So just to kick things off here, um, you know we're gonna be talking a lot about AI here, and it makes sense to to to really get into what we mean when we're talking about AI for business. How does AI serve business compared to the way it serves consumers. That's a great question to get started on. UM so redeveloped a thesis a couple of years ago about really how AI for business would be different from consumer AI. Think of consumer AI, which we all know work with our smartphones, smart speakers, social media, photos, everything what it comes. But when it comes for AI for business, it's really very very different. AI for business is all about automation, optimization and making better predictions, and it requires really a very different set of technical capabilities, like you would have to understand how to deal with language, have to deal with what does automation means, and then be able to have the explainability and trust up AI. UM. So that's sort of the big difference between commercial AI and AI for business. So we know that one of the big AI projects at IBM is Watson. Could you tell us about Watson and explain how Watson fits into the broader picture of recent advancements in AI. Sure you you might have heard of Watson, and our audience might have heard of Watson, which came out when we first did our UH in Jeopardy and people remember Watson from there. But fast forward, a lot of work done around Watson. Think of Watson as our definition of IBM AI. We evolved a lot um since then, and our strategic intent always has been to have what's an available anywhere meaning available on any cloud. UH. We have focused on Watson. UH we call with Watson meaning it's embedded in almost all your applications. So for example, UM, I use the world a lot for AI for AI. What does that mean? Like, how do we embed AI in our data sciences and in our data data platforms and such. The other parts of evolution has been you know, as I said earlier, from our AI for business is all about automation. How do we UH evolve into the workflow AI that matters for our clients and our our society. So the workflows could definition could be you know, customer care, uh in I t asset management, in your regulatory or compliance, in supply chain or in your planning and budgeting. Right, these are how you can really embed AI and that is where Watson has really evolved into. And we have also been delivering now Watson an AI capability in a in our integrated single platform we call cloud Pacer data. So a long way. We came from Jeopardy Days and then you just heard from nome where we landed with Debater. So speaking of Debater, what capabilities has IBM commercialized from Project Debater into Watson? So that's a great question. Um, A lot of commercialization has happened. We have uh pretty good rich set of products like what's an assistant, what's on discovery, what's on knowledge, language understanding? And I know the are just works, but let me just give a bit of a background on what what's an assistant is? What's an assistant is? Our conversational AI platform really helps provide customer fast, straightforward answer, accurate answers UM across any application, device or cloud right, UM and our discovery is all about enterprise search and AI search technology that truly retrieves specific answers to your questions while you're analyzing trends and relationships in the enterprise data. So we've been looking at debater and some of the key technologies. Let me give you an example of few UM like sentiment analysis. Uh, let me pose a problem statement, what does that really mean? So, for example, today Watson does not understand idioms or sentiment shifters, and neither does any other competitor operates out there also, So think of elements which include hardly helpful, over the moon, cold feet, UM all years. You know, how do you make that analysis and figure figure this out? What is the real context behind this? So what we have done with that is that now what's on leverages this debat technology and looks at these idioms and sentiment shifters and does the analysis starting with better understanding of this sentiment and analysis is one of the most widely used API s for us UM. This already exists today in our product portfolio. What's coming into the future is UM. It's around all around documents. So let me put a perspective around a problem statement. There are many regulatory documents such as contracts or security filings which contains important clauses that have really really serious business implications for example, payment terms, obligations made to regulatory bodies, or warranties. Such humans can spend countless hours reading and extracting the information so they remain compliant. Although we can provide some of the out of the box models for contracts and invoices and such, but it creates UM but client may still need to create their own element classifications of business classes. So the solution has been with our debaters birth based classification technology into these products so we can learn with few one samples to do new classification of elements. Business documents could include contracts, invoices, and procurement contracts. The end of the day, it really really excelerates the outcomes what the businesses would be looking for. UM. Other technology is around summarization. So the problem statement here is like when you're looking for information customer or employee who may have aggregate research from different sources, clicking through multiple links and pages and finding exactly what they need can be very very difficult, right. It can take months, weeks and months sometimes to your years. So with Watson and Debater technology, we can analyze variety of these sources and provide a summary or brief of the ideas and the information which is contained within UM that's coming up. We're going to be leveraging this technology in our Watson discovery portfolio in second half. The other interesting UM issues we see today is like in our traditional UH rule based systems for contact centers, it categorizes large fraction of calls in a very generic bucket like it says, you know, like not uncommon to see more than maybe of calling a call center for a bank, which says, hey, this this call was just made for generalized checking, and it prevents the company from creating any robust self service. So with Debater technology, now we can leverage advanced topic clustering, which enables users to cluster this incoming data in a meaningful topics of related information and automatically this can be analyzed. So think of discovery of a content minor which will be enhanced with this type of a technology to extract better topics from very large data sets and then make the topic extraction more business user friendly. So a lot of stuff. I give a lot of examples, but sort of the gist of all this is, Look, it's going to impact businesses real outcomes, right, It's going to save them time, is going to automate the process, it's going to remove a lot of human error which comes with it, and really speak towards the productivity. Is going to speak towards the clients UM and P as their own promoter scores and such, and so that's really the gist of what we're looking to drive out of the debater technology. If I'm understanding this correctly, this is interesting that it's interesting that this kind of functionality would come out of an AI debate tool, because debate and persuasion that will seem like the kinds of things that would be inherently the most difficult to master with AI, because you've got all these elements of style and subtlety things that are really difficult to quantify to make into two understandable data. But out of the debater technology, it sounds like you're saying that you're actually getting a lot of derivative technologies that are good at dealing with algorithmic types of text like legal documents. Am I getting this right? Like that you could have a piece of software that works like a lawyer. Uh, and it can explain this contract to you when it's going over your head. And this kind of thing is possible now because of how formulaic and algorithmic legal documents tend to be. Would that be a correct understanding? Yeah, no, totally And if I may, UM give you one of the client example, especially as you started talking about legal UM, Legal Nations platform actually provides this in house legal teams and outside console the ability to respond to their lawsuits UM and draft their initial round up discovery requests literally less than two minutes right um and which shaved off about ten hours of attorney times on each of these lawsuits. So the real direct outcomes of usage of this technology. So you've been talking about big business applications, but I also wonder about applications directly for the consumer. Where, for example, because you have a program that ingests legal documents, so you you feed it some contract you're thinking about signing, and then you say, I have a question because I'm not a lawyer, I don't understand what I would be bound to do under this agreement. And then you could feed the contract in and pose questions to your AI legal assistant in natural language. Can you see a future like that. We do, and we already have a product like what'son Assistant, which is for customer care. It feeds on a lot of you know, pre train models, like especially now in COVID nineteen right, Uh, a situation where our government offices and our healthcare are getting in dated by calls. Right, So leveraging this UM what'son Assistant in front is really helping them deflect a lot of those phone calls and get the accurate answers in hands of the consumers. So you know, this is what we are focusing on around customer care. But yeah, in the future, I mean this similar technology and leveraging UM the from debater, we can actually go into any domain. Right, we have the right framework and we have the right technology to go pursue those different domains. I guess this sets us up for a bigger question, which is what is the overall role of natural language processing in the landscape of AI today and also which are the elements of natural language processing that we've really gotten a lot better at and which are the ones that are still a major challenge. Yeah, great question. As we all know, right, language have existed. I don't know a hundred thousand plus years. You know, started as speech probably people started to talk and the writing came perhaps much later. Um, and we write in ways we don't talk also, right, it's a lot more descriptive and more reflective. And so now with things where we can compute at larger with open data sets and transfer learnings, n LP natural language processing really really is the inflection point, right, And some of the examples I shared earlier around the sentiment analysis and summarization and clustering, these are all such critical aspects of taking LP, not just natural language processing, but natural language understanding, natural language generations is all going to come through all of that. And we really think with the Debater technology it really puts us in a in a leader quadrant here a lot more work to be done, but the the end goal is yes, we can continue to research on these things, but how quickly we commercialize it and how quick quickly we help our clients and users to see the outcomes what are needed here and make them a lot more productive. So how many languages does Project Debater and Watson together, how many do they understand support today? We started with obviously English, we are expanding now to French, Spanish, German in in the second half of this year, and then very soon will expand to Dutch, French, Arabic, Chinese both traditional and simplified, and Italian. UM And obviously we are choosing these based on where we are seeing most of our growth and an adoption. What are some additional examples of how these commercialized capabilities can be used by clients? Great question, um. I gave you an example earlier on legal missions. The other one, which is very close to my heart is um RBS with Watson. Watson RBS built Cora, which is it? Which is their digital assistant that helps better serve their customers through first time problem resolutions. Cora is trained with the were one thousand responses to more than two customer queries. However, if she doesn't know an answer or she sends that customer is getting angry or frustrated, she will transfer it to a live agent. Now, with improved sentiment analysis from Debater, as I mentioned earlier, we hope that clients like RBS will be able to better serve their customers by having digital assistance that better understand the subtleties of the of the sentiments of the clients. So for example, the phrase over the moon might be interpreted as literally about the planetary satellite and not as excited or elated. Right. So this is what with Project Debater core AI built into IBM Watson, it can understand these idioms helping clients like RBS to better serve their customers. The other example switching into financial like Credit Mutual, they had over five thousand branches and they receive more than three fifty thousand online inquiries a day and the volume is growing at least twenty three percent a year. So now with Watson infused email analyzer, they can help deflect and address of the three daily emails received by banks client advisors. So the implementation of the topic clustering from Debater, we believe now clients with similar needs that Credit Mutual will enable more self service by identifying clusters are commonly as topics and can be converted into self service content. Right. So to me, the examples like this are just amazing because I can totally then connect the dots between technology, the usage and the outcome, right, a win win situation. We've got multiple other examples as well, Roberts, and we're going to continue to be focusing on how do we really not just commercialize it, but I believe in AI is really meant to improve our society as well, right, make us more productive and do better things, especially the world we are living in with COVID and other things which are happening around us. Right, Um, the goodness of AI needs to be there, so very critical overall, what do you see as the best possible role for AI, not just as a tool for business, but as a society. What could it do for us in the best case scenario? Yeah, I mean that's a great question, right um. To me fundamentally, I mean there are many examples, but one most critical which comes to my mind is how AI can really help us detect bias? Right, A lot of our data sets and it has been built by humans with unbiased goes into those data. Right, AI can really start separating that help us detect bias and and make our products better, makes our society better. So that to me is the would be sort of the holy grail if I can achieve that. All right, So there you have it. Thanks once again to know I'm slow name and Maduka char for taking time out of their busy days to chat with us about this topic. For more information on smart Talks, visit IBM dot com slash smart Talks, and if you'd like to learn more about n LP, you can go to IBM dot com slash Watson, Slash Natural dash Language, dash Processing. And if you would like to learn more about our show, well, you can find us wherever you get your podcasts and wherever that happens to be. Just make sure you rate, review and subscribe. Huge thanks as always to our excellent audio producers eth Nicholas Johnson. If you would like to get in touch with us with feedback on this episode or any other, to suggest a topic for the future, or just to say hello, you can email us at contact at stuff to Blow your Mind dot com. Stuff to Blow Your Mind is production of I Heart Radio. For more podcasts for my Heart Radio, visit the i Heart Radio app, Apple Podcasts, or wherever you listening to your favorite shows,