The big promises — and hidden challenges — of AI

Developers of Artificial Intelligence have made big promises about how transformational AI will be. But what it can really deliver may be something else — and that difference could impact your life.

Arvind Narayanan, professor of computer science at Princeton University. Director of the Center for Information Technology Policy. Author of "AI Snake Oil: What Artificial Intelligence Can Do, What it Can’t, and How to Tell the Difference."

Part I

MEGHNA CHAKRABARTI: The artificial intelligence industry is already enormous. Generative AI, just one type of AI, is predicted to be a $1.3 trillion market by 2032, according to Bloomberg. And it is already pervasive. Banks are using AI to determine loan approvals. AI is integrated into manufacturing robots.

It helps you spell correctly when you send an email. And it helps you open your phone with an image of your face. AI is also now part of the job hiring process.

KEVIN PARKER: To increase diversity and reduce human bias in hiring, we've added artificial intelligence to make video interviewing even better.  Now, every candidate gets the chance to interview, and the interviews are consistent, fair, and based on objective criteria.

CHAKRABARTI: That is an excerpt from a promotional video featuring Kevin Parker, CEO of HireVue, a company that promises to use AI technology to evaluate candidates in order to cut down time and cost for corporate HR departments. And HireVue isn't the only one in this space.

According to the market research firm Grandview Research, the market for global artificial intelligence in HR in 2023 was estimated at $3.25 billion.

Braintrust is another one of these companies. Co-founder Adam Jackson promises in another promotional video that their AI will start with writing a client's job posting and then manage a significant part of the hiring process.

ADAM JACKSON: The Braintrust AI will analyze these applications and figure out which ones should go onto a live video screen.

Let's say there's 93 of them out of the 400. Braintrust AI then takes over and will schedule a live interview with each of those 93 and then actually conduct the interview for you. During the interview, the applicants are asked questions about the role or about their background. It's very conversational. It's very similar to how a human would conduct the recruiting interview. 

CHAKRABARTI: You did hear that correctly. A human does not conduct the interview. The job applicant is actually talking to an AI system, which also then does some analysis of the applicant's performance.  Now, this is a perfect example of the double-edged sword that is any major advancement in technology.

The advancements make humans more efficient. Because they largely free up time, energy, and money that would otherwise be sucked into drudge work that is, in fact, let's admit it, better done by machines. However, that quest for efficiency can go too far. And in this case, does an AI system really do a better job at interviewing humans than actual humans do.

What are the ethical implications? Arvind Narayanan is a professor of computer science and director of the Center for Information Technology Policy at Princeton University. And he's co-author of a new book called “AI Snake Oil: What artificial intelligence can do, what it can't, and how to tell the difference.”

Professor Narayanan, welcome to On Point.

ARVIND NARAYANAN: Hi, thank you so much for having me.

CHAKRABARTI: Okay, so HR and AI in the world of HR is what galvanized you to write “AI Snake Oil.” Can you tell me that story?

NARAYANAN: This was five years ago. I kept seeing products like the ones that you just talked about, where the pitch was, to these HR departments, look, you're drowning in applications.

You're getting hundreds, maybe a thousand applications for each open position. You can't possibly manually go through those CVs, do those interviews, let our AI take over. And in a lot of cases, the pitch with these video interview products was that the candidate would just upload a 30 second video of themselves talking about not even their job qualifications, but their hobbies and whatnot.

And the AI would supposedly use body language and facial expressions and things like that, in order to be able to figure out the candidate's personality. And using that, how good a fit they would be for the job or what their job performance would be. And I looked at that and went, huh?

CHAKRABARTI: Listening to you going, Oh my God.

NARAYANAN: I looked around and there was no evidence that anything like this can work. And it's an affront to common sense, I would say. So I basically stood up and said that. Coincidentally, at that time, I was invited to give a talk at MIT. And I said, look, there's a lot of AI snake oil going around.

I don't think products like this can work. These are elaborate random number generators, is the term that I used.  I had a surprising aftermath to that talk. I put the slides online and they went viral and things snowballed from there.

CHAKRABARTI: Okay, so professor, let's get down to some fundamentals here. Because I've always believed in this, I think it's a scientific truism, right? That what you choose to measure is an important part of what determines the outcome of those measurements, right? So in this case, what are some of the promises that these HR AI interviewing software companies are promising that they are actually measuring, in a 30 second clip of someone talking about their knitting hobby?

NARAYANAN: So I don't think it's something that they would be able to easily explain. The pitch sounds very nice. Oh, we're increasing diversity. We're minimizing human bias. We're increasing efficiency. But what are they measuring? I have not really seen a good answer to that.

The way that these machine learning systems often work is you train it on past data. Previous job candidates, and whatever features the AI extracted from those video interviews, which is usually not clear, how those correlate with what performance reviews those candidates got later on, or some other measurement of that sort.

CHAKRABARTI: Okay so here's a case, though, where I'm going to state everything I'm sure you've already thought of.  A system that promises to root out bias, human bias, in a hiring and interview process, is training itself on data that was generated by the biased process of humans making these hiring decisions and writing the performance interview, performance reviews.

NARAYANAN: Ultimately, it always comes down to humans somewhere in the loop, data generated by humans, or it's the developers are choosing what to measure. You can't really take humans out of the process. I think that's common sense. All you can do is hide it behind this veil of technology and put some moral distance between yourself and the judgment that inevitably comes into play.

CHAKRABARTI: But you see what I'm saying, right? If they're promising something that, and look, bias in the hiring process is a real problem. I'm not diminishing that. It's a huge issue that a lot of companies and corporations are trying to do better on.

And we don't want humans to be making decisions about a candidate based on their hairstyle, their race, or if the candidate was just having an off day. And I don't know, maybe just had an itch on their arm and that gave them a twitch for that day. We don't want that to happen. And yet, it's this imperfect data set, though, that is rife with bias, that is determining how the AI is going to decide.

And I guess what I'm saying is that there’s a chance that it produces the exact opposite effect in terms of it doesn't eliminate bias, but might actually supercharge it.

NARAYANAN: That's quite possible. I will say in the company's defense, that when you have this kind of algorithmic decision-making system, you can tweak it so that you ensure that the ratio of maybe the gender composition of the people who are selected for live interviews matches the gender composition of the applicants, things like that.

Those things are possible to do. And I think in many cases, these companies are taking those steps, but what we fear is that they're doing so by essentially making arbitrary decisions. If you have, again, a random number generator, it's easy to make sure that it's unbiased. But if you look at the kinds of things companies are measuring in these processes, video interviews is one.

There are others where the candidate is asked to play a game, where the AI is looking at how long they take until they pop a balloon. And that somehow measures your risk preference. If you wait a long time, you might get a big reward, but also the balloon might pop before you choose to do so, and you might get no reward.

And this somehow measures how good you're going to be at the job. And if what we have replaced is a biased system with something completely arbitrary, maybe that's better. Maybe it is decreasing bias, but I think we should be honest about what we're doing.

CHAKRABARTI: I see. Okay. That's why you keep using the random number generator analogy, right?

Because sometimes a random number that is generated can be the right one, right?

NARAYANAN: That's right.

CHAKRABARTI:  So what, so is that your primary concern here with the influx of AI systems or companies in the HR space? What is the snake oil here that you're concerned about?

NARAYANAN: It's not just in hiring. It's all over the place. The most, perhaps, high consequence applications of this kind of logic of using AI and machine learning to make predictions and decisions about people, those are in the domains of criminal justice, in the medical domain. In criminal justice, when a defendant is arrested, their trial might be months or years away, right?

Should they spend that time in jail? In jail or should they be free? And this is an enormously consequential decision. And I think more often than not in our country, we have algorithmic risk assessments that are being used in large part to inform or make those decisions. And what the algorithm is predicting is how likely you are to show up for trial versus being a flight risk, how likely you are to commit a crime if you're released, and so on.

And again, when you look at the accuracy of these systems, here we have a lot of data. Unlike the hiring companies, since this is more in the public sector, through FOIA requests, through Freedom of Information Act requests, we have a lot of data, and we know that the accuracy of these predictions is only slightly better than random.

And that should not surprise us. We can't predict the future. Because who's going to commit a crime is not determined yet, and somehow, we've decided to collectively suspend common sense when AI is involved. The way I would put it is that these criminal risk prediction systems, mostly what they're getting at is that people who have been arrested before are more likely to be arrested again.

CHAKRABARTI: We didn't need AI for that.

NARAYANAN: It goes back to your point about measurement. The AI can't actually measure who's committing a crime. That is not observable. It can only go based on who is getting arrested. Essentially, it's replacing the biases of judges with the biases of policing. It's not clear to me that's an improvement.

I don't think we should be using these algorithmic risk assessment systems in criminal justice.

CHAKRABARTI: So criminal risk prediction systems, I think they made a movie about that starring Tom Cruise. But Professor Narayanan, hang on for just a second. He's author of AI Snake Oil, and we will discuss a lot more about the promises and where AI actually can fulfill and fall short of those promises, when we come back.

Part II

CHAKRABARTI: Professor, before we get deeper into the other examples that you write about in the book, I just want to make it clear, you're not one of those Chicken Little types here about AI, right? This isn't a book where you're screaming that like the robots are going to take over and all is lost.

NARAYANAN: Certainly not. And if our point was that all AI is bad and dangerous, we wouldn't have needed a whole book to say it. The whole point is to break down different types of AI, look at what can be useful to us and what can be harmful, risky. And that risk is not about robots killing us, but it's more about the kinds of examples.

We've discussed automated hiring, criminal justice, that sort of thing.

CHAKRABARTI: Yeah, I wanted to make that clear because I even find myself doing this sometimes, I have a natural pessimism about the extreme promises that are made early on in the era of new technology. There are some very influential voices though that have been sounding alarms about AI.

And I'll ask you about that in a minute, but to make something else clear. The previous examples we talked about were in the realm of predictive AI, right? And there's another example I want to go over in just a second. But in all fairness, what would you say are the potential benefits or uses, the non-snake oil aspects of predictive artificial intelligence?

NARAYANAN: Predictive AI is about using AI to make predictions about people and decisions based on those predictions. And this can be very valuable in many cases. One story is from Flint, Michigan, where there's this horrendous issue of lead pipes that need to be replaced, but the city has a limited budget, so they really need to be careful about where they do the digging to even figure out if there's going to be a lead pipe, and then to spend the money to replace it.

So here's where technology can be very useful. Researchers built an artificial intelligence system, a machine learning model, whatever you want to call it, that looks at existing data from the city where we know what kind of more likely or less likely to have lead pipes. And then predict which houses would you want to dig under to have the highest chance of finding lead pipes so that your money can be better spent.

So I think there are some risks here, but I think overall this is a system that's very valuable and there are many other examples of predictive AI that are either valuable or are the best option that we have. So when banks need to lend money they need to be able to predict risk.

Otherwise, they'll go out of business.

CHAKRABARTI: And those risk predictions often come from pretty concrete data sets. That's why we have these exhaustive credit histories, right?

NARAYANAN:  Yes, the data sets are a little better, but there is still the problem that the predictions are not going to be that accurate, because again, the reason that someone might default on a loan is because they might lose their job, and that might not have been something that is even predictable in principle. So I think despite those limitations it's the best option that we have in those cases.

CHAKRABARTI: I see. Okay. So, wait, going back to that Flint example, which I think is very compelling here. What is the difference between what is promised with that kind of technology versus what was promised with the HR examples that we started off with?

Because just trying to use those two as a way to drive to your overall point that, like, we need to learn how to tell the difference between useful and fanciful AI.

NARAYANAN:  Definitely. So in the Flint, Michigan example, when you look at the numbers, the accuracy that you can get with trying to figure out which houses might have lead pipes is much higher.

So I think it does change the moral calculus if the accuracy is so high that we're able to justify, that we can relatively reliably make this decision. We're not digging under your house because there's only a 3% chance that there would have been a lead pipe. As opposed to in the job situation or in criminal justice, it's really anybody's guess. Whether someone succeeds at a job might depend much more on the environment than the intrinsic characteristics of the candidate, which, of course, also plays a role.

So that's one difference. And I think the reason the accuracy can be so high is because in the lead pipe example, you're not actually predicting the future. You're predicting the past in a certain sense, right? Which pipes, which houses had lead pipes put under them 50 years ago, and that data has now been lost, and you're trying to reconstruct that data.

That's a fundamentally more tractable technical problem. And I would say a third difference is that this is really about helping people, and there's a limited resource, which is the city's budget. ... And the criminal justice example, for instance, it's not necessarily a limited resource.

Criminal justice reform, of course, is a big movement and people are talking about what are ways in which we can decrease the need for a bail and money bail altogether. So there are other options there. So the availability of other options is another major factor.

CHAKRABARTI: Okay. So let's talk about one more predictive AI example, which, I have to say, stopped me in my tracks, because I had never heard of it before.

And it was, it's something else. This is, it's called the Allegheny Family Screening Tool. And it's a program that was developed for use in helping evaluate child welfare cases in Allegheny County, Pennsylvania. Starting in 2016, the program was used by social workers to determine which families ought to be further investigated.

So here's a clip from the county's website.

And it features Professor Rhema Vaithianathan, and she's explaining the tool's purpose.

RHEMA VAITHIANATHAN: It tries to give idea to a frontline worker, given the background history of this family, what the chances are that this child will be removed from home. The methodology we use to create the algorithm, in very simple terms, is we do a statistical technique called data mining, where we're basically looking through millions of patterns in the data, in historical data, seeing how it correlates to outcomes, and then matching each new case with those patterns that we've seen in the previous data.

CHAKRABARTI: Professor Narayanan, I have to say the chipper music in the background produces a major bit of emotional and cognitive dissonance for me in that she's talking about using an AI predictive system to decide which children to remove from homes. Talk to me more about this example.

NARAYANAN: Yeah, so this is a very hard example to talk about.

I think this is also different from the ones we've talked about so far. I know some of the people who are involved in building this, and I think they, if one is going to use an AI system for informing this kind of decision, they did the best job that one could. They thought about a lot of the ways in which things can go wrong.

And to me, it's not a clear-cut case of saying that we should not be using risk assessments here. So that's a different view that I have, compared to some other uses of risk assessment here. With all that said … I think there is a lot that can go wrong. For instance, one of the things we point out in the book is that the slice of the population on which these tools are trained is not necessarily the same on which it is being deployed.

There are systematic differences. Because of that, when you build and validate the tool. You can't be confident about how accurate it's going to be when you deploy it. So you have to, once you do deploy it, you have to keep monitoring it, spend a lot of effort in follow up and so forth.

So there are dangers here. I think there are ways of minimizing those dangers. But even with all that said and done, I think it's a very fraught use of algorithms, and the reason has less to do with the algorithm itself and more with the fact that the system overall, the way it works is that if a child is predicted to be at great risk, they're removed from their family and placed in the foster system or something like that.

So really, I think the concerns come from the design of the system itself. And I think reforms also have to talk about what can we do to change that fundamental dynamic.

CHAKRABARTI: Those are always the concerns. The fundamental design is the heart of every piece of concern or every avenue of concern for technology, but especially AI.

I want to just note, though, that at least as of 2023, the Department of Justice began investigating the Allegheny Family Screening Tool because of the concerns of the potential impact if it was aiding in making incorrect decisions about child removal. So this was quite a serious thing.

I'm not sure if that investigation has completed yet. Do you know anything about that?

NARAYANAN: I don't know. Okay. But yeah, that is very much worth pointing out.

CHAKRABARTI: Okay. We will look around and see if we can find more information about the status of the DOJ investigation here in a second. You have actually said, you've used a specific word a couple of times in this conversation already that I think is important, that one of the things. And I'm inferring here, so correct me if I'm wrong, but one of the things the use of these AI tools can promise is a sort of moral distance. You used the word moral a couple of times.

NARAYANAN: Right.

CHAKRABARTI: Between the humans involved in a system and the decisions that they have to make, it provides them a moral distance between those two things, which is a lot of people actually want. Can you talk more about that?

NARAYANAN: Yeah, that moral distance is very appealing to decision makers. So there was another case study we were looking at recently with organ transplant algorithms.

So these days when a person dies and an organ becomes available for many types of organs, right? Hearts, livers, et cetera, there is some kind of algorithmic system. I don't necessarily want to call it AI, but the issues are largely the same. There are algorithmic systems that are used to determine who should get that particular organ? And that's based on, sometimes it's based on a calculation of who might benefit the most from getting this particular organ. So we were looking at the UK's liver transplantation system.  And there are big moral questions here. Because to some extent, you want to incorporate predictive considerations.

Who is predicted to live the longest if they get this liver? You also want to incorporate other considerations that are more fuzzy. So for this person, their liver disease is because of their behavior pattern, for instance, of alcoholism. And if they're the ones to get the liver, that pattern might recur.

And so the benefit might not be that high. And also, how should we take into account their past behavior and making this decision? Who would want to be the decision maker? Sitting in the national health service in the UK, who is programming in, right? This should be the factor that encodes society's distaste or a penalty towards alcoholism.

And that this is the amount that person should be deprioritized. That's never going to happen. And because you don't want to manually program in these things, there is a big appeal of using data driven systems where the decision makers can say, we have deferred this whole ethical conundrum to the data.

The question of who is most deserving. Or who will benefit the most from getting this liver. And so this kind of multi-dimensional ethical quandary just gets collapsed into one single mathematical calculation that accounts for some factors, but not others that society might want to consider in making these difficult decisions.

And we think that's definitely a problem. We're not saying we shouldn't use algorithms. But there needs to be more public understanding and debate about how we're making these decisions as a society.

CHAKRABARTI: Exactly. We're going to come back to that in the final part of the show here. But I have been focusing on predictive AI a lot here.

There's a whole other field of AI, about generative AI. And I believe this is one example of that.

This is Joshua Browder. He's the founder of a company called DoNotPay, and he's a champion of AI technology in the legal field, and Browder says he built his company in order to help consumers have greater success in the law. And as of 2023, the company DoNotPay has claimed to have resolved over 2 million legal cases successfully.

Last year, Browder positioned himself as an anti-establishment entrepreneur. ... And he says that lawyers are resistant to AI in the courtroom because it shows, in his estimation, that a lawyer's job is just largely copying and pasting paperwork.

JOSHUA BROWDER: And so it's a combination of loving rules, being worried about their profession, and also just disliking these young kind of CS students trying to take them down.

CHAKRABARTI: Now, Professor Browder also tried a publicity stunt that caught your attention. Can you tell me about that?

NARAYANAN: Sure. Yeah, so this particular publicity stunt was saying that the company DoNotPay would pay $1 million to any attorney who used the supposed robot lawyer. That DoNotPay had built in order to argue a case in front of the Supreme Court.

And the way that would happen is the lawyer would wear an earpiece and the robot lawyer would tell them in their ear what to say to the justices. And first of all, there's no evidence that this robot lawyer exists. And also, we can infer that this is just a publicity stunt because electronic devices are not even permitted.

And so this was never going to fly, even if the technology existed. And so I don't believe they were ever serious about it, but they did manage to convince a lot of people that they had built a robot lawyer. And I think that's very problematic. They did get into trouble with the Federal Trade Commission.

And they settled the FTC's investigation. And if you look at that complaint document, there are so many juicy details of the ways in which this company had made up stuff. So this is an example of generative AI, things like ChatGPT. So backing up a little bit, we make a big distinction between generative AI and predictive AI.

We're not nearly as skeptical of generative AI as we are predictive AI.

We ourselves, me and my coauthor, Sayash Kapoor, we are heavy users of generative AI in our own work.

And as people who often do computer programming, it has just really changed how we go about that. It's hard to even imagine going back to a time before we had the assistance of AI in order to write code, not because it does a better job than us, but because it takes so much of the drudgery out of it, right?

So definitely want to acknowledge that potential in the long run. We're broadly positive about generative AI. But along that path from here to there, I think we're going to encounter lots and lots of wild claims. And I think this is one of them. And if I may, the last thing I want to say, is that the claims that lawyers are so resistant to this, because they feel threatened, is complete nonsense. Legal tech is a very mature industry. Law firms are very eager to try to get efficiency gains from incorporating this technology and things like LexisNexis and Westlaw and so forth. And those have been very successful at automating or partially automating, again, the more mundane aspects of the job, but not of the creative aspects.

CHAKRABARTI: Yeah, and this is exactly where technology does excel. But when we come back, Professor, again, we're going to want to dive into your so your prescriptions, if I can use that phrase, on what we as consumers of this technology should do to get the most out of it. So that's what we'll talk about in a moment.

Part III

CHAKRABARTI: Professor, so what is AI? To put it simply, is your concern that right now we're in a phase of AI technology and business development where companies are frequently over promising or claiming that their technology can do things that it can't yet do, or it can't do well?

NARAYANAN: That's exactly right. That is it in a nutshell.

CHAKRABARTI: Okay, so it's like everything's Theranos until it's not. It makes me wonder if Elizabeth Holmes, if she had only done AI and not actually a piece of hardware, maybe she would have, she could have stayed out of prison. But I'm joking about that.

But what, so why is this allowed? Why is this happening? Why are companies so easily, there's a lot of work going into their products, I get it, but they're so frequently putting out products that are not actually capable of doing what they're promising.

NARAYANAN: In a sense, it is not allowed, like I was saying earlier, this robot lawyer company got into trouble with the Federal Trade Commission, a number of other companies have, and I think it takes a while for regulators to adapt to a new technology, even if they're just enforcing existing laws on truth and advertising, for instance.

So I think we'll need some new regulation, but also regulators to really start focusing more on these overhyped AI claims, that is happening. It'll continue to happen. But I think a big reason for that is there is genuine progress happening in generative AI. The mathematical and engineering techniques are advancing quickly, but there is a big gap between that and having useful applications that can do things for us, that we want to get done in our everyday lives or in our workplace.

And there's a further gap between different types of AI, right? Generative AI versus predictive AI, versus social media algorithms. And so since people are confused about all of this, companies are able to exploit that confusion without explicitly making false claims.

CHAKRABARTI: Do you mind if I offer some like a philosophical interpretation of this for a second professor?

NARAYANAN:  Go for it.

CHAKRABARTI: Because I was thinking about this. I think it was Google a long time ago. If I don't have this right, it's a major company a long time ago that first started talking about perpetual beta, right? Like digital technology allows developers to do something that no other kind of product or technology was ever able to do, that existed in the hardware world, the physical world. And that is you could put a product out there, that functioned, but it didn't function to the best of its ability because you would just be able to constantly update it, right?

Like nobody questions that your apps on your phone are being always updated. It's not even weekly. Sometimes it's daily. Now we've just been habituated to that. We've been habituated to getting digital products that don't do a great job at the beginning, but we just think, Okay give them a couple of months and several updates and version like 6.7 will be better. And I wonder if that sort of societal reduction of expectation now is simply supercharged with AI? And that it's like it's more acceptable to put out a product that may not work perfectly at the start, because we're just going to assume that AI is developing so fast that it will catch up to its own hype.

NARAYANAN: That is absolutely true. And I think this perpetual beta thing made, I think, a decent amount of sense with traditional software. Because you would have a core of the product that was mature and working well, and new features that the company was trialing were rough around the edges and they would be getting data on what should be improved about those features.

So it was not like the product was useless if there were some unfinished features. But with AI, we're seeing something different. Companies are not even necessarily figuring out, with generative AI, what is useful for before putting it out there. And I can see why, they're putting models like the GPT models behind ChatGPT out there, and saying, you figure out what you're going to use this for.

We're just giving you this chat box and it's a terrifyingly powerful user interface. You can put whatever you want and get results out that are good for your work or your life. I think that has not worked that so far, because people are not necessarily aware of the serious limitations.

Lawyers many times have gotten into trouble for submitting briefs to courts that had AI generated so called hallucinations, that have made up fake citations, right? And if you're not thinking about this as a product, but as a technology, you're putting directly into people's hands, it's something beyond perpetual beta.

It's like putting a buzzsaw in people's hands. It's very powerful. We're not going to tell you what to use it for, how to use it. If you figure it out, it can be very powerful, but otherwise you're going to get hurt, and it's not our responsibility.

CHAKRABARTI: Is there another wrinkle here with AI in that when companies are asked why didn't it work?

Why did it produce these hallucinations that have led me to lose my court case because all my briefs got thrown out, that they oftentimes can't tell you? Either they won't because of proprietary reasons, or they don't actually really understand how their own AI works.

NARAYANAN:  I think that that is true to a certain extent.

I think there is a lot more research that companies could and should be doing that goes into how explaining AI works. But culturally, the AI research and research community as well as industry has been about, build first, ask questions later. And so we've had many cycles of this where a technology starts working and people start using it.

And then researchers come in and try to figure out why is it working? Where does it fail? And that sort of thing, which is, by the way, great for people like me. It's job security. A lot of my research is about better understanding and explaining these limitations, because the companies themselves are not doing that.

But that said, I think part of the reason is less about understanding and more about just being transparent about the limitations. ChatGPT will tell you in very small fine print that it can make mistakes, but what they should have done from the get go is when someone asks a question about the election, for instance, to say, this is a tool that's not reliable for this.

Go to this election website to get reliable information. They were only forced to start doing that after many media organizations and other public advocates put pressure on them to do so, I do think they should change their behavior in this regard.

CHAKRABARTI: So the pressure had to come externally. Okay.

But that leads to a really interesting notion you put forth in the book about, broken AI appeals to broken institutions. What did you mean by that?

NARAYANAN: And this goes back to a lot of the examples we were talking about earlier. So if this HR product is not as accurate as claimed, even if it's a random number generator, even if the HR department knows this, they're still going to buy it. Because otherwise they just literally don't have a way to go through all these applications, and any sort of band aid they can put onto the situation feels like a big relief to them. And it's not necessarily helping the problem, it's just leading to an AI arms race.

Candidates are now using AI to automatically generate resumes and send them to as many different jobs as possible. It's just an escalation we're seeing now. But you can imagine how from the perspective of the HR department or colleges, sometimes are using these AI tools to screen students in some ways. But when higher education is in such financial distress or media organizations, who are forced so much to cut budgets, it's very tempting to try to look at, Oh, can we have an AI reporter instead of instead of a human reporter?

CHAKRABARTI: Ladies and gentlemen, I am not yet a bot. I'm just gonna say that.

NARAYANAN: (LAUGHS)

CHAKRABARTI: (LAUGHS) Not going down that path as of yet. So really interesting, because the mutual brokenness that you're talking about, it comes back down to something very human also, which I think, I keep thinking of the HR example, it's FOMO, right? Once there's a technology out there, there's a lot of corporations who are like, my competitor's using it, we should use it. Which kind of, I think, amplifies or accelerates the problem.

But this takes us back to the set of solutions. That we as a society, and again, as individuals, should be seeking in order to be able to reduce the impact of the AI snake oil, as you call it. One of them is regulation, and the regulation also has to incorporate some sort of set of ethical guides.

But you also point out that you believe the same companies creating these AI products already have an outsized influence over the very ethical debate over their own usage.

NARAYANAN: That's right. And I think we all have a part to play in changing this. I think societally, culturally, we're too deferential to these companies.

One reason is that we think, Oh, these are tech geniuses and what they tell us about the technology must be right. But when you look at the evidence, I think these highly technical folks, if anything, are worse than the average person in anticipating what the societal effects are going to be, or thinking about what the limitations are going to be when you take it out of this toy lab setting and use it in the real world. So I don't think we should be so deferential to them. And I don't think policymakers should be so deferential to the power that these companies have commercially. And I think antitrust enforcement is really important. So all of that is one aspect.

My second aspect is that I think as individuals, we all should be doing more to educate ourselves, and this includes me, even as someone who has written a book about this topic, about, okay, what are the AI tools that I might potentially use, in my own work. Because there probably are going to be some that are useful, and so it's not a matter of blanket rejecting all of them, but to understand AI enough to be able to push back on commercial claims.

If we don't think a particular tool is going to help us. Third, I think as workers and companies, we can take collective action. Right now, too many decisions are being taken by management and that's based on FOMO. And I think workers advocating together can change that equation. And fourth, I think I will also have a role in our personal lives, as the parent of two young kids, is already a tool that makes certain learning interactions a bit more fun.

It's not central in any way. I'm not saying AI is essential, but it has been an interesting thing I've found to incorporate into my interactions with them, where we learn stuff. And I do think that as they grow up, generative AI is going to be a big part of their lives. So to be able to teach them a little things about the technology, right now, what is ChatGPT, that what is it capable of, that it doesn't have feelings.

I've enjoyed teaching my young children this, they're going to have to learn that at some point. And I do think it's better to start young.

CHAKRABARTI:  Yeah, ChatGPT doesn't have feelings. I've encountered kids who, once they realize that, they're just mean to the ChatGPT.

NARAYANAN: (LAUGHS)

CHAKRABARTI: Now, are these your avenues of change that you talk about in the book or not?

NARAYANAN: These are definitely avenues of change. We also talk about things that are more structural. What do we do about the fact that AI is built on human labor, text or images on the internet, which can be photographs, paintings, poems, right? Creative works. And that are taken without compensation.

And then further labor goes into annotating, as it's called, the information that goes into AI, and these are usually tens of thousands of workers in developing countries who are paid low wages, prevailing wages for their local economies. But nonetheless, the nature of this work is such that I think that's still unfair compensation, because a lot of the time, they have to filter out the gore.

Right? And the darkest parts of the internet before it goes into AI. So that's how they keep the outputs of AI relatively clean. Those are things we can't necessarily change as individuals. So I do think structural change needs to happen. There's a lot of things the media, governments should all be doing.

CHAKRABARTI: You know what? Actually, give me one more specific example of a structural change that you'd like to see and see rather quickly.

NARAYANAN:  Sure. The fact that there are a few large companies in the U.S. who have such outsized power, right? I think there are a few ways to change that, antitrust enforcement.

That is one that I was mentioning. But also, what would it mean for governments, for instance, to fund a wider variety of entities to develop and deploy AI technologies, whether that's academia, or grants to small businesses. There are a variety of ways to change this kind of concentration of power in the industry.

CHAKRABARTI: Got it. We've got a couple minutes left here, and I keep thinking, I'm trying to rack my brain for examples of what did we collectively do as a species, in fact, not just as a country, but as a species, when we created what is truly civilizational changing technology. And for me, the one that keeps jumping to mind, it may not be the best example, but is the splitting of the atom.

So we learned how to create, release nuclear energy, which can help go to power cities, but it also can help go to evaporating millions of people off the face of the earth.  That technology was pursued relentlessly and achieved. And it was after its use that we collectively as humanity said, Hey, now we know we can do this.

It has a very massive downside. Let's come together and come up with some treaties, some regulations essentially, over its ethical and moral use. So I see that as actually as an example of how good things can happen. Structural change can happen. With huge leaps forward in technology, with that in mind, do you have some optimism here about our ability to inject humanity into our development of AI?

NARAYANAN: We're absolutely optimists. If we weren't, we wouldn't have written the book, because what's the point? Let me, I think there are some parallels to the nuclear analogy. Let me give you another analogy. This one I often find myself going to. Which is the Industrial Revolution and electricity.

So in the wake of the Industrial Revolution, while it's true that eventually, it brought a great rise in living standards. What happened for the first few decades was the mass migration of labor from towns and villages to these big cities, right? Living in tenements, horrendous worker safety conditions, long work hours, poor compensation.

Workers didn't have collective bargaining power against capital. And so that is what led to the modern labor movement and lots of other structural reforms. And to me, that's a closer analogy to the kinds of harms of AI we're seeing, which are much more diffuse and prevalent across society, as opposed to the concentrated harms of nuclear technology.

And I think the reforms that we need are also more similar to the Industrial Revolution than to nuclear energy. Because it's not about putting it back in a box. It's about accepting it's widely available. And thinking about how to shape it for the better.