What happened when AI went after welfare fraud

Artificial Intelligence algorithms are being used to decide who gets welfare benefits, and how much. Some experts say it’s leading to “devastating” cuts in benefits for those most in need.

Kevin De Liban, founder of TechTonic Justice, an organization which helps low-income people left behind by artificial intelligence (AI).

John Maynard, principal industry consultant at SAS.

Brant Fries, professor emeritus of Health Management and Policy at the University of Michigan and president of inteRAI, a research group that provides state and governments with statistical predictive algorithms.

Part I

MEGHNA CHAKRABARTI: Elon Musk does not like Social Security, Medicare, or Medicaid. He was on Fox Business just this week, plainly saying what he'd like to do with those programs.

ELON MUSK: Most of the federal spending is entitlements. So that that's like the big one to eliminate.

CHAKRABARTI: Musk's Department of Government Efficiency, or DOGE, is reportedly using artificial intelligence to aid in those government slashing endeavors. Last month, the New York Times reported on Thomas Shedd, a former Tesla engineer who's now head of the Technology Services Department at the Federal General Services Administration.

Shedd reportedly told staffers that AI would be used as a tool to, quote, detect fraud and waste. Asked just last week if AI was indeed crawling through government databases, communications, and contracts, Musk told reporters, quote, Right now, we're not using that much AI. End quote. But that hasn't stopped congressional Republicans from celebrating Musk's use of AI, or his goals. Here's House Speaker Mike Johnson.

MIKE JOHNSON: He's created these algorithms that are constantly crawling through the data, and as he told me in his office, the data doesn't lie. We're going to be able to get the information, we're going to be able to transform the way the federal government works at the end of this, and that is a very exciting prospect.

It is truly a revolutionary moment for the nation.

CHAKRABARTI: It's true, the data doesn't lie, or the data as I prefer to call it, but the conclusions made by an algorithm based on what it finds in those data, that's a little different. Because AI is extraordinarily good at completing program tasks rapidly. It can analyze massive mountains of data very fast.

But should it then be making decisions about federal programs? Which ones to keep, which ones to cut, who gets benefits, and who does not? Globally. There are already examples on how this can go wrong without tight oversight. In the Netherlands, prime Minister Mark Rutte's government resigned in 2021 after a major AI scandal there.

An investigation found that 20,000 families were falsely accused of committing child welfare fraud. A court ordered the Dutch government to repay approximately $32,000 to each affected family. In Australia, a system called Robodebt accused 400,000 welfare recipients in Australia of misreporting their income, and those recipients were fined significantly.

Robodebt was implemented in 2016, and by 2019, a court ruled that the program was unlawful. And the government was forced to repay $1.2 billion, according to Reuters. And here in the U.S., a lawsuit was filed and settled in 2024 against the state of Michigan, when an anti-fraud algorithm indicated widespread unemployment insurance fraud.

The algorithm was wrong, and 3,000 plaintiffs were reimbursed to the tune of $20 million. Amos Toh is Senior Counsel of AI and National Security for the Brennan Center for Justice. He previously worked as the Senior Researcher of AI at Human Rights Watch. And while he was there, Toh wrote extensively about the use of AI in welfare systems.

And he says the pattern of predictive algorithms and AI cutting Services is something that should concern everyone, not just the people receiving benefits from welfare systems.

AMOS TOH: It's no surprise that they became like they were like the first playgrounds for AI experimentation, right? And something we have warned for a long time is that what is being experimented on, when it comes to people of low income, who have less means of fighting back, will invariably be rolled out to the broader population.

CHAKRABARTI: This hour we want to take a much closer look at how AI could potentially help states and the federal government when it comes to welfare systems, what works, and also what doesn't work, and what impact could AI technology have on the future of social services? So Kevin De Liban joins us. He's the founder of TechTonic Justice, that's T E C H, T O N I C.

It's an organization dedicated to helping low-income families confronting the use of AI in welfare systems. And he joins us today. Kevin, welcome to On Point.

KEVIN DE LIBAN: Hello, Meghna. How are you doing?

CHAKRABARTI: I'm doing well. You have had some first-hand experience. Obviously, that's why we've invited you on the show today, with a very particular case of how AI had an impact on benefits received by folks.

Can you tell us a little bit about that story?

DE LIBAN: Sure. Medicaid is the state's only health insurance program for low income people and it will pay for in home care that people who otherwise would have to be institutionalized in places like nursing facilities would receive, in order so that they can stay in their homes and out of those facilities. And it's much better for their independence and much better for generally much cheaper. And so what happened in 2016, the clients on this program, who had been receiving a set number of hours for many years, suddenly started calling with news of devastating cuts, seeking help to fight them.

These are people who have cerebral palsy, quadriplegia, conditions that don't get better. And the state was cutting their hours from, say, 7 or 8 hours a day of care, which was really not enough, but folks would try their best to get by, to something like four or five hours a day of care.

CHAKRABARTI: Kevin Can I just jump in here?

DE LIBAN: Sure.

CHAKRABARTI: This is the state of Arkansas, correct?

DE LIBAN: Oh, yes. Pardon me. The state of Arkansas. Yes.

CHAKRABARTI: It's okay. Yeah. No and you were working at legal aid in Arkansas at that time?

DE LIBAN: Correct.

CHAKRABARTI: Okay, so go ahead.

DE LIBAN: Sure. So clients would be calling, reporting this devastating loss of hours that meant people were lying in their own waste. It meant people were getting bed sores from not being turned, just intolerable human suffering. The only clue that they had that something funny was up is when they would ask the nurse, the same nurse who came to assess them last year, and the year before, the same nurse who gave them six, seven, eight hours a day of care, why are you cutting my care?

And universally, the nurse would say, It's not me, it's the computer. And I heard that story from a couple dozen people, figured, ah, there's something funny going on here, not in the 'ha' way. And we'd look into it and find that the state of Arkansas had implemented an algorithmic decision-making system to determine how much care to give these folks.

CHAKRABARTI: Okay. When was this algorithmic decision-making system implemented?

DE LIBAN: January of 2016.

CHAKRABARTI: 2016. And what was the original purpose for it?

DE LIBAN: The state waffled a lot on what they said the justification was. Ultimately, it was to cut benefits. That was the purpose of it. They would give various justifications about wanting to standardize or objectify the process.

Things like this that we ended up proving were lies.

CHAKRABARTI: Okay, I want to know exactly how this played out. Because I've seen some stories here that said this algorithmic decision-making system was implemented, but were people who received the benefits, were they asked about like how much care did they need, how much did they receive, how often did they need specific types of care around the house, and then that stuff was inputted into this algorithm?

DE LIBAN: No, not exactly. So the first part of the process was a 286-question assessment. So the state nurse would come out and ask you 286 questions, usually taking an hour and a half to two hours to do, which is an exhausting process. At the end of that, the nurse would push a button and the responses from those 286 questions would be run through an algorithm that would group people into one of 23, what they call the acuity categories or severity categories.

And then each of those 23 categories had a fixed number of hours attached to them, that really couldn't be deviated from at all.

CHAKRABARTI: It couldn't be deviated from at all. I want to know more about that. Yeah.

DE LIBAN: Sure. So the way the algorithm works is of those 286 questions, it turns out only around 60, give or take a few, ever matter.

But they don't all matter in the same way. They only matter sometimes when aligned with other factors. So the best way I've learned to describe it, and this is almost ten years on, is as constellations, right? Sometimes the stars form something when you see them with other stars. But other times, they're just a star, right?

That's how the algorithm worked with those questions from the assessment that actually mattered. Some of them mattered some of the time, and there was never any explanation about which mattered which of the time. And the state couldn't explain it, let alone expecting beneficiaries who are on the program to suddenly understand how come their care is being cut drastically, when their conditions haven't been improved.

CHAKRABARTI: Now I'm seeing that at least at one point in time the state of Arkansas had said that the AI system was supposed to eliminate what we know is a potential problem, which is human bias, right? That people could be, could have favorites. They could increase the amount of benefit someone's getting for sometimes arbitrary decisions, right?

DE LIBAN: That's what they said, and that was one of the lies that we proved. So if you thought that was actually an issue, that somehow a nurse in one part of the state was giving more or less hours than a nurse in another part of the state, you would think there'd be a trail of that. There'd be some paper identifying that this was a problem.

Some nurse supervisor or some program supervisor would have talked to a nurse to talk to them about the issue. There would have been a little study, something. None of that had ever happened. No nurse supervisor had ever talked to any nurse about giving too few or too many hours. There was no paperwork showing that.

So the state was really lying. What they were trying to do is cut benefits. And that's what the algorithm did. And it did it drastically, and in a way that was just abjectly cruel, and it devastated the lives of several thousand disabled people.

CHAKRABARTI: Wait this is the thing about AI, right? It does what it's told to do, and it does it really well.

So did you present evidence in court? Because this of course, did go to trial. And you and the people you were representing won that trial. Did you find evidence that you presented in court that the purpose of the AI was told to cost cut?

DE LIBAN: Yeah. That's what it was programmed to do.

So the best-case scenario, so under the previous system, where nurses decided how much care to give people within guidelines, a step by the state, the maximum was eight hours a day of care. And again, that is not enough for most people with cerebral palsy or quadriplegia. Most other states actually provide significantly more than eight hours a day of care as a maximum.

When the algorithm came in, the best case scenario for most people was five and a half hours a day of care. So there was no scenario in which you got subjected to the algorithm and somehow won an increase. And that's where the shell game is of this, is your whole possibilities are structured to decide how much of a cut you get, not whether there's any chance of actually getting an increase in hours, at least for the people at the highest levels of need.

Part II

CHAKRABARTI: Kevin, hang on here for just a second, because we actually also spoke to the person, the man behind the company that developed the tool that the state of Arkansas used. Now, that company is called inteRAI, and Professor Brant Fries designed the algorithm that inteRAI used in Arkansas.

We spoke with him, and he tells us that statistical modeling, broadly, is very different from full blown AI. The conclusions AI draws are sometimes shrouded in mystery. Professor Fries told us. ... But statistical algorithms, like the ones he uses, are reviewed, he says, by other scientists.

BRANT FRIES: We say, here's the statistical methods we used, here's the data that we used, here's what the variables that we considered, here's why we considered these variables, what's the conceptual framework of why we think these variables are important to be considered. We put all that into a scientific paper, gets peer reviewed.

People look at it and other scientists may say why didn't you consider X and Y and then say we didn't work, or we can say good question. We reanalyze the data and Uh-Oh, I will. We didn't consider this, and we put it in the model, and now the model really changed. It really looks different.

We've got a better model than we had before.

CHAKRABARTI: So essentially, he's saying there that the kind of model that he uses, a statistical algorithm, is more transparent than full blown predictive AI. But Fries also told us that the scientific review process doesn't completely insulate the algorithms from human error.

FRIES: We put it out there and say, here it is. If you think it's good, you're welcome to use it. We can't tell a state that they have to use it. We can tell a state it's available. It does such and such. Here's the scientific article about it. If you want to evaluate it, we'll be glad to answer any questions.

It's up to you to use it and they can use it or they can misuse it. Now, they can use it for the wrong thing. We say, you have to use our whole instrument, don't change it, because then the science goes away of the instrument and its reliability and validity.

CHAKRABARTI: Okay, this is a really interesting point, because Professor Fries tells us, inteRAI's algorithm wasn't the problem.

He says the problem was that the software company that implemented it changed some of the algorithm's functions, to the detriment of the outcomes.

FRIES: It only came out after the trial when we started to go back and say when I calculated, I get this answer, and you're telling me that your software tells you that answer, and those are not the same, and I am correct, and this sort of clouded what happened in the case.

CHAKRABARTI: Professor Fries continues to stand by inteRAI's work. InteRAI's algorithms or instruments have been used in many states as well as governments in New Zealand, Hong Kong, Finland, and more. And he says the algorithms help improve how welfare systems function overall, which then leads to the benefit of the greatest number of people.

FRIES: It's very complicated when you bring a poor person and with a wheelchair who is horribly disabled and so forth, and someone saying you're not giving them enough services. And it's really hard to not be very sympathetic and judges and juries are very sympathetic to the individual say, yeah, they should get more services.

Yeah, they should. Except everyone should get more services. And why is this person eligible for more services and not all those other people who are not getting service? In fact, going to get less services if you give this person services.

CHAKRABARTI: So that's Brant Fries, Professor Emeritus of Health Management and Policy at the University of Michigan and President of the research group inteRAI.

Kevin De Liban, what do you think? What's your response to what he's saying there?

DE LIBAN: Yeah, I guess there's a few things. The first thing I think for people who aren't in the weeds of this stuff to understand is that AI isn't a term that has a set meaning, right? Oftentimes it's a hype marketing term that can encompass a lot of different technologies.

And from the perspective of somebody whose life is being judged by it, it doesn't necessarily matter whether it's the latest generation of AI that just came out yesterday or some advanced statistical modeling system that came out a dozen years ago or two dozen years ago. The purpose is the same. Every time these systems are introduced to decide the lives of low income people, they result in benefits cuts.

Every time. We don't have a single example where one of these systems was introduced and somehow people magically got more care or got better opportunities in life. Never. Now, Mr. Fries, or Dr. Fries, pardon me, also said something else interesting. There was an error in the software, but that wasn't the only problem.

If the algorithm had worked perfectly as designed, the best case scenario was still this five and a half hours a day of care, more or less, for most people. So the best case scenario for the way the algorithm works was a huge cut that left people still lying in their own waste or getting bed sores or missing out on physical therapy appointments or doctor's appointments or going out into the world.

Now the fact is the way Arkansas had implemented that algorithm was flawed and made it even worse. But the best case scenario was still horrific.

CHAKRABARTI: Was there any way that then this outcome could have been avoided, right? If how it had been implemented in Arkansas, if the goals were different, if the AI had told, was told to seek a different optimization, could it have actually helped people?

Or was the AI fundamentally flawed, as you're saying?

DE LIBAN: No, it's fundamentally flawed because its purpose is to make benefit cuts. Now, one thing Dr. Fries highlights that I agree with is that these systems generally are underfunded, right? And lawmakers putting more resources and funding into social services generally and health care, Medicaid, SNAP, public benefits, disability, the whole thing, unemployment, would help, right?

It would give people a little bit more stability in life and get above these very meager, barely subsistence level benefits that currently exist. But even it is not strictly a policy problem because every time you see these tools implemented, AI based tools, statistical modeling, whatever you want to call them, it leads to cuts.

It leads to lost services. It leads to false accusations of fraud. It leads to horrific things for low-income people.

CHAKRABARTI: Let me just bring another person in here. Sorry, if you heard a little pause in my voice there, it's because I'm just still trying to think, there seems to be a ... I'll bring in the other guest here for a second, Kevin, because there's almost, the question is almost more philosophical rather than technological, right?

And so to that point, let me bring in John Maynard. He's the principal industry consultant at SAS. It's one of the world's largest data analytics and AI companies. And he's also former state Medicaid program integrity director for the state of Ohio. And he served in that position from 2015 to 2018.

John Maynard, welcome to On Point.

JOHN MAYNARD: Hello, Meghna.

CHAKRABARTI: So first of all, just, Answer Kevin De Liban's assertion there, that there isn't yet an implementation of AI when it comes to welfare benefits, that isn't designed specifically to cut costs or cut benefits. That's just how it's been deployed out there at states and in federal governments around the world.

MAYNARD: I couldn't answer that in terms of everything that's out there, because I'm not sure what, everything that is out there. What I would say is when we say keep the human in human services, and we look at this as an opportunity to use analytics and AI to help automate processes, to basically augment the human to bring information to their fingertips so that they can make a decision, when you allow an algorithm to make the decision and half of the person.

I do think that they're correct that they're not perfect. An AI model is not perfect, as a person is not perfect. And so we think that the humans should make that decision.

CHAKRABARTI: Okay. So tell me at what point in the development of a instrument or system that a company like SAS would want to sell to, let's say, a state welfare department. At what point are the goals of that program built into the instrument? Is it, do you go to the state first and say, what would you like to accomplish? Or do you create the instrument first and then say, here's what it can do.

MAYNARD: It can be both.

Sometimes we will create something on behalf of a customer that they would like to see get done. And then there are some things that we sell out of the box. So we do have fraud solutions. We have different solutions to do AI medical record review. But everything that we do is really to help augment the human.

And if a customer said, we want you to build a model and do intelligent decisioning and run it across this data set. And make decisions on behalf of a person. We won't do it. Our founder Dr. Goodnight doesn't allow us to do that. I'm glad that he doesn't allow us to do that. Personally, I think that's a good decision.

Because when a model can make a mistake, because it's not perfect, it can make that mistake in volume. And, Kevin, I think is making a really good point here. So I actually used to be an eligibility worker for cash assistance and food assistance and Medicaid and child care and things like that, workforce training. And what Kevin is saying is right, is that when you make these types of decisions, as a blanket, there's a policy, there's this chance that you could hurt the people that you're intending to potentially try to help there.

And I think for the welfare population, they don't recover from those kinds of mistakes very easily or very well, because they're on the edge. And that's why we say keep the human in this. And I think you want an eligibility worker to make an eligibility decision. You want a nurse to make a health care decision.

CHAKRABARTI: Kevin, go ahead. I'd love to hear your response.

DE LIBAN: No, I'm excited to hear that a vendor the size of SAS wouldn't use AI for decision making purposes, and I think that highlights the danger of this. I'm focused on AI based decision making where it's making a decision about a vulnerable person's lives and, in those contexts, the danger is just too high, and you also have to understand the practical consequences, or practical circumstances into which this stuff is implemented.

A lot of states have been cut, right? There aren't enough state eligibility workers like Mr. Maynard was, they are overworked. They don't necessarily have the resources. They've lost a lot of expertise over the years as positions open and people leave, or they're eliminated, cuts to services.

And so what you have is really inadequate oversight of these systems in the government context. So in the state of Arkansas, for the algorithm we've discussed, for example, nobody on the state staff, nobody, not a single person could explain how the algorithm worked, either mechanically, in terms of what you put into it leads to how you get what you get out of it.

Or statistically, why the factors that matter are the ones that matter and why others don't. Not a single person on the state could do it. They didn't test it prior to deployment, which is how you get the errors that intensified the injustices that were already baked into the system. And they didn't care to fix it when the issues were raised to them.

And that's not unique to Arkansas. That's true of very many states. So I think there's a lot there. There's also something that Mr. Maynard referred to, that I'd like to explore a little bit, which is the notion that somehow these can go wrong at scale. And actually, some states have weaponized it to make benefits intentionally harder to get.

CHAKRABARTI: Yeah, Kevin, hang on here for just a second. I'm Meghna Chakrabarti. This is On Point. So tell me more, that you're saying there's deliberate weaponization of the scaling ability of AI.

DE LIBAN: Yeah. So one recent example of this is with what's called the Medicaid unwinding during the pandemic or the height of the pandemic. There were protections put in place so that states could not terminate people from Medicaid who became eligible.

And the whole idea was there, Hey, we have a pandemic going on. People need access to health care. We don't need to be cutting people off when their vital health care needs. In 2023, 2024, those protections ended, and state started this process of reviewing eligibility of everybody, at that point, over 90 million people.

You had 20, more than 20 million people lose Medicaid coverage, due to AI based decision making in many states, most of which were for paperwork reasons. They weren't because folks were actually ineligible, they were because AI made incorrect inferences from information they had, sent out more paperwork to people than needed to be happening, confused folks, and a lot of people lost coverage.

CHAKRABARTI:  John Maynard, go ahead.

MAYNARD: And then that's a policy decision about how you're going to implement that, so I think you can, you can always run some model that's going to say these people are likely to stay, these people are likely to leave, but at the end of the day, you still need an eligibility worker to look at that on a case by case basis and make decisions about it.

It sounds like in some cases it was not the AI or the technology as much as it was a communication problem with the actual recipients of the benefit. And I've seen that sometimes happen because especially with COVID, they can be transient. They can be difficult to get a hold of. So you're sending notifications because that's what the law requires, to an address where that person no longer lives.

So they don't know that you're trying to contact them. They don't know that you're trying to send them paperwork to fill out, in order to maintain their benefits. That's an age-old problem before AI, just of how things work in the system and how people come and go.

CHAKRABARTI: John, let me ask you this. If, and I definitely take your point that human oversight is critical and shouldn't be eliminated after a new AI tool is implemented in a welfare organization. But if you need continued human oversight, if there's the opportunity for errors, let's put it that way, to be inserted into the tool at implementation, et cetera.

What exactly is AI helping with? What is it making more efficient or better?

MAYNARD: So for me, I wouldn't, and we say, don't use it to make those kinds of eligibility decisions, but there's a lot of things that it can do to help make it easier for caseworkers and people who make those decisions to do better.

So one of the things I always remember, the tough, one of my toughest days as a caseworker is I had someone who, a woman who had cancer. And she wasn't eligible for Medicaid, because she needed a disability determination to come from Social Security. As it turned out, the day after she passed away, I got a letter saying she was eligible.

And her cancer would have been treatable. So the policy's been changed and people like her can now get Medicaid faster. But things like we're doing today is, we're working with an AI medical record review on disability. We put it in a few years ago, and it's looking at 16 million records a night.

All it's doing is it's organizing them, it's analyzing those, it's indexing them, handwriting, and it's then putting that together in a quick, easy way for decision makers. And so that's produced 400 hours, FTE hours of productivity.

Part III

CHAKRABARTI: Today we're talking again about welfare benefits or social services and the use of AI. Kevin De Liban joins us. He's founder of TechTonic Justice, and John Maynard is with us as well. He's Principal industry consultant at SAS and former state Medicaid program integrity director for the state of Ohio. And by the way, I just wanted to say quickly, when I said SAS is one of the biggest out there, according to their 2023 corporate report, more than $3 billion in global annual sales and 90% of Fortune 100 companies or their affiliates are SAS customers.

So this is a major company here. And John, you were telling us a little bit more about that example of the woman who had passed away before she got her benefits. And why was that relevant?

MAYNARD: I think it's relevant because when you have a decisioning process that's very manual and human based, like a disability determination, generally there's backlogs, but when you have someone like her, who's waiting for that decision, and it's her life was on the line, that needs to be done faster.

That needs to be done more efficiently. Kevin's right. One of the things that we're seeing now is the boomers are leaving government. They're leaving these programs. It's harder to recruit younger people into him. So those staffing shortages aren't going to go away. That's why I really feel like social benefit agencies globally, they need AI to keep up with the demand. And I so I think that's it, and also to help fight fraud in the program. Also, fraud against beneficiary. So we've seen criminal acts against applicants and beneficiaries as well.

CHAKRABARTI: How much fraud is there on the path of beneficiaries in these programs?

This is something that gets said all the time. But, have AI systems like actually found masses of fraud? John?

MAYNARD: You can find fraud. So generally, the general rule is 10% to 15% is going to be fraud, waste, and abuse. And fraud is probably 2% to 3% of that. There's actually 3 to 4 times the number of errors than there is actual fraud.

CHAKRABARTI: But you mentioned, but you said fraud. Here's why I'm asking this, okay? Because I was looking at your bio, John, and I have become, lately, I've become a believer in the importance of org charts and here's why.

MAYNARD: (LAUGHS)

CHAKRABARTI: No, here's why, seriously, because your experience is very much valued in this conversation.

And I was noticing that at least at one point in time, your position was under SAS's global fraud practice. Is that correct?

MAYNARD: Yes, it was. We've just been rebranded. I still work on the fraud stuff all the time. It's something I really like to do. And don't tell anybody, but I have a real propensity for figuring out how to steal money from a system.

That's my white hat for that.

CHAKRABARTI: Don't tell anybody, except for the millions of people who just heard this. No, the reason why I'm asking is this. Is that, I'm just going to use you as an example, but if a company is creating an AI tool that it's trying to sell to a state social services organization, but that company falls under the org chart of global fraud practice, again, to use you as an example, that is success in reducing fraud is the metric that group is going to be measured by within their corporation.

So they're going to be building tools that are really good at that. They're going to cut costs by finding, quote unquote, finding fraud. That is not a tool that is designed to help make a social service department more efficient and provide better services. It's just not baked into the goal of the team that's designing the tool, John.

MAYNARD: Yeah, and so what I would say to that is a fraud solution is a fraud solution. That's really what it's designed to do. But we also do a lot of other things on top of that. And there's a lot of SMEs, even though we're, cause we're in the fraud, risk and compliance piece, so we're just more than fraud.

And so I worked on things like total cost of care, episodes of care, value-based payments. I'm very familiar with home health. I'm familiar with a lot of those things that we're talking about. And because I come through the system and from the system, I don't code. I don't, I'm not a data scientist.

And so what I do is I talk to people inside our company about, here's the struggles that I see in these places. Here's, we need a solution to help them do this. We need a solution to help them do that. Things like a medical record review. We also created a model that is helping with SNAP. And it's basically just looking for high risk error cases.

And it's something that the state can have to help get their error rates down. And what I would say is, you're losing hundreds of millions of dollars in these overpayments. But what you heard Kevin say is, somebody else needs more health care benefits. Guess where the funding for those health care benefits they need?

Can be coming from, and it's these errors and the waste that we have in other programs or even the same program. There's a lot of things from a policy standpoint, you might want to do that you don't have the money for, but if you're wasting money in other areas, that's not really providing any value or any good to anybody.

Then you need to find that money and reallocate it to where it needs to go.

CHAKRABARTI: Kevin, are you crawling the walls yet?

DE LIBAN: I might need to start. But no, I guess the first point is, look, instead of turning necessarily to automated solutions, particularly the more close they get to a decision that affects a vulnerable member of the public, we need to just think about investing in government capacity, right?

And that's a universal thing, both at the state levels and at the federal level. Social security, for example, has been running a massive staffing deficit from where they used to be and where they need to be. For, I think it's two decades now. So the first thing is not to go to like fast seal or kind of nice shiny, dangling solutions like AI, but actually invest in government capacity, human capacity to do these decisions.

The other thing is, I want to really focus on the dangers of this focus on fraud. So statistically, in the snap program, which many people know as food stamps, of every hundred dollars of SNAP benefits paid out to recipients, only ten cents are ever improperly paid due to recipients' intentional fraud.

So there isn't really fraud on the part of beneficiaries. But this whole notion of fraud, waste, and abuse ends up justifying, right now, the destruction of the federal government, what we see happening at DOGE. Not an objective thing, but using fraud as a cudgel to identify anything that the president decides he's opposed to, and then destroy it.

And we hear it in state benefit programs as ways that end up going after beneficiaries, right? Going after people getting the programs and these fraud detection systems don't work when they come to individuals. The example in Michigan that you stated earlier, Meghna, is actually an understatement.

Over 40,000 people were accused of fraud. When an Auditor General, the state's Auditor General, went in and did a review, 93% of those were false positives. 93%! Or, during the pandemic era of expanded unemployment benefits, there were routinely flagging people for possible identity fraud at huge scales, right?

And at the end of the day, it was established that most of those were false flags, right? And so what you're doing with these fraud systems is, if you're not destroying government outright, you're making benefits harder to get, putting up barriers that frustrate access, and you're doing it in a way that ensnares a whole bunch of people who are eligible.

So that's why this isn't, it's just, it's a dangerous concept because it allows, it justifies doing things that ultimately are harmful and frustrate people's ability to get benefits during really desperate times of life, I should note, right?

CHAKRABARTI: But what you're saying here then, let me put it this way, because originally, I had asked John, hey if you're just building tools underneath the aegis of the group in the company that's supposed to pump out fraud detection tools, you're going to build fraud detection tools.

But what you're saying, Kevin, is actually, I think, more important. And that is the people who are deciding, in government, obviously right now at the federal level, but also at the state level, you're saying that there are policy decisions that are being made specifically because they believe there's a lot of fraud going on.

And so therefore, of course, they're going to use these particular AI tools whose, you know, whose programmatic strength is fraud detection, Kevin, right? So the problem is the people.

DE LIBAN: No. I want to push back there, because I don't think in many cases it's a sincere belief that there is fraud, right?

I think fraud is a convenient cudgel to allow people who are ideologically opposed in many cases to any sort of safety net benefits or systems to go in and attack those systems. I want to be really clear about this. A lot of this is not good faith.

CHAKRABARTI: Agreed, but then the problem isn't the AI. The problem is the people.

DE LIBAN: It's both. It's both, right? The problem is the people in the sense they're out to destroy things or make benefits harder to get, but AI is the perfect weapon to do that, right? For some of the reasons that John pointed out, it's scalable. You put that in and suddenly you're affecting the benefits of everyone in a way that you couldn't if you had a caseworker have to manually review or make decisions about whether or not somebody intended to deceive the government.

And that's another key point here. Is there is a definition to fraud, right? It is intending to deceive the government in order to receive benefits, or in the case of government contractors, contracts that you wouldn't otherwise. Most of what's being attacked by this isn't intentional fraud. People are getting caught up in it, but it's not demonstrating any intent.

And the pernicious danger, and to your question of is this a technology problem, here's where it is. Once somebody is flagged by AI, or any sort of automated process, as being fraudulent, that ends up serving as evidence of the intent. That ends up being, in many cases, the judge, jury, and executioner. Is the automated decision, because it has this veneer of being objective.

And it isn't. And human reviewers are oftentimes not able to meaningfully and adeptly challenge it or review that. You get tagged as fraudulent by the algorithm; you might as well lose your benefits. It's sign sealed, delivered.

CHAKRABARTI: I think that point is very well taken about the complexity of the technology makes it hard to appeal or to push back or to get a human reviewer. Unless the system is built, the office, let's say the organization is built to say yes, a human reviewer can automatically just reject the AI's conclusion.

I totally take that point, we've only got about four-ish, less than that, minutes left, gentlemen. Look, AI is here to stay. There is no doubt about it. With every second of every day, more and more, it's worming its way more and more to every aspect of our lives. And I think to potentially huge positive benefit, but we have to keep being, we have to keep scrutinizing it, as well.

So with that in mind, John, is there a better way? What could companies like SAS do to create tools that actually don't lead to the high number of examples that I started off with, of AI actually hurting people rather than helping them.

MAYNARD: I think one is the decision in question that we've talked about.

But SAS has a data ethics practice. And so we're focusing on human centric design. We're focused on inclusivity, accountability, transparency, understanding what a model is doing, why it's doing it. Robustness, which means once it's placed in operation, you're constantly monitoring it to make sure it's doing what you intended, and it's not having unintended consequences.

And then also working on privacy and security. I think what Kevin said there is the data itself is not an indication of fraud. And as a human being, I've fallen in that trap through the years where I got something and I said, Oh, this must be fraud, only to investigate it and find out. No, it wasn't. So you always have to be careful.

The data itself and what the model is telling you isn't mean it's fraud. It means it's an unusual anomaly. You need to look at it. You need to prioritize that and give it a review.

CHAKRABARTI: But then don't sell algorithms that are designed as decision making software, right? Because ultimately, it's not just analyzing the data.

It's pumping out a decision on benefits.

MAYNARD: Not the ones that we do, and I wouldn't recommend, and I wouldn't recommend that because, and I would say, and I say that, because as a caseworker, as an auditor, as an investigator, every situation is a one-on-one decision. There's something about every one of those patients, there's something about every one of those families, which would make a decision go one way or the other, and an AI model can't account for all of that.

And that's why you want a human to make that decision. And I know that there's human bias, but there can also be bias inside those models, because human beings make them, and human beings use them. So let the human make the decision. Let AI augment the processes around that to help them make better decisions faster.

CHAKRABARTI: Okay, so Kevin, you get the last word today on how we make it better, because I hear what John's saying, but we started out with you describing to us what happened in Arkansas, and humans making the decision, ultimately on how many hours of assistance these folks were getting. I don't recall you saying that was how the system was being used, so how can we make it better?

DE LIBAN: Yeah, first you have to change the broken incentive structures that are here, and the broken accountability structures. There's no political accountability usually for state government officials or elected officials when things go wrong for poor people. There's very little market accountability in most of these situations, because you have only a select number of very big vendors.

And so you see vendors who have demonstrated histories of failure still get big contracts, because states don't feel they can go anywhere else, and you have limited legal exposure, because of some things we didn't get a chance to get into today, but what you need is you can't rely on the goodwill of people who design these tools.

You have to legislate around them. You have to ban certain decision-making uses. You have to require really heavy duty vetting and deployment and monitoring. You have to have a mechanism for meaningful oversight by the people who are affected by the decisions that are to be made. And you have to make sure that there are disincentives, legal disincentives to doing anything that's going to be massively harmful to really vulnerable people. And the only way to do that is through significant legislative and regulatory interventions.