'Proceed with caution:' AI poses issues of discrimination, surveillance

Artificial intelligence is seeping into everyday use. AI tools can help organize your mountain of family photos that grew over the holidays by identifying faces and grouping pictures of the same person. On social media, AI-altered images are often indistinguishable from those made by humans. The use of chatbots has grown in academic settings for better or for worse.

There are many uses for AI as the technology becomes more accessible and normalized, but not everyone is excited about that premise. AI scholar and activist Joy Buolamwini is one of those critics. She’s the founder of the Algorithmic Justice League and author of the book "Unmasking AI: My Mission to Protect What Is Human in a World of Machines."

She started her journey as a scholar enamored by the promise of AI. But her views changed when she tried face-tracking software and it didn’t work well on her dark skin. However, the software registered her when she put on a white mask.

Buolamwini questioned whether her problem was unique or would happen to others with dark skin. And she found that the data was skewed.

“The data sets we often found were largely male and largely pale individuals,” Buolamwini says.

This poses problems especially when AI is used by law enforcement agencies to identify suspects, assess whether a defendant will commit another crime, and assign bond limits or flight risk status. Buolamwini uses Porcha Woodruff’s story as an example. Woodruff was 8 months pregnant when she was mistakenly arrested for carjacking after being misidentified by Detroit police’s facial recognition software.

“We're creating tools for mass surveillance,” Buolamwini says, “that in the hands of an authoritarian state can be used in very devastating ways.”

Buolamwini stresses that even if data set bias was addressed, accurate artificial intelligence could still pose problems and be abused.

“We're asking, ‘Do we even want that system to exist in the first place?’” she says. “It goes beyond a question of data set bias or the accuracy of the model, but greater questions about, what kind of society do we want to live in? And what kinds of technologies do we permit in that society?”

Law enforcement isn’t the only sector that uses facial recognition. Banks use the technology to verify credit card users and universities use it to ensure online test-takers are who they say they are. But in Buolamwini’s opinion, the benefits of efficiency and validation do not outweigh the risks.

The European Parliament seems to agree, as it totally banned live facial recognition in public spaces in June 2023. Buolamwini says the U.S. should evaluate its data privacy and protection laws before artificial intelligence becomes even more commonplace.

Buolamwini uses social media as a cautionary tale. When it first rolled out, regulations were not robust enough to prevent the rampant misinformation on platforms now.

While she points out the risks, she says that consumers need to be responsible with their use of AI. She recommends reading the fine print on chatbots and other tools before jumping right into using them. And she says it's the government’s responsibility to set the parameters for using new technology.

“I think the guardrails we can put in place are quite sensible if you're using AI in a high-risk setting like determining if someone gets a job or not, if somebody gets an organ transplant,” Buolamwini says. “There are possibilities for how those tools could be used in a beneficial way. So it's not saying every possible use is terrible, but it's saying we have to proceed with caution.”

Shirley Jahad produced and edited this interview for broadcast with Peter O'Dowd. Grace Griffin also adapted it for the web.