A Harvard student asked her professors to grade ChatGPT's essays. It got mostly A's and B's

Maya Bodnick, a rising sophomore at Harvard, wanted to see if professors could tell if an essay was written by her, or the artificial intelligence program ChatGPT. The results of her experiment illustrate that big changes to student assessment may be necessary as easily accessible technology advances.

In fact, it's already led to some changes on the Ivy League school's campus.

"I think that homework is never going to be the same again," Bodnick told WBUR. "Students are going to use ChatGPT whether it's allowed or not. And AI detectors just don't work. Professors can't tell."

For the experiment, Bodnick recruited eight professors, and asked them to grade essays that she said were written either by ChatGPT or herself. In reality, all of the essays had been written by ChatGPT, which Bodnick says was to minimize response bias.

Bodnick assumed that the AI “might even be better than the average college student.” Her findings — which she first documented last week for the newsletter Slow Boring — suggest that it could at least come pretty close. Across all the evaluations in her study, ChatGPT's unedited essays received almost all A’s and B’s, giving it a GPA of 3.34. Bodnick said that score is "lower than the Harvard average, but still well above passing."

"ChatGPT is only nine months old, so I think these results were pretty shocking, because what's going to happen in a couple years, right?" she said.

Bodnick remembers first encountering ChatGPT last fall as something her sister used for jokes over the Thanksgiving break. Within a few months, she began hearing classmates mention using it to study or even to potentially write essays.

"I was really interested, as someone in a lot of classes where writing is the primary assessment method," Bodnick said.

Detecting AI-generated work is a mounting challenge for professors. And despite attempts to keep pace with the advancing technology, Bodnick thinks “students are going to [find] other things to get around detection."

Some professors are simply eliminating take-home essays completely. After participating in Bodnick's experiment, Harvard economics professor Jason Furman tweeted that he plans to drop the essay in his 700-student introductory economics class this coming year, "in part because ChatGPT has reduced the marginal net benefit that comes from the essay" due to the potential for cheating.

“Unfortunately, I think that’s going to be the solution for a lot of classes,” said Bodnick.