Harvard researchers explore how to use generative AI for student feedback

Generative AI programs stirred up a lot of controversy this year — especially around their use in schools. Worries and think pieces proliferated on what to do about students using ChatGPT to pen essays, or whether K-12 schools should punish kids caught completing assignments with AI.

But could this emerging technology also serve as a teaching tool to free up instructors' time on routine tasks? Could AI perhaps deepen students' engagement with the coursework? A research team at Harvard sought to find out.

In a Harvard Graduate School of Education study, the team focused on testing changes to periodic teacher feedback in a graduate-level course designed around what's known as a "maker space," or a project-based, collaborative classroom.

The students were required to learn to use several tools, including a laser cutter, a sewing machine and a 3D printer. Then, armed with new knowledge, they were asked to make an educational toy.

Typically, class instructors would write up individualized feedback for the students — from noting progress they'd made on their project, to offering words of encouragement.

But this time, the graduate researchers had instructors use a generative AI platform pre-loaded with student information from previous self-reflections and surveys to compose the messages. Teachers would review the feedback for errors and add details if they wished before giving the notes to each student.

"[The maker space is] a very complex space where students need a lot of feedback," said Bertrand Schneider, principal investigator of the project. "Usually that kind of message would be difficult to write by yourself and very time-consuming."

Using AI, he said, sped up the process significantly. It also helped instructors better tailor their messages by including additional copy aimed at helping students develop what's known as "a maker mindset," or a way of approaching their coursework with persistence, creativity, curiosity and an openness to collaboration.

To measure the AI tool's effectiveness, the team surveyed students on what they thought about the class overall and the feedback they got from teachers.

"People felt that these AI-augmented messages were spearheading the caring classroom culture," said Gahyun Callie Sung, one of the study's co-authors. "What we found was that, statistically, it had a significant impact on the sense of student belonging and levels of burnout."

However, for the subset of students who self-reported as struggling through the class, the AI feedback was perceived to be "unhelpfully short and insensitive."

Sung and Bertrand said there was some degree of randomness involved with how the AI-generated messages were created, making it hard to say why the output for struggling students could sometimes vary in quality. They added that while it is possible to get better feedback by tinkering with the prompt, spending time exploring that question could take just a long as it would for the instructor to compose a personalized feedback message.

The team hoped to improve on the AI program in future research, but said learning about the limitations of the technology was a helpful exercise.

"The takeaway [from this study] is that there is potential in using AI to help teachers," said Schneider. "It's like having an intern to do tasks for you ... so you can focus more on learners and their struggles."