Cheat Sheet Or Open Book: Putting Tests To The Test
Afshin Gharib and William Phillips are associate professors of psychology at the Dominican University of California. They are also carpool buddies.
On drives to and from work, they used to argue over what kind of exams work best to test student knowledge, encourage retention and keep stress low.
Gharib decided to change his traditional, closed-book testing style about 10 years ago. He got tired of curving exam scores and decided to try out the open-book style.
"At first I thought that everybody would get an A," Gharib tells NPR's Neal Conan, "but oddly enough, the scores were normally distributed and not that much higher than on a closed-book, closed-note test."
Though the test results didn't change, the students seemed much happier, so Gharib chose to stick with the open-book style.
"But whenever I drive with [Phillips]," Gharib says, "we have an argument and Bill disagrees with me."
Phillips typically allows students to create a cheat sheet as an aid for exams, based on the idea that the process of creating the aid actually increases learning.
To settle the debate once and for all, they did what researchers do: They ran an experiment.
The professors used a couple of their classes to put the question to the test. During the course of one semester of introductory psychology and statistics classes, they gave one open-book exam, one cheat sheet exam and one closed-book test.
In his statistic class, Phillips only used the cheat sheet and open-book methods, so that students didn't have to memorize all of the formulas.
As for Phillips' belief that the process of organizing a cheat sheet helps students with learning and preparation, the experiment proved that to be incorrect. Though Phillips says students spent more time studying and preparing for cheat-sheet exams, there was no correlation between the detail of the aid and the scores.
The professors also found that the style of testing does not influence retention rates.
"We gave a pop quiz about a couple of weeks after each of the tests," Gharib says, "and we found the scores were pretty much exactly the same in both classes ... regardless of the type of test the students are prepared for."
Gharib and Phillips would be interested to see if these trends apply across the board. "Introduction to psychology and statistics are as different pair of classes as we could find," Gharib says. "So we think that it's pretty broadly applicable."
The Student Makes The Difference
One discovery that seems pretty concrete: The student makes a big difference. "A good student did well regardless of what type of test they were given," Gharib says. "A poor student did poorly regardless of what type of test they were given.
"And that, I think, was a really interesting finding — that, in a way, the type of exam really makes very little difference."
Phillips says he will probably start using more of an open-book format to keep student anxiety low.
So, as for the drive to work, this particular argument is settled for now.
"It was a friendly bet," says Gharib, "but clearly, I won."
Phillips will be doing the driving, at least for a while.
NEAL CONAN, HOST:
Ask students what kinds of exams they prefer and no surprise, most say they like open-book tests. Ask their professors and you're likely to get an argument. Some hold to the traditional closed-book exam as the most rigorous. Some say the process of creating a cheat sheet teaches students both the material and how to organize it, while another camp believes open-book exams are just as effective and much less stressful. So as the fall semester wraps up, we want to hear from college professors: What have you learned about exams over the years? Give us a call, 800-989-8255. Email: firstname.lastname@example.org. You can join the conversation on our website, that's at npr.org, click on TALK OF THE NATION.
Afshin Gharib and William Phillips had this argument in the carpool that takes them to Dominican University, where they're both associate professors of psychology. So, of course, they decided to run an experiment. They join us now from member station KQED in San Francisco. Nice to have you on TALK OF THE NATION today.
AFSHIN GHARIB: Thank you very much.
WILLIAM PHILLIPS: Thank you. It's great to be here.
CONAN: And Afshin Gharib, what exam did you argue for?
GHARIB: Well, about 10 or so years ago, I got tired of always curving my exams. I always used to give the traditional closed-book, closed-note tests. So I tried once to give an open-book exam, and what I found was that, oddly enough, at first I thought that everybody would get an A, but oddly enough, the scores were normally distributed and not that much higher than on a closed-book, closed-note test. But students seemed to like it a lot more. So ever since then, I've always given open-book, open-note tests. But whenever I drive with Bill, we have an argument and Bill disagrees with me, so...
CONAN: So, Bill, what's your favorite?
PHILLIPS: Well, I often give a cheat sheet as an aid for exams, and part of that came from some literature I saw where learning is actually enhanced by going through the motions of organizing your notes. And just going through that process is believed to increase learning. And so I argued for a cheat sheet of some sort to increase learning.
CONAN: I have to ask, were anybody else in the carpool besides you guys?
GHARIB: No, it's a small carpool.
GHARIB: It's a very small carpool.
PHILLIPS: It's a mini.
CONAN: Because I think they may have checked you out if there've been a bunch of people in the carpool. So how did you decide to resolve this argument?
GHARIB: Well, what we decided to do is take a couple of our classes and put the question to the test. So we had three different exams over the course of the semester, and we gave one open-book exam, one cheat sheet exam, one closed-book exam in both introduction to psychology class and in a statistics class.
CONAN: And how did it turn out?
PHILLIPS: Well, I was teaching the statistics sections, and one of the things I did - I didn't actually used a closed-book exam, just because making students memorize formulas I didn't think was very proper. And so I only used two of the conditions, the cheat sheet condition and the open-book condition. And we basically found that though performance changes, like one of the interesting things was retention doesn't change that much.
CONAN: Between any of the three methods?
GHARIB: That's right. We gave a pop quiz about a couple of weeks after each of the tests, and we found the scores were pretty much exactly the same in both classes, exactly the same, regardless of the type of test the students are prepared for.
CONAN: Did the students - I know you asked them some questions too. Did the students prepare the same when they knew it was an open-book test or a cheat sheet test or a closed-book test?
PHILLIPS: Interestingly, students actually spent a little bit more time studying in the cheat sheet condition, and we believe that that difference was actually preparing and organizing their notes.
CONAN: Interesting. And that would suggest that those who organized and prepared the best notes, did best.
GHARIB: And that's exactly what we didn't find.
GHARIB: We looked at the correlation between the depth and detail of the cheat sheet and student scores, and there's essentially no relationship.
CONAN: So the person who can organize their notes brilliantly doesn't do any better on the test.
GHARIB: Or remember more material two weeks later.
CONAN: That's sad. So your conclusion is - did you bet anything on this, by the way?
GHARIB: It was a friendly bet, but clearly, I won.
CONAN: So that means...
CONAN: So that means you have to do the driving, right?
PHILLIPS: That's right.
CONAN: And so the conclusion of this - now, you're in psychology. Do you think this applies the across the board?
GHARIB: That's a very good question, and we'd be interested in seeing if it does. If I was making a prediction, I'd say that it would just because within the discipline of psychology, introduction to psychology and statistics are as different pair of classes as we could find. So we think that it's pretty broadly applicable.
CONAN: And have you - oh, by the way, have you changed the tests you give now, given the results of these experiments?
PHILLIPS: Well, yeah. I've instilled using a cheat sheet and open-note, but that's because we've kind of been collecting more data as we go along. So we've kind of continued this study.
CONAN: Ah, so you don't have to give in because you're continuing the study, I get it.
CONAN: All right. So you don't actually change. Let's see if we get some callers in on this conversation. We want to hear from college professors today: What have you learned about exams over the years? And which method do you prefer? Have you changed? Give us a call: 800-9898-8255. Email us: email@example.com. And then we'll go to Derrick(ph). Derrick is on the line with us from Truckee, California.
CONAN: Hi. You're on the air, Derrick. Go ahead.
DERRICK: Yeah. I just wanted to add. I moved over a few years ago from traditional exams to open-note only exams. And I don't have any data to really - to support it. It was just something I wanted to try. And I let them use notes, handouts, old tests, anything but the textbook. And my rationale there is I want them to sort of write their own textbook as they go through the semester. And it definitely encouraged them to stay more organized as they go. And I get good student feedback, makes the discussions more interesting. Grades really haven't changed much, but it's something that seems to work really well, and I like doing it that way.
CONAN: William Phillips, he's on your side.
PHILLIPS: Yeah, yeah, pretty good.
CONAN: Will you change consider the - considering the results of this experiment?
PHILLIPS: Well, I think I probably will. I'm going to - I think I will start using more of an open-note format, open-note, open-book because the data seem to suggest that it lowers anxiety, and I think that's better for the student.
CONAN: Derrick, might you reconsider?
DERRICK: I don't think so.
CONAN: Stubborn. OK.
CONAN: We like that in a professor. Thanks very much. What do you teach, by the way?
DERRICK: I teach biology.
CONAN: Biology. Thanks very much for the call. Good luck.
DERRICK: Thank you.
CONAN: So long. Let's see, here's an email from Chris in Macomb, Illinois: Just to comment, the test I used to give were mostly open-book. That said, they weren't tests of knowledge. They were tests of application. If you didn't know the material thoroughly when you entered the room, you were never going to complete the test no matter how many references you brought with you. And Afshin Gharib, that's a good point. Just because it's an open-book test doesn't mean you don't have to study.
GHARIB: That's true. However, in my classes, particularly in an introductory psychology class, that's really not the case because a lot of the questions have to do with facts that you can, in fact, find in the textbook if you look through it. I think what happens is that students don't prepare as well for an open-book exam. And as a result, they don't do as well as they think they would on those sorts of tests.
CONAN: We go next to Chris(ph) - and excuse me. Chris is on the line from Columbus.
CHRIS: Hi. Thank you for taking my call.
CONAN: Go ahead.
CHRIS: I teach at a community college here. And I teach - it's a basic comparative linguistics class between English and American Sign Language. And the students have a lot of terminology to learn. And for a lot of students, it's their first time in the classroom and they're very intimidated. And so I always begin my class with a first quiz that they get to work in groups, so they have an idea of the kind of questions that I'm going to ask. It's not like memorize this term but it's more of an application. Here's a linguistic sample and can you apply a term that we've talked about.
And then throughout the rest of the semester, I tend to let students, maybe work with a partner or in pairs. And I found that it's been incredibly successful. Students - it gives the opportunity to students to think out loud. They can talk to their partner and ask questions, and their partner can say, well, I kind of think of it this way. And the other student might go, well, gosh, I never thought about asking about it from that approach.
And I don't give all quizzes that way. Sometimes there is - if I know that this one with a lot of terminology, I'll let them work together like that. And what I've also found is that it does alleviate their frustration and their concern. But it also helps prepare them because they want to look good working with a buddy in class.
CONAN: Yeah. They want to look good to their buddy. And by the - then they will look good to you too.
CONAN: Have you...
CHRIS: And to themselves. Yeah.
CONAN: Afshin Gharib, I'm hearing another experiment in the making.
GHARIB: Absolutely. We're very interested in trying different types of examinations, so we've been talking about take-home exams as another type of exam to look at. The group exam seems like a great idea. We'll put on the list of types of exams to look at next.
CHRIS: Oh, that's great. I'll look forward to that research.
CONAN: And, Chris, are you in the middle of exams now?
CHRIS: We just finished up, and I'm going home to grade 40 projects where they had to take all that terminology and hopefully apply it. And we'll see how well they did.
CONAN: Good luck.
CHRIS: Thank you very much.
CONAN: We're talking with Afshin Gharib and William Phillips, both associate professors of psychology at Dominican University of California, co-authors of "Cheat Sheet or Open Book: A Comparison of the Ethics of Exams." You're listening to TALK OF THE NATION from NPR News.
Here's an email from Julie in Oakland: Open book. The whole world has become open book with the Internet. I am now testing the students' ability to discern and evaluate the information. I teach textile science at a fashion college and there's a lot of information to cover.
Quick story: I had a final review on September 10, 2001 with the exam to be on 9/17/01 and had always been giving closed book exams. After 9/11 happened, I decided that my students should get a break and gave an open-book exam. I was surprised to find I still had a full range of grades, from A to F, and that students did not appreciably change their grades with the open book. And, William Phillips, that's what you found too.
PHILLIPS: Yes, I think that's correct. There is a slight increase in grades between the three exams when you look at it across the same student, but it's very small. It might be a half-a-letter grade.
CONAN: And if you were scaling it, it shouldn't make much difference at all.
PHILLIPS: No, no, no. And, in fact, as Dr. Gharib suggested, you find that you really don't have to scale. You don't need to scale.
CONAN: Let's get another caller in. This is Jeff, and Jeff's on the line from Gainesville.
JEFF: Hey. How are you? I'm a TA at the University of Florida, Gainesville in nuclear engineering, and we have both methods in our department. There's a professor who does open book. Take as long as you want because as an engineer you're going to have your resources available, and you're not going to be necessarily given a one-hour time limit to solve your problems then. And people will usually take three to five hours on his exams, and he's OK with that.
But on the other hand, most of the professors will use a cheat sheet and, you know, students tend to prefer the cheat sheet because the professors who opt for that option can give absolutely brutal exams if it's open book.
CONAN: I see. So if...
JEFF: A lot of it is professor-dependent.
CONAN: But as a TA, you get to see different professors trying these different methods.
JEFF: Right. Absolutely. And it's interesting to watch because I think the purpose is the same, but a lot of the - and my question to the statisticians is how much correlation and experimenting should be found on learning versus test scores, you know, because you can dump a lot of stuff - I mean, you mentioned that two weeks later (unintelligible) Do you follow up on any of that?
GHARIB: Yeah, that's a really good question. I mean, I think one of the most interesting things we found was that for any individual student, their scores on the different types of tests were highly correlated. So a good student did well regardless of what type of test they were given. A poor student did poorly regardless of what type of test they were given. And that, I think, was a really interesting finding, that in a way, the type of exam really makes very little difference.
CONAN: But the anxiety level, that makes a difference. Jeff, in your experience, are students - get very anxious about the big test?
JEFF: Again, you've got the same correlation across students that do that. Some of them psyche themselves up. Some of them psyched themselves out. And it doesn't matter what type of test it is. I absolutely agree that the type - the good students tend to do well no matter what, and the bad students are - have a struggle no matter what.
And again, it's the psyched-up for a test and don't handle stress well versus the one - but the ones who don't handle stress well who have - who always have a time pressure, like the open-book, open time-limit, they really appreciate that and - because they - the time pressure is gone. So a lot of it, I think, is just the time pressure correlation as well.
CONAN: Interesting, engineering. I would think journalism, you should have a deadline. That should be appropriate for that exam.
JEFF: Sure. Engineering, you're going to be given a couple of weeks for a project, not a couple of hours.
CONAN: All right. Thanks very much, Jeff. Good luck.
JEFF: Thank you.
CONAN: And let's see if we can get one more caller in. This is Nancy, Nancy on the line with us from Naples in Florida.
NANCY: Yes, I'm here.
CONAN: Go ahead.
NANCY: Yeah, I was just saying I - arts is a little different. Our students are not taking our courses for credit. They're learning English as a second language. But a lot of our test are electronic. And we allow them to take it multiple times because many of them will take it two or three or four times, trying to get all the answers right instead of just aiming right for the grade. And because it's electronic, they can do it and they get the feedback right away with what their score was.
CONAN: And, Bill Phillips, obviously, electronic tests or online tests are - that's another category too.
PHILLIPS: Yes. In fact, I think it's very promising that our results show that there isn't a huge difference between the three types of exams because when you give an online exam, that's going to be automatically pretty much an open-note, open-book situation. And - however, our results suggest that, again, it doesn't meant that it's going to be an automatic A for the student and that the retention is going to be same as if it is a closed book.
CONAN: Nancy, thanks.
NANCY: Thank you.
CONAN: And here's an email from Mary Beth(ph) in San Antonio: Not using text in notes encourages cheating. My personal testing method is that students often memorize material and promptly forget it. Using testing by application forces them to apply what they've read and learning to - and learned to differing situations. Is there any way to test about cheating?
GHARIB: That's a good question. I mean, that's a problem that, of course, we come across all the time, and it's going to be increasingly a problem. I don't know that our study can really tell us anything about the likelihood of cheating and which style of test makes it most likely. My guess would be with the open-book, open-note exam, cheating would be lower because you have all the material with you. There's no need to check your neighbors.
CONAN: Or scribble something extra on your sleeve cause you've got the paper in front.
CONAN: Anyway, Afshin Gharib and William Phillips, associate professors of psychology at Dominican University in California. They co-authored the study "Cheat Sheet or Open Book." They joined us today from member station KQED in San Francisco. Thanks very much for being with us.
PHILLIPS: Thanks for having us.
GHARIB: Thank you. We'd also like to thank Noel Matthew(ph) who was an undergraduate assistant that helped us with this research.
CONAN: We'll talk to you again on Monday. It's the TALK OF THE NATION from NPR News. Transcript provided by NPR, Copyright NPR.