Handing out gold stars can boost participation in online courses
Grammar school teachers often hand out gold stars or similar badges to reward and encourage certain types of student behavior.
Now a Stanford-led research team has used the same approach to motivate college-level students taking massive open online courses, or MOOCs.
At the International World Wide Web Conference the researchers presented a paper showing how they prompted students in one MOOC to read more pages in discussion forums, and cast more votes rating the quality of the posts and comments they read.
In a separate but related aspect of the study, the researchers looked at what people actually do after they sign up for MOOCs, because only a fraction finish the course. Thus, thinking of MOOCs as analogous to offline classroom courses may be misleading.
Researchers found that many people seem to use these free, instructional opportunities as reference tools to be kept on the shelf until needed; they ignore the tests and grades. A separate subset of registrants uses the MOOC as a refresher course or a way to test knowledge; these users ignore the instructional videos but complete the homework assignments and take the tests.
“MOOCs get an unfair rap,” said Stanford Computer Science Professor Jure Leskovec, who worked with Stanford doctoral candidate Ashton Anderson, and Cornell University computer scientists Daniel Huttenlocher and Jon Kleinberg.
“All you hear is that people don’t finish them, when they are often used as needed, like textbooks,” he said. “You don’t evaluate the value of a textbook just by the number of people who read it from start to finish.”
The study’s common thread is engagement – increasing it by using a badge system and understanding the diverse behaviors exhibited by the students.
Targeted motivation
In the experimental portion of the study, the researchers sought to prod students toward higher levels of engagement by awarding them badges when they engaged in a course Q&A forum: by reading a certain number of posts or casting votes to rank the quality of posts by others.
“We wanted to be very specific in the types of behavior we encouraged,” Leskovec said. He likened reading and rating posts in a MOOC forum to playing a constructive role in a classroom discussion.
But the researchers did not want to encourage students to write posts or comments solely to get recognition. So they only awarded badges for posting if other students endorsed their contributions by casting positive votes.
This targeted recognition worked: activity for which badges were awarded increased significantly, while types of activity for which no badges were awarded remained unchanged. The researchers observed a five-fold increase in the proportion of frequent voters and heavy readers in their experimental MOOC. But the proportion of frequent posters and commenters held steady.
“We showed that we could be very precise in the behaviors we incentivized,” Anderson said.
In this portion of the paper, on targeted motivation, researchers studied three MOOCs offered through the online learning company Coursera.
All three MOOCs taught the same course in machine learning, a subject related to artificial intelligence. The classes were offered at different times, to different students.
About 60,000 students enrolled in each of the first two sessions. The researchers simply observed the behavior of these 120,000 registrants: how many people posted or voted frequently, etc.
About 112,000 students enrolled in the third class, the one in which the researchers introduced their targeted badges.
Their badge findings are based on comparing the behaviors of these two similar-size groups.
MOOC mystery
In addition to trying to incentivize participation from those who take an active role in the MOOCs, the researchers also sought to learn more about the majority of registrants who enroll but don’t complete the courses.
Here they analyzed behavior in six MOOCs: the three machine learning classes previously cited, plus three classes in Probabilistic Graphical Models (PGM), another computer science course.
Roughly 100,000 people registered in the three PGM classes, creating a study pool of about 230,000 registrants for all six MOOCs.
Looking at this large pool of registrants, the researchers observed and described five categories of behavior:
- Bystanders: the largest group, about 55 percent of all registrants, did little or nothing after signing up.
- Viewers: about 20 percent of MOOC registrants watched online lectures but did not hand in assignments.
- Collectors: roughly 15 percent of those who enrolled downloaded course videos but did not necessarily watch the lectures or hand in assignments.
- All-rounders: About 10 percent of registrants watched lectures and handed in assignments, performing activities that would qualify as completing the class.
- Solvers: About one person out of every hundred who signed up did not watch the lectures but handed in assignments to be graded, presumably testing previously-acquired knowledge.
Reflecting on these findings, Leskovec said many registrants have found their own valuable ways to use MOOCs.
“If I’m a teacher and I want to get my knowledge out there to the world, you shouldn’t judge my success simply by asking how many people finish the course,” he said.
