In this short film, I look at the various data available per-course and per-specialization on the Coursera website. I focus on the number of ratings in an individual course as a rough proxy for course completion, and therefor a rough proxy for rigour.

Transcript

So I had a potentially interesting insight into Coursera’s completion rate and the numbers that they publish on their website.

The first thing is if you just Google “Coursera completion rate”, there is this claim that it’s 15% of enrolled students. (http://www.katyjordan.com/MOOCproject.html)

Looking at the Data Science specialization by Johns Hopkins, they publish this number up front of 880,000 students. But if we click in here you see some more data: one is 72,880 ratings. Now I added up all of the ratings for each course [in the specialization], and they total to 72,880. So, this number [72,880 ratings] is not 73,000 people rating the specialization, this number is 73,000 people rating one of the courses in the specialization.

So if we carry that theme a little bit further, we notice that the first course has 26,000 ratings. 5,500 reviews, but reviews are really just a subset of ratings, so I think this [the ratings] is the more accurate number if we are going to assume that ratings somehow correlates to completion rate.

And we notice that as we get into course 2 the number decreases to 18,000, course 3 it’s less than half at 7,150. Course 4 it’s down again to 5,300. As we get all the way to the final course [in the specialization], we are down to 1,000 ratings. That’s down from 26,000 for the first course.

Hopkins rating data

So, you might ask the question: of the people that complete the final course, how many of them leave a rating? Well, we know that the rate of review is a quarter of the ratings. So let’s use that number and say that maybe a quarter of the people [who complete the final course] leave a rating. So that would indicate that 4,000 people out of the 880,000 [enrolled] students actually completed all [ten] courses of the Johns Hopkins Data Science Specialization.

The reason I wanted to ask these questions is, you want to kind of know how rigorous a course or specialization is. Is it a “degree mill” in the sense that anybody who starts finishes? Or is it very difficult and few people finish?

Looking at another course, the Kotlin programming course by JetBrains, the company that invented Kotlin, and the company that built the Android Studio toolkit for making Android apps: we see that there are 41,000 students enrolled. But again, there are only 1,100 ratings.

So using our math from before, and saying that 25% of the students leave ratings when they finish the course, it could be 4,000 out of 41,000 finish the course. And that’s 10% so maybe that’s approximately [near the 15% that the study above is claiming.]

What is it about the branding of this course that doesn’t communicate that it is in fact a pretty tough thing to pass, and that passing it does actually indicate something about the bearers competence with Kotlin.

If this were branded as “Kotlin Certification”, or something like that, it may be that it would carry more weight with people reviewing resumes.

For a lot of this work, just completing it puts you in the 85th percentile, or higher depending on how these numbers sort out.

Conclusion

Early on there was all this criticism of massively open courses that the completion rates were abhorrently low. What you saw is companies starting to obfuscate those numbers, focusing on enrollment, and keeping the actual success rate private.

I think we are past those early days now though, and people really do see some value in this type of course. However now we have this problem that, using the example above, if I finish the Hopkins data science specialization it looks like I’m one of 880,000. Whereas really (and heuristically), it looks like only 4,000 people of the 880,000 actually complete the specialization.

Even if the completion estimate based on reviews is off by 5 fold, we are still talking about series where 93% of people who enter, don’t come out the other end.

What I’m arguing is that is a badge of honor, not shame for Coursera and specialization holders. If all of the people who attempt the certification pass, then it has no value.

Finally, and this is subjective, Coursera has a low cost compared to say an undergraduate or masters degree. If a “real” degree had these completion rates and the ensuing financial burden with no payoff for students, we might cry “foul!”. However since Coursera is so cheap, the fact that the courses are actually quite difficult is a good thing.