OUseful Info

Online Course Analytics

For some time, I've been running Google Analytics on the OUseful.info blog, and - err - somewhere else I'll post about another day.

One thing I'm keen to do is get access to OU search analytics, both internal and relating to search engine referrals, to see whether we can gain any useful curriculum development intelligence from the data: seeing what keywords students use to find courses, for example, or whether there are any frequent searches we can't accommodate.

I'm in the process of planning a course on gaming and interactive media design, for example, so it'd be handy to see whether anyone is searching the OU courses site with terms relevant to such a course, whatever they may be. (I've posted kite flying thoughts elsewhere about how we might be able to use search as a predictive tool: Search Powered Predictions and Click Powered Predictions...?).

Analytics can be used elsewhere in the system, of course - I'm personally not that interested (at the moment...!) in "academic analytics", using data about student behaviour to predict success in a course, identify students who may be struggling/about to drop out, and so on, though I do know that there are various people around the OU who have started taking an interest in such things. After all, when students drop out, we don't get any funding for them...

What is of interest to me, though, are analytics that describe how students may be interacting with our online course materials (this data of course is likely to feed in to "academic analytics", particularly with respect to identifying struggling or disengaged students, for example).

I hope to be able to post something a little more concrete about this over the summer, but for now I'm pondering the question of what sorts of data we want to collect.

After all, web analytics of the Google kind seem to be best suited to tracking user behaviour on sites with clearly defined goals, such as successfully completed shopping transactions.

For online course materials - like the Relevant Knowledge short courses - where 10 weeks worth of material are online, organised as weekly topics - identifying relevant goals and sensible "questions" we might ask of the analytics data (with a view to optimising/improving our course materials) is something we haven't really addressed.

So - what should we track? Or indeed, to try and simplify the problem, what might we track.

The Relevant Knowledge courses all tend to have a similar structure (although each course does tend to introduce it's own innovations). A course calendar lays out the content for each study week, as does an index of the course materials. Links to online course forums are provided at the top level of navigation. Courses may or may not have a glossary and/or a resource page that contains links to external resources referenced in the course materials. A search box that allows users to search the web based course materials (but not external sites, our downloadable course content (I think...) is provided on each page, along with the option to print the page on the course content pages.

(Searching over content linked to from the course materials is easily achieved using a bounded/custom search engine. For example, see OpenLearn Unit Search Hub which describes an automated route for creating a bounded search engine over domains linked to from an OpenLearn Unit.)

The course content itself is chunked into 10 to 20 HTML pages per week. Links are often provided to external websites. Links to downloadable documents (such as PDF files, movie or audio files) may be provided. The use of embedded audio/visual material (along with download links, as well as transcripts) is increasingly likely. Course materials include self-assessment exercises as well as narrative content. So-called SAQs (formative self-assessment questions) comprise of a question along with sample answer text. The answer text is revealed by clicking a form button or link (i.e. a trackable action is taken to reveal the answer). Online summative assessment materials are also provided - a CMA (computer marked assignment) around week 5, and an ECA (end of course assessment) in week 10. All assessment materials are "seen" from the start of the course.

So - what to track?

Here are some preliminary thoughts:

• number of unique visitors (are all students logging in? from how many different browsers?);


• number of visits per week (how often are students logging in, and when? Is there periodicity in access behaviour, aggregated over the cohort?);


• average time per visit (what's a sensible chunk size in terms of study time?);


• average number of page views per visit (again, useful intelligence relating to chunk size of teaching content);


• bounce rate (but why are students bouncing? Using the course pages for navigation/to go elsewhere? If so - where?);


• External links followed (are students following links at all? should we be providing other access points to external links, offering link collections by other means, such as feeding live bookmarks?);


• Pages printed (are students printing out the course content, or consuming it online?)


• Clicking through to course forums (looking at forum stats would also be useful, e.g. what percentage of students look at what percentage of messages?)


• click-thrus to SAQ answers - and time spent looking at SAQ answer pages;


• visitor behaviour around assessment pages (how often/how many times do students visit assessment pages? when? for how long? in the context of visiting other pages (possibly via search results) etc.);

Identifying sensible goals (which, if tracked, enables additional analytics, such as reverse path tracking (what links did a user follow to reach the goal).

Of course, making sense of course analytics is not just a question of collecting the data, but also interpreting it appropriately and not being misled by naive readings of it...

If you have thoughts about what else we could sensibly track, please post a comment or drop me a line. References to anyone else exploring this area would also be much appreciated...

PS one question I meant to pop down: if we were optimising online course content pages, what would we be optimising them for?

Posted by ajh59 at July 12, 2007 10:36 PM