Learning Gains

Overview

This three year longitudinal ABC learning gains project compared the relative performance of 195,250 students’ learning gains using two specific Phases:

1. Secondary data analysis of existing ABC data for academic years 2013-2015;
2. 2) Mixed-method study of new ABC data for academic years 2015-2017, with longitudinal interviews of 10-15 participants per institute.

The ABC project primarily explored the “validity” of the concept of ABC learning gains using existing data structures placed within universities. While many other learning gains projects struggled with getting sufficient responses from students over several waves of data collection, the simplicity of the idea of using existing university data to “measure” learning gains was naturally appealing. For affective learning gains, our multi-level modelling research highlighted that using readily available student satisfaction data as proxies for affective learning gains was not a fruitful avenue for further research. Primarily this was related to a lack of consistent data over time for sufficiently large numbers of “representative” students. Furthermore, without standardisation of student satisfaction approaches, constructs, and items across institutions it was not possible to compare potential differences in learning gains over time. For behavioural learning gains, our multi-level modelling showed that linking behaviour engagement from one module to the next based upon Learning Management System data led to inconsistent findings.

The focus of the project was on cognitive learning gains and this is the area where we have made most conceptual, theoretical, and empirical progress, evidenced by 50 dissemination activities at leading conferences and academic journals, and public engagement. We used five different approaches to measure cognitive learning gains (i.e., within module, from one module to the next one, within a qualification, across qualifications, across institutions). Overall, our findings indicated that although assessment scores could potentially be used as proxies for cognitive learning gains, a large proportion of these gains were not related to individual student ability, effort, and/or cognition. How teachers designed courses, marked and graded assessments substantially influenced the grade trajectories of students, which were disciplinary, and institutionally influenced and determined.

Although it might be attractive to use grades as a proxy for learning outcomes, both practice and our research highlights large risks in terms of potential gaming if learning gains are based upon students’ performance alone. This was further confirmed from fine-grained triangulated qualitative analyses of lived experiences of 52 students, whereby some students with “low” quantitative learning gains made substantial learning gains over time, while other high performing students expressed substantial ABC concerns. Overall, our findings indicate that policy makers need to be extremely cautious in using simple learning gains proxies (e.g., grades, entry score to final degree classification) to establish Teaching Excellence within and across institutions.

The role of IET

IET was the main project lead and contributor to this project, leading the main project and specifically the evaluation and publication of results.

Impact

In The Open University, the ABC project had a substantial impact on the teaching and learning practice, as the longitudinal grade trajectories were substantially different than expected (see below under section “success criteria”). This has led to extensive discussions within and between Faculties, boards of exams, Faculty learning and teaching committees, as well as within the Assessment Programme Management Group, and senior exec. In a drive to provide more consistency in long-term grading and supporting the students to be successful at the OU, the OU has moved towards single component assessment in 2018/19, while providing more specific grade descriptors within and across qualifications, and finally more explicit sign-posting of the students’ journey through their qualification(s). 

50 presentations at national and international conferences has helped to shape our thinking about the complexities of measuring ABC learning gains, as well as having critical discourse how our findings would resonate across the HE sector, and potentially be different at other institutions. In particular the last two knowledge exchange events have been essential to compare and contrast our findings with other institutions, and critically reflect upon the generalisation of our complex findings. In total 10 publications have been published in a range of outlets.