We thought about how we could use data to improve educational design, to see how courses are built up and what effect this has on students’ behaviour.
Jan-Willem Doornenbal, blended learning co-ordinator
Challenge
To understand course design quality through data
As students interact with a learning platform, they generate data. This data offers a valuable insight into levels of engagement, students’ progress, participation in group activities and more. Educators can harness these insights to evolve teaching content, delivery and course administration and thereby optimise results. They can also use it to generate action that can help students maximise their potential by, for example, intervening to bring struggling learners back on track.
AUAS recognised these benefits and opportunities that come from using Brightspace.
In 2021, the university set about using data insights to understand the impact of educational design on study results. By mining the data, and drawing conclusions, the team aimed to reveal the effect of course design on student behaviour.
“The biggest challenge was thinking how to convert data, like the number of topics students visit and the number of assignments they take, into measurements,” Jan-Willem Doornenbal, blended learning co-ordinator, explains. “We needed to work out how to convert the data to tell us something about the quality of course design.”
Solution
Comprehensive data generates student engagement insights
Jan-Willem and colleagues used a theory of change model to track backwards from the desired outcome, being student success, to the activities and outputs required to get there. These include setting clear expectations, establishing ways of working, motivating students, evaluating and giving feedback, communicating and establishing a safe environment.
The team then organised the many individual actions that underpin these activities-for example, “guide students in peer assessments”, “create a sense of community and build rapport”-into four categories: evaluate, organise, activate and share. Then, they mapped Brightspace data to these categories.
We involved the lecturers and the research director of the faculties. We asked what we could do with the data that is available in the Brightspace system.
Jan-Willem Doornenbal, blended learning co-ordinator
The team used a comprehensive set of data inputs to inform their analysis. This included data from Brightspace’s tools and features such as announcements, course awards, grade objects, release conditions, quiz and survey objects.
Student-use data was also included, as Jan-Willem explains: “We look at what students actually do inside a course. Which topics they visit, which topics they complete, which assignments they do, how much time they spend, which external tools they open. This all adds up into a timeline of student activity.”
Additional data provided insight into assignment feedback that students had accessed, discussion posts they’d written and read, and quizzes they’d completed.
All data and the solution and approach were extensively assessed against the value that would result and ethical and privacy considerations.
The total output was then presented to educators as a visual dashboard, with courses scored against each of the categories. These scores were qualified, with detail on the data that contributed to them, and actioned through suggestions on tweaks that could be made to improve outcomes, for example by adding in announcements or introducing more interactive activities. This further generated a student engagement dashboard across courses.
Result
Actionable insights through data analysis
The work the AUAS team has done in establishing a model for harnessing data from Brightspace and turning it into actionable insights will stand it in good stead for continuous course improvement.
Now we can measure, on a day-to-day basis, how active students are.
Jan-Willem Doornenbal, blended learning co-ordinator
Jan-Willem believes it will take time to fully capitalise on the output but relishes where it will take them. “We will build on what we’ve delivered, through many cycles,” he says. “It will take time to demonstrate what we can bring, what it means for teachers, and how it can help improve their course design.”
The team is working with teaching staff to provide the tool in a way that delivers the biggest self-evaluation benefit. “Some really look forward to having a tool like this, giving feedback about what students really do and knowing the direct effect of teaching,” says Jan-Willem.
However, Jan-Willem also acknowledges that quantifying impact is a new direction and one that will give some teachers concerns. Accordingly, the team is working to ensure engagement data is harnessed positively to provide helpful insights for continuous course design improvements.
AUAS set out to use the wealth of data generated through Brightspace to reveal the extent of students’ engagement with courses. By taking a systematic approach to categorising actions, mining data and mapping it to categories, the team developed a tool that makes best use of the insights from analytics. The output from the tool will be used to inform decision-making as courses are refined and evolved to support the best student outcomes.
To discover Brightspace for higher education, visit d2l.com/en-eu/higher-education.
