How Teachers Influence Student Adoption and Effectiveness of a Recommendation System for Algebra

Abstract

Advanced learning technologies (ALT) have become increasingly available to teachers for classroom use. Research has suggested that many factors can influence teacher adoption and fidelity of use of ALT in the classroom, including teacher beliefs, knowledge and experience, technological factors, and instructional factors. However, there has been scarce research linking teacher factors to student adoption of ALTs. This study examined the relationships between teacher characteristics and practices and student adoption and learning gains with a video recommendation system embedded within a virtual learning environment (VLE) for Algebra. Secondary data was obtained from an experimental study conducted over one academic semester in middle and high schools in a southeastern state of the United States. The sample included 52 teachers and 2936 students. The data included teacher responses to three surveys, and student demographic and achievement variables. A random forest was used to predict the rate that the students followed video recommendations in the VLE. The results show that the recommendation followed rate is related to the teachers’ fidelity of use, frequency of student monitoring, and experience with the VLE. Most of the survey items specifically evaluating teachers’ beliefs about the recommender were important predictors of students’ following video recommendations. Teacher monitoring through a dashboard was the most important predictor. The analysis of treatment effect heterogeneity of the video recommendation system was performed using the generic machine learning inference (GenericML) method paired with random forests. Results show that teachers of students who benefited most reported spending more time using the videos of the VLE and following student progress through the dashboard, but less time on the VLE than teachers of students who benefit the least. Teachers of students who benefitted the least had larger classrooms, struggled more with the challenges due to the Covid-19 pandemic, and spent less time with classroom planning. The results support the recommendation that teacher professional development for ALT should engage groups of educators in increasing their experience with the application so that they build comfort and confidence in its use in ways in which students are most likely to benefit.