Going Beyond the Brick: Assessing and Supporting Creativity Using AI-Powered Digital Games

May 6, 2024

Going Beyond the Brick: Assessing and Supporting Creativity Using AI-Powered Digital Games

Abstract

To support creativity, one should first assess it accurately. New techniques such as stealth assessment that use digital environments (e.g. digital games) can be used to assess and support creativity. In this paper, I discuss my passion for creativity and how my academic journey in creativity research started. Then, I discuss key studies and papers I wrote with my colleagues (including my Ph.D. dissertation). Specifically, I discuss the effectiveness of digital games on creativity, the effectiveness of creativity support tools in digital games, and the stealth assessment of creativity in a Physics game. Finally, I conclude with the future directions of my research in these areas and other areas such as the use of Artificial Intelligence (AI) in assessing and supporting creativity.

Download Complete PDF

Authors

Seyedahmad Rahimi
University of Florida
srahimi@ufl.edu

Unreliable Continuous Treatment Indicators in Propensity Score Analysis

May 6, 2024

Return to Spotlight Research

Unreliable Continuous Treatment Indicators in Propensity Score Analysis

Abstract

Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom.

Download Complete PDF

Authors

Gail A. Fish
University of Florida
gail.fish@ufl.edu

Walter L. Leite
University of Florida
walter.leite@coe.ufl.edu

An Introduction to Bayesian Knowledge Tracing with pyBKT

May 6, 2024

Return to Spotlight Research

An Introduction to Bayesian Knowledge Tracing with pyBKT

Abstract

This study aims to introduce Bayesian Knowledge Tracing (BKT), a probabilistic model used in educational data mining to estimate learners’ knowledge states over time. It also provides a practical guide to estimating BKT models using the pyBKT library available in Python. The first section presents an overview of BKT by explaining its theoretical foundations and advantages in modeling individual learning processes. In the second section, we describe different variants of the standard BKT model based on item response theory (IRT). Next, we demonstrate the estimation of BKT with the pyBKT library in Python, outlining data pre-processing steps, parameter estimation, and model evaluation. Different cases of knowledge tracing tasks illustrate how BKT estimates learners’ knowledge states and evaluates prediction accuracy. The results highlight the utility of BKT in capturing learners’ knowledge states dynamically. We also show that the model parameters of BKT resemble the parameters from logistic IRT models.

Download Complete PDF

Authors

Okan Bulut
University of Alberta

Jinnie Shin
University of Florida
jinnie.shin@coe.ufl.edu

Seyma N. Yildirim-Erbasli
Concordia University of Edmonton

Guher Gorgun
University of Alberta
swapnakumar@coe.ufl.edu

Zachary Pardos
University of California Berkeley
zp@berkeley.edu

How Teachers Influence Student Adoption and Effectiveness of a Recommendation System for Algebra

May 6, 2024

Return to Spotlight Research

How Teachers Influence Student Adoption and Effectiveness of a Recommendation System for Algebra

Abstract

Advanced learning technologies (ALT) have become increasingly available to teachers for classroom use. Research has suggested that many factors can influence teacher adoption and fidelity of use of ALT in the classroom, including teacher beliefs, knowledge and experience, technological factors, and instructional factors. However, there has been scarce research linking teacher factors to student adoption of ALTs. This study examined the relationships between teacher characteristics and practices and student adoption and learning gains with a video recommendation system embedded within a virtual learning environment (VLE) for Algebra. Secondary data was obtained from an experimental study conducted over one academic semester in middle and high schools in a southeastern state of the United States. The sample included 52 teachers and 2936 students. The data included teacher responses to three surveys, and student demographic and achievement variables. A random forest was used to predict the rate that the students followed video recommendations in the VLE. The results show that the recommendation followed rate is related to the teachers’ fidelity of use, frequency of student monitoring, and experience with the VLE. Most of the survey items specifically evaluating teachers’ beliefs about the recommender were important predictors of students’ following video recommendations. Teacher monitoring through a dashboard was the most important predictor. The analysis of treatment effect heterogeneity of the video recommendation system was performed using the generic machine learning inference (GenericML) method paired with random forests. Results show that teachers of students who benefited most reported spending more time using the videos of the VLE and following student progress through the dashboard, but less time on the VLE than teachers of students who benefit the least. Teachers of students who benefitted the least had larger classrooms, struggled more with the challenges due to the Covid-19 pandemic, and spent less time with classroom planning. The results support the recommendation that teacher professional development for ALT should engage groups of educators in increasing their experience with the application so that they build comfort and confidence in its use in ways in which students are most likely to benefit.

Download Complete PDF

Authors

Walter L. Leite
University of Florida
walter.leite@coe.ufl.edu

Amber D. Hatch
University of Florida
hatcha@ufl.edu

Huan Kuang
University of Florida

Catherine Cavanaugh
University of Florida
cathycavanaugh@coe.ufl.edu

Wanli Xing
University of Florida
wanli.xing@coe.ufl.edu

Propensity Score Analysis with Unreliable Covariates: A comparison of Five Reliability-adjustment

May 6, 2024

Return to Spotlight Research

Propensity Score Analysis with Unreliable Covariates: A comparison of Five Reliability-adjustment

Abstract

Propensity score analysis (PSA) is often used by researchers to control for selection bias due to multiple covariates in quasi-experimental studies. However, covariates with low reliability have been shown to lead to biased treatment effects estimates in PSA. Latent variable analysis is a promising strategy to reduce the negative effects of observed variables’ measurement error. This Monte Carlo simulation study compared the performance of five methods to adjust propensity scores for unreliability. The results indicate that the latent variable model with inclusive factor score (PSIF) generated the lowest relative bias of treatment effect estimates, followed by propensity score estimation with structural equation model (PS-SEM). However, only PSIF provided unbiased treatment effect estimates across conditions with high, medium and low reliability. The results also show that evaluation of covariate balance can be misleading when there are unreliable covariates, because treatment effect estimates can be biased when covariate balanced is deemed adequate.

Download Complete PDF

Authors

Huibin Zhang
University of Florida

Walter Leite
University of Florida
walter.leite@coe.ufl.edu

AI Made by Youth: A Conversational AI Curriculum for Middle School Summer Camp

May 6, 2024

Return to Spotlight Research

AI Made by Youth: A Conversational AI Curriculum for Middle School Summer Camp

Abstract

As artificial intelligence permeates our lives through various tools and services, there is an increasing need to consider how to teach young learners about AI in a relevant and engaging way. One way to do so is to leverage familiar and pervasive technologies such as conversational AIs. By learning about conversational AIs, learners are introduced to AI concepts such as computers’ perception of natural language, the need for training datasets, and the design of AI-human interactions. In this experience report, we describe a summer camp curriculum designed for middle school learners composed of general AI lessons, unplugged activities, conversational AI lessons, and project activities in which the campers develop their own conversational agents. The results show that this summer camp experience fostered significant increases in learners’ ability beliefs, willingness to share their learning experience, and intent to persist in AI learning. We conclude with a discussion of how conversational AI can be used as an entry point to K-12 AI education.

Download Complete PDF

Authors

Yukyeong Song
University of Florida

Gloria Ashiya Katuka
University of Florida

Joanne Barrett
University of Florida

Xiaoyi Tian
University of Florida

Amit Kumar
University of Florida

Tom McKlin
University of Florida

Mehmet Celepkolu
University of Florida
mckolu@ufl.edu

Kristy Elizabeth Boyer
University of Florida
keboyer@ufl.edu

Maya Israel
University of Florida
misrael@coe.ufl.edu