Stealth Assessment and Digital Learning Game Design

Stealth Assessment and Digital Learning Game Design

Abstract

Stealth assessment is an innovative way to measure a set of student competencies through gameplay. The process starts with a competency model that is comprised of everything you want to measure during the assessment, the theoretical concepts being assessed. The competency model is the glue of the stealth assessment, as alignment of the task model and evidence model to the competency model is key to creating a valid stealth assessment. The following chapter examines the different design challenges and approaches used for creating and implementing a stealth assessment through the lens of two design cases. In the first case, a stealth assessment is implemented into an already developed game. In the second case, a stealth assessment is being expanded and a new competency model is needed to represent the expanded content. The chapter concludes with a discussion comparing the two approaches to implementing stealth assessment and implications for future design.

Authors

Ginny L. Smith
Florida State University

Valerie J. Shute
Florida State University

Seyedahmad Rahimi
University of Florida
srahimi@coe.ufl.edu 

Chih-Pu Dai
University of Hawaiʻi at Mānoa

Renata Kuba

Getting the First and Second Decimals Right: Psychometrics of Stealth Assessment

Getting the First and Second Decimals Right: Psychometrics of Stealth Assessment

Abstract

Stealth assessment, like all assessments, must have three essential psychometric properties: validity, reliability, and fairness. Evidence-centered assessment design (ECD) provides a psychometrically sound framework for designing assessments based on a validity argument. This chapter describes how using ECD in the design of a stealth assessment helps designers meet the psychometric goals. It also discusses how to evaluate a stealth assessment’s validity, reliability, and fairness after it is designed and implemented.

Authors

Seyedahmad Rahimi
University of Florida
srahimi@coe.ufl.edu 

Russell G. Almond
Florida State University

Valerie J. Shute
Florida State University

Chen Sun

Pedagogical discourse markers in online algebra learning: Unraveling instructor’s communication using natural language processing

Pedagogical discourse markers in online algebra learning: Unraveling instructor’s communication using natural language processing

Abstract

Despite the proliferation of video-based instruction and its benefits—such as promoting student autonomy and self-paced learning—the complexities of online teaching remain a challenge. To be effective, educators require extensive training in digital teaching methodologies. As such, there’s a pressing need to examine and comprehend the intricacies of instructors’ communication patterns within this context. This research addresses the pressing need to understand pedagogical discourse in online video lectures in Algebra classes by employing computational linguistic tools and natural language processing (NLP). Using transcripts from 125 Algebra 1 video lectures—comprising 4962 instances of pedagogical discourse—from five instructors at Math Nation, a virtual math learning environment, we analyzed the conveyance of linguistic, attitudinal, and emotional nuances. With the aid of 26 Coh-Metrix and SÉANCE features, we classified educators’ language choices, achieving an accuracy of 86.7%. Furthermore, variations in language choices, as signified by discourse markers, were examined through a K-means clustering approach. The resulting 17 clusters were grouped into interpersonal, structural, and cognitive pedagogic functions. Through this exploration, we demonstrate the promising potential of NLP in efficiently deciphering pedagogical communication patterns in video lectures. These insights open a new avenue for research, aimed at assessing the efficacy of digital instruction by scrutinizing pedagogical discourse characteristics in computer-based learning environments.

Authors

Jinnie Shin
University of Florida
jinnie.shin@coe.ufl.edu

Renu Balyan
SUNY Old Westbury
balyanr@oldwestbury.edu

Michelle P. Banawan
Asian Institute of Management
mbanawan@aim.edu

Tracy Arner
Arizona State University
tarner@asu.edu 

Walter L. Leite
University of Florida
walter.leite@coe.ufl.edu 

Danielle S. McNamara
Arizona State Universty
dsmcnamara1@gmail.com 

Scaffolding Block Coding Through Multiple Entry Points

Scaffolding Block Coding Through Multiple Entry Points

Abstract

A major tenet of Universal Design for Learning (UDL) is providing flexibility and choice. This pilot study investigated a UDL-based approach called multiple entry points for project creation in Scratch as a way of scaffolding learning for block-based programming projects. Allowing student choices for interactions with computer science (CS) learning decreases barriers to entry for all students, especially those with disabilities. Multiple entry points provide varied challenges and scaffolding and aligns with the UDL framework as delineated by CAST [3]. Multiple entry points involve presenting learners with different versions of projects, requiring different kinds of interactions and scaffolds that assist learners in achieving the goals of UDL: Multiple means of engagement, representation, and action & expression – especially those related to comprehension, perception, and language & symbols. The pilot identified that the names given to the options as well as the order of the options impacted learner selections. This information can inform the development of future multiple entry point projects.

Authors

Joanne Barrett
University of Florida

Maya Israel
University of Florida
misrael@coe.ufl.edu 

How are Autistic People Involved in the Design of Extended Reality Technologies? A Systematic Literature Review

How are Autistic People Involved in the Design of Extended Reality Technologies? A Systematic Literature Review

Abstract

The primary aim of this systematic review is to investigate the inclusion of autistic individuals in the design process of immersive technologies. This study follows the preferred reporting items for systematic reviews and meta-analyses standards for systematic literature reviews. To ensure the research questions and subsequent stages of the review incorporate pertinent parameters, the problem, interest, context framework has also been employed. Findings highlight that, while early proponents of immersive technology emphasized the importance of user involvement in design of new technology, immaturity of the technology often limited the implementation of direct user input to the design process. Nonetheless, analysis of the literature published between 2002–2022 identified 20 studies in which substantial influence of autistic individuals and stakeholders was found in the design process of immersive technologies. The roles of autistic individuals varied from active co-designers and co-creators to essential contributors in refining prototypes and providing critical feedback, ensuring the final products align with their needs and preferences. Results underscore the need to align research and design of immersive technologies more closely with the priorities and preferences of autistic individuals. Further is needed regarding actively involving autistic individuals in the design and implementation of immersive technology applications. On this basis, we maintain that more inclusive and effective deployment of immersive technologies is needed in order to ensure that resultant technologies are fit for purpose and address the actual needs of the autistic community.

Authors

Nigel Newbutt
University of Florida
nigel.newbutt@coe.ufl.edu 

Noah Glaser
University of Missouri

Marc Sonley Francois
University of Florida
francoism@ufl.edu 

Matthew Schmidt
University of Florida
matthew.schmidt@coe.ufl.edu 

Sue Cobb
University of Florida