RMC Research is working with Imagine Learning, a company which seeks to provide differentiated language and literacy instruction to students using individualized technologies, to evaluate a Universal Screener. Universal Screeners identify students who are at risk for poor academic outcomes. The evaluation aligns with five technical criteria used by The National Center on Response to Intervention (NCRTI) to rate the quality of Universal Screeners. The criteria include: classification accuracy, generalizability, reliability, validity, and disaggregated reliability, validity, and classification data for diverse populations. A brief overview of each criteria is provided below.
Classification accuracy provides information about the extent to which classification based on the screener accurately identifies students who are at risk of poor academic outcomes. The criteria for generalizability is primarily based on samples used for studies of universal screeners; for example, screeners that meet high generalizability criteria were assessed using a sample that is large, nationally representative, and cross-validated. Reliability, or the extent to which screeners are consistent in classification of students, is considered high quality if the screener demonstrates high reliability on at least two types of reliability analyses (e.g., split-half reliability, coefficient alpha). Validity, the extent to which a tool measures what it intends to measure, can be demonstrated with expert reviews to ensure each item and the overall assessment includes appropriate content according to what it intends to measure, as well as statistical analyses. For example, analyses may examine correlations between scores on the screener and scaled scores on a state reading assessment to identify its predictive validity. Finally, the NCRTI emphasizes the importance of ensuring a universal screener is appropriate for all groups. Thus, they require the classification accuracy, reliability, and validity analyses are replicated for a disaggregated subgroup of students. RMC Research employs a number of analysis techniques to evaluate Universal Screeners according to these criteria.
For more information on the NCRTI rubric and each of the technical criteria, go to: http://www.rti4success.org/essential-components-rti/universal-screening