PHOTO BY ALLISON SHELLEY/THE VERBATIM AGENCY FOR EDUIMAGES

Most dropout prevention models focus on crisis management (addressing those emergent issues regarding the schooling or welfare of a student), short- and long-term plans for student and family engagement, and intervention plans and contracts crafted together with the student and family. Relationship-building and mentoring are frequently mentioned in the data.

Other processes include fostering the data culture surrounding dropout prevention, early identification, longitudinal analysis, and progress monitoring. Early warning systems (EWS) using analytical tools that have been trained against prior years’ data, can reliably predict dropout risk in individual students so that educators may intervene early to help avert this from happening. Risk profiles for dropouts aren’t always useful since students often do not conform to the profiles. Researchers with the Montana Office of Public Instruction developed objective, evidence-based indicators that respond to student context. These indicators show when a student may be at risk and provide signs of when to intervene.

The ability of EWS to reliably predict students at risk of dropping out has been noted in different national contexts. However, research into the processes and outcomes of these models is limited. The research has focused on the development and norming of indicators. Evidence of processes and outcomes in the context of effectiveness is uneven across systems and school types. There are few rigorous or evidence-based studies on EWS, and the evidence base in rural areas is thin. The link between the diagnostic tool and the outcome of the interventions in Montana (improvement in graduation rates, for example) warranted further investigation.

Montana developed an EWS in 2012 that focused on early identification and the development of a data culture. To better understand these processes, our study looked at the level of adoption of schools in the Montana EWS program and the policy changes that occurred at the school level. We looked at the development of a data culture, which involves institutionalizing processes that address the whole course of an intervention, including early identification and progress monitoring. While schools using early identification show that they are using the tool, progress monitoring focuses on the need to reassess interventions and is a sign of the robustness of a data culture. This frequently occurs among schools we classify as high adopters.

Analysis of adoption of an EWS is important since it yields data on the scope of a school’s implementation by providing a framework for analyzing how well they implemented established models. Marken, Scala, et al. (2020) mention that EWS schools should focus on understanding the fidelity of their model to established research-based evidence on the level of adoption, duration, program goals, and student engagement. By focusing on schools that used Montana’s EWS, we are contributing to the research and policy literature about dropout and early identification. The duration is measured by the number of years schools have been using the Montana EWS and the intensity of this use can be seen in the number of times data was shared with the state per year to generate student risk assessments. The location of the research is important. Few EWS studies have focused on rural areas and trends with how rural populations experience the EWS. Program goals can be seen in the perceived value of the program and the vision to implement it. Student engagement is found through the noting of the importance of relationships as evidenced by the vision of school leaders and the mentoring practice of teachers.

Early Identification and Triage

The core of Montana’s EWS focuses on early identification understood through the lens of the probability a student will drop out of school. These systems are integrated into student information systems, Montana’s Statewide Longitudinal Data System — a unit of the Montana Office of Public Instruction. The EWS is built into an online portal containing student- and school-level data. The dropout probabilities are supplemented by various risk factors: academics, attendance, behavior, and mobility indicators.

An EWS score measures the probability that a student will drop out given the current risk that may inhibit student success. In Montana, the system itself is efficient. There is a strong inverse relationship between a student’s EWS scores and eventual on-time graduation. Among students flagged as at an extreme risk of dropping out, only 63% graduate on time, while 97% of students never predicted to drop out do indeed graduate within four years of their ninth-grade entry. The EWS scores are strongly associated with eventual dropout. Data from the prior year’s outcomes is trained against student profiles of the current year. The predictions between years are consistent. EWS scores indicate a higher probability of dropout than happens each year for the student, implying that schools that use the system are alerted in advance of the student dropout risk.

The process of engaging with the EWS data by administrators and faculty is known as triage (O’Cummings & Therriault, 2015). It is one marker of the level of adoption. Effective schools have learned to triage students and define student needs based on the findings. High-adoption schools of the Montana EWS model emphasize a direct tie of data to the intervention. This involves using data over time and readjusting interventions as needed. The core of triage is to establish thresholds of interventions based on student needs that allow for the greatest short-term impact. These thresholds reflect policy choices in Montana’s schools. This is done both with early identification and progress monitoring. The end goal of the intervention is to have a universal screening in each school that provides data for all students that allow tiered responses in line with Multi-Tiered System of Supports (MTSS) frameworks.

As triage implies, local policies determined the thresholds to which students would be given interventions. This means that academic, attendance, and behavioral support are given to students at different rates depending on the demands of the intervention and local capacity of the system. Locally defined thresholds identify specific factors to target. For example, a student challenged by attendance may find attendance incentives meaningful, and faculty may find this an indicator of student growth.

High-adoption schools in Montana tend to have established procedures linked to MTSS or school-based intervention teams, formal and informal dissemination of EWS data, a clear tie of the data to intervention, and explicit follow-up procedures to an intervention (progress monitoring). In Montana, variation between high adopters did occur based on progress monitoring, where some schools more frequently used data to follow up to an intervention. Issues such as vision, value, data culture, and opinions surrounding professional development remained the same.

Schools with medium adoption have few of these traits. One thing noteworthy about these schools is they tend to not have active MTSS processes. Dissemination of the data occurs among a limited number of stakeholders, involves informal practices, and data use surrounding intervention is limited to early identification. They have the tool; however, there are questions surrounding data use that only future policy making and professional development can address. In many cases, school officials and staff continued to do what they would have done in the absence of an EWS, such as gathering data from multiple sources about attendance, behavior, and coursework. In these cases, EWS may have served as a reference and a means to verify local and anecdotal data.

Scaling Up

The EWS program did experience challenges while scaling up. Many schools that started the EWS process uploaded data only a handful of times (low adoption). Archival data suggested that the motivations (economic disadvantage, locale, student demographics, graduation rates, attendance, and achievement) were similar between medium to high adopters and low adopters, meaning that the justification to participate in the Montana EWS, or any EWS, is similar. School size is important. Small schools are more frequently among low adopters than with medium to high adopters. Medium- to high-adopting schools were more frequently larger schools and were from town areas.

There were many significant differences when comparing non-adopters to EWS adopters. These indicated that schools in the non-adoption category have less economic disadvantage; are more likely small schools; and have higher graduation rates, satisfactory attendance rates, and rates of proficiency on state assessments. They also are predominantly elementary schools. Although EWS encompasses grades three to 12, there is less of an emphasis on the program in the lower grades. Hence, the risk profile for schools that showed interest in the EWS (adopters) is different than the rest of the schools in the state that showed no interest.

Data on the effectiveness of the tool begins with the key indicator: progress monitoring. Unfortunately, few districts indicated that they do this step. This does not mean that follow-up is not occurring. Indeed, the survey did indicate a high degree of follow-up. It indicates a lack of integration of data in the process. Of those schools reporting, small schools (fewer than 450 students) are more likely to engage in follow up. This indicates the importance of relationship building in small schools, effective goal setting, and clear measures to base follow-up on.

Schools were hesitant to discuss the success of their processes. They were more than willing to discuss individual outcomes. Many survey and interview respondents cited that at least 75% of identified students (at risk and extremely at risk) went on to graduate or go on to the next grade. However, the respondents expressed the need to adjust their processes and make them more relevant to changing student needs. The tenor of the responses focused on the need to reshape the data use model to address the remaining 25% that tends to have changing risk situations.

The Role of the State

Key to the success of educators who participate in an EWS program is the degree to which their efforts are supported by a framework of implementation provided by the state education agency. Administrators report that the support from the Montana EWS team is responsive: “I feel like I have always been heard.” A data-driven intervention culture is needed at the state level to support the development of an EWS. The focus is on providing responsive and adaptable professional development.

Many of our research themes can inform the evidence base behind the processes involved in EWS. We focus on the development of the data culture and the crucial role of engagement with the data. The primary finding is that schools should make longitudinal data more accessible. Moreover, the state has a role: developing a data culture at the state level which can in turn support educators at the local level. There is evidence that this process of engaging with the data does indeed become more intense over time.

Schools that started the process before 2015 are more likely to use EWS data than schools that started after. It may indeed take time, and an understanding of scale, capacity, and priorities to make EWS successful in these schools.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305S210011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute, the U.S. Department of Education, or the Montana Office of Public Instruction.