Supplemental COVID Slide Report Data
This morning, EmpowerK12 released a report on COVID-19’s Impact on Student Achievement and Academic Growth in DC. We received more than 30,000 students' fall baseline assessment data from DC Public Schools and charter schools across the city to estimate the pandemic's impact on student achievement. Our initial report and slide deck go into more detail, here are the key takeaways:
Students have lost 4 months of learning in math and 1 month of learning in reading
At-risk students have lost 5 months of learning in math and 4 months of learning in reading
Achievement gaps are growing in DC, and at-risk students are falling significantly behind
DC COVID learning slides are similar to national COVID learning slides
The spring COVID slide was not as dire as predicted this spring
Evidence of further slide, particularly for math, during the extended fall assessment window
Students who are typically lower performing or attend schools that serve higher percentages of at-risk students were more likely to be missing from the fall data
The report contains data we were able to squeeze into production timelines. This blog post contains additional analyses related to the report, including thorough statistical significance testing of key data points from the report and cuts of the data that fell outside the report’s focus on grades K-10 math and reading.
Data by Ward of School
Our report looked at results District-wide and by student group. We did offer cuts of the data by where students attended school, either east or west of the river, but not by individual ward. Below are some of the key charts and graphs from the report by ward of school. Note, for ELA percentiles in grades 3-8, the primary source of data is from charter schools, and they are not located in every ward.
Statistical Significance Testing
For many of the data graphs and charts presented in the full report, we also ran statistical significance testing. The following is a comprehensive list of the analyses we conducted for key student groups. We compared outcomes for each student group between fall 2019 and fall 2020, “Within Subgroups,” and between student groups looking for differentiated COVID impact.
Math Within Subgroup Results
For all analyses of NWEA MAP Math test results within subgroups and across years, we conducted a paired two-sample two-tailed t-test for means.
There was a significant effect for At-Risk students, t(978) = 5.69, p < .001, with At-Risk students experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for Not-At-Risk students, t(1373) = 4.42, p < .001, with Not-At-Risk students experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for students with an IEP, t(468) = 0.498, p = .618, with students with an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for students without an IEP, t(1883) = 7.54, p < .001, with students without an IEP experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for students who live East of the Anacostia River, t(545) = 4.70, p < .001, with students who live East of the Anacostia River experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for students who live West of the Anacostia River, t(1758) = 5.42, p < .001, with students who live West of the Anacostia River experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for At-Risk students with an IEP, t(251) = -0.282, p = 0.78, with At-Risk students with an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for Not-At-Risk students without an IEP, t(1156) = 4.35, p < .001, with Not-At-Risk students without an IEP experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
Math Between Subgroup Results
For all analyses of NWEA MAP Math test results between subgroups, we conducted a two-sample two-way t-test.
Test Assumed Equal Variances:
There was not a significant difference between COVID’s effect on At-Risk students and Not-Risk-students, t(2351) = -1.21, p = 0.22, when looking at the difference in F20-F19 and F19-F18 achievement.
There was not a significant difference between COVID’s effect on students who live East of the Anacostia River and students who live West of the Anacostia, t(2303) = -1.01, p = 0.31, when looking at the difference in F20-F19 and F19-F18 achievement.
There was a significant difference between COVID’s effect on At-Risk students with an IEP and Not-At-Risk students without an IEP, t(1407)= 2.10, p=0.036, when looking at the difference in F20-F19 and F19-F18 achievement. Specifically, COVID affected Not-At-Risk students without an IEP more than At-Risk students with an IEP.
Test Assumed Unequal Variances:
There was a significant difference between COVID’s effect on students with an IEP and students without an IEP, t(727) = 2.96, p = 0.003, when looking at the difference in F20-F19 and F19-F18 achievement. Specifically, COVID affected students without an IEP more than students with an IEP.
ELA Within Subgroup Results
For all analyses of NWEA MAP and iReady ELA test results within subgroups, we conducted a paired two-sample t-test for means (two-tailed).
There was a significant effect for At-Risk students, t(94) = 2.73, p = .008, with At-Risk students experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for Not-At-Risk students, t(150) = -0.15, p =.88, with Not-At-Risk students experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for students with an IEP, t(48) = 1.00, p = .32, with students with an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for students without an IEP, t(196) = 1.49, p = .13, with students without an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was a significant effect for students who live East of the Anacostia River, t(61) = 4.14, p < .001, with students who live East of the Anacostia River experiencing a decline in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for students who live West of the Anacostia River, t(183) = -0.436, p =0.66, with students who live West of the Anacostia River experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for At-Risk students with an IEP, t(24) = 1.62, p = 0.11, with At-Risk students with an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
There was not a significant effect for Not-At-Risk students without an IEP, t(126) = 0.011, p = .99, with Not-At-Risk students without an IEP experiencing no change in the average difference in Fall to Fall achievement between the current school year and the previous year.
ELA Between Subgroup Results
For all analyses of NWEA MAP and iReady ELA test results between subgroups, we conducted a two-sample t-test (two-tailed).
Test Assumed Unequal Variances:
There was a significant difference between COVID’s effect on At-Risk students and Not-At-Risk students, t(168) = -2.39, p =0.018, when looking at the difference in F20-F19 and F19-F18 achievement.
Specifically, COVID affected At-Risk students more than Not-At-Risk students.
There was not a significant difference between COVID’s effect on students with an IEP and students without an IEP, t(66) = -0.344, p < 0.73, when looking at the difference in F20-F19 and F19-F18 achievement.
Test Assumed Equal Variances:
There was a significant difference between COVID’s effect on students who live East of the Anacostia River and students who live West of the Anacostia, t(244)= -4.04, p < .001, when looking at the difference in F20-F19 and F19-F18 achievement.
Specifically, COVID affected students who live East of the Anacostia River more than students who live West of the Anacostia.
There was not a significant difference between COVID’s effect on At-Risk students with an IEP and Not-At-Risk students without an IEP, t(29) = -1.54, p = 0.14, when looking at the difference in F20-F19 and F19-F18 achievement.