What if STAR included growth for DC high schools?
In December, OSSE released the new DC school report card and first STAR school quality ratings. For elementary and middle schools, the new rating system included academic growth in math and reading for the first time, and growth was worth the plurality of points in the school score. However, for high schools, there was no growth measure included. The primary reason for the lack of HS growth is that the PARCC consortium had yet to finalize and validate a high school growth measure like the one for grades 4-8 by the time OSSE had to make STAR business rule decisions.
However, OSSE did pass along PARCC student growth percentile data for high schools late this summer after the consortium calculated the data for the first time, and the Public Charter School Board allowed schools the option to include their growth data in the latest PMF score release back in October.
This blog quickly examines how STAR scores could change for the public charter high schools in DC if OSSE had been able to utilize the data in time. As per usual, when it comes to this level of data analysis there are a few key caveats our readers should understand:
This analysis only includes data from the public charter schools. DC Public Schools has not released growth data for its high schools. This means we calculated the 10th and 90th percentile using results from just 18 schools.
We weighted high school growth at 20 possible points (10 each for ELA and math.) We did not adjust any of the other possible points for metrics, so the new total max points possible for high schools is 115. Why a growth total of only 20? Well, the elementary school framework has growth at 40 points covering two grades. In high school, we only test once, so we decided to take half of the 40 points as our choice for this blog post.
Our analyses only focus on the scores for the "All Students" subgroup since PCSB only publishes data at the school level and not subgroup level. In STAR, other student groups like special ed and at-risk combine to count for 25 of the 100 possible overall points.
We still have a few outstanding concerns about the business rules around using high school growth data for accountability, including students with atypical course progressions and whether students who were not consecutively enrolled at the school between assessments should count. What do we mean by this? For example, a sophomore who took the required Geometry test at School B but was at a different School A for their freshman Algebra year was still fully counted for growth at School B in the PCSB calculation. Also, students who repeat a course may not have a growth percentile because the PARCC consortium did not have enough data for that course progression to calculate a reliable SGP for the student.
So what does the 2018 charter school growth data look like?
First, to help contextualize the PCSB-published median growth percentile data, it is important to know that the national average among PARCC consortium states for student growth percentiles is 50 with a school-wide median growth between 40 and 60 representing "typical growth." The chart below shows the average ELA and math MGP for public charter high schools in 2017-18 (data pulled from PCSB PMF report cards.)
Most of the schools fall in the "typical" MGP range, meaning their student growth is keeping pace with peers across the PARCC consortium. Four schools have above average growth and four are below average. (Data for BASIS High School and DC International are not available, likely due to the fact the schools administer Integrated Math 2 instead of Geometry and the PARCC consortium does not have enough schools with that course progression to validly calculate growth.)
What is the correlation of high school MGP with at-risk and other STAR metrics for the charter high schools?
The graphic below is a correlation matrix showing correlations between different components of the STAR report card. The high school MGP metric has the second lowest correlation with at-risk among all the report card metrics for high school, and it also had a statistically significantly high correlation (p<0.05) with each school's original STAR score.
PCS high school STAR metric correlation matrix with MGP included
To us, this means there is merit in working through any outstanding issues in utilizing the new high school MGPs because the data is capturing information about school quality besides the demographics served at a school. Also, it is comforting to know that the combination of other current metrics on the report card (attendance growth, re-enrollment, graduation rates) have a correlation with and impact on student academic growth. Logically, this is what we were hoping to find.
Including high school math and ELA growth in the STAR rating would also cut the predictive impact of demographics on charter high school scores by 20.2%. Another positive effect of utilizing growth data at the high school level.
How do the charter high school STAR scores change when PARCC MGP growth data is included?
As a reminder, we were only able to analyze each charter high school's "All Students" subgroup scores. The table shows what each school's All Students subgroup score was on the 2018 Report Card, what it would be if MGP was included at 20 points, and the difference between those two values.
The inclusion of growth would not have changed any school's STAR rating this past year. However, schools like Friendship Tech Prep and Capital City with above average growth gained almost 10 points on their All Students STAR Rating score.
For those who read our recent blog post analyzing additional STAR calculation models, here is what the charter "All Student" subgroup STAR scores would look like utilizing our "EK12 Preferred Model" that gives additional weight for the best of growth or achievement and adjusts growth for summer learning loss.
This chart utilizes an EK12-preferred theoretical model for calculating STAR scores
Our final high school recommendations
We strongly encourage OSSE to work with the PARCC consortium and psychometricians (assessment data science experts) to work through the remaining validity and reliability issues to publish growth data for all DC high schools. There may be other valid growth measures possible besides MGP that have yet to be explored, including one similar to our ES/MS Growth to Proficiency metric and one utilizing Normal Curve Equivalents in the calculation.