Stats, stories, and policy design

In my last post, I mentioned Illinois’ new testing plan, which sets different testing standards based on student demographics including race and class. The policy oozes the flawed logic that has defined the accountability era: Statistics—and experimental and quasi-experimental studies, in particular—represent the gold standard of educational research. 

Before you either tune me in or out because of the above paragraph, let me make a few points: First, I am not a qualitative zealot. I don’t hate statistics. Research questions determine methodology. The questions in which I am interested just happen to be open-ended and relate to “how” and “why.” 

Second, rigor and scope—not methodology—determine the value of a study. How do we know what we know? And, how does the study inform social issues? In terms of rigor, qualitative researchers have not always provided compelling arguments to policymakers about the utility of their work. While some have attempted to develop standards, others have critiqued the epistemological underpinnings of the whole endeavor. Who wants to invite wet blankets to the policy design party when all they’ll do is philosophize? Policy is about doing, not thinking: Politicians want to know if they should or should not fund a reading program. Yes or no. 

In terms of scope, policymakers have failed. Methods are tools to understand complex social issues. Each tool serves a unique function. Just as no one expects a hammer to saw, no one should expect an ethnography to inform policy in the same way as an experimental study. Policy designs, based on a limited scope of understanding, fail to account for the full bloom of social life. Imagine how we could improve implementation if policymakers combined the insights from a variety of rigorous studies. 

Frustratingly, smart people are discussing the issue. The ever-thoughtful Mike Rose talks about the importance of stories to portray nuance and complexity. Thomas Pikkety, in one of the most hyped books from an academic press in recent memory, and a NY Times bestseller, rallies against pedantic, overblown statistical methods. 

There seems to be an emerging consensus that stats only tell part of a story. And yet, researchers and policymakers motor along.

Social justice and policy design

A few weeks ago, I read about Illinois’ new testing plan. It includes a number of points. The most notable is the state’s decision to use different standards to measure achievement among student groups. By 2019, Illinois expects 85% of white students—compared to 73% of Latino students and 70% of black students—to pass the state reading assessment. In other words, some students will be held to higher standards than others. State officials have presented the system as an improvement to No Child Left Behind. Remember, NCLB required 100% of students to be proficient at state assessments (even though, test scores became a bit of a moving target). 

I don’t know how you interpret the convoluted new plan. I tried to consider all perspectives, but had trouble remaining neutral. In fact, the words “idiotic” and “racist” came to mind. I then attempted to explain the policy as some sort of affirmative action. However, affirmation action tries to reverse discriminatory practices—not create them—in order to provide opportunity.

Of course, Illinois state officials provide perfectly acceptable rationale. They even mention all of the familiar buzzwords. Data. Data. Data. Growth. Growth. Growth. And also, a few nods to poverty and after-school programs. Everyone can rest comfortably, they argue. The state’s low expectations are all backed by science. This braintrust, I’m sure, had nothing to do with the last policy iteration—the one that used to be touted as the next best thing and is now evidence they use to justify the new best thing.

My primary objection with Illinois’ reform—like so many across the country—is the degree to which it is divorced from common sense and the day-to-day lives of students. How does a mom explain why the state has different expectations for her child? What happens at the lunchroom when a group of friends try to figure out their test scores? And, why did the state create a policy in which the major accountability measure affirms current inequities, rather than eliminates them? 

We ought to be able to design policies that provide opportunity for all students.

In my next post, I’ll discuss how research contributes to the problem and also provides a potential solution.