Stats, stories, and policy design

In my last post, I mentioned Illinois’ new testing plan, which sets different testing standards based on student demographics including race and class. The policy oozes the flawed logic that has defined the accountability era: Statistics—and experimental and quasi-experimental studies, in particular—represent the gold standard of educational research. 

Before you either tune me in or out because of the above paragraph, let me make a few points: First, I am not a qualitative zealot. I don’t hate statistics. Research questions determine methodology. The questions in which I am interested just happen to be open-ended and relate to “how” and “why.” 

Second, rigor and scope—not methodology—determine the value of a study. How do we know what we know? And, how does the study inform social issues? In terms of rigor, qualitative researchers have not always provided compelling arguments to policymakers about the utility of their work. While some have attempted to develop standards, others have critiqued the epistemological underpinnings of the whole endeavor. Who wants to invite wet blankets to the policy design party when all they’ll do is philosophize? Policy is about doing, not thinking: Politicians want to know if they should or should not fund a reading program. Yes or no. 

In terms of scope, policymakers have failed. Methods are tools to understand complex social issues. Each tool serves a unique function. Just as no one expects a hammer to saw, no one should expect an ethnography to inform policy in the same way as an experimental study. Policy designs, based on a limited scope of understanding, fail to account for the full bloom of social life. Imagine how we could improve implementation if policymakers combined the insights from a variety of rigorous studies. 

Frustratingly, smart people are discussing the issue. The ever-thoughtful Mike Rose talks about the importance of stories to portray nuance and complexity. Thomas Pikkety, in one of the most hyped books from an academic press in recent memory, and a NY Times bestseller, rallies against pedantic, overblown statistical methods. 

There seems to be an emerging consensus that stats only tell part of a story. And yet, researchers and policymakers motor along.