Stop bashing methods. Help create a better world. #BMJnoQual

Originally posted at www.21stcenturyscholar.org

1.

Last year, the British Medical Journal rejected an article. Such an action does not ordinarily generate attention; editors reject articles every day. The author, however, tweeted the rejection: “Thank you for sending us your paper. We read it with interest but I am sorry to say that qualitative studies are an extremely low priority for the BMJ. Our research shows that they are not as widely accessed, downloaded or cited as other research.” The tweet produced a lively strand of responses, responses like “shocking and shameful,” “epistemological oppression,” and, “I guess nothing qualitative ever happens in a clinical setting.”

As scholars, we learn where to submit (and where not to submit) our work. I conduct qualitative research with policy implications. I know, however, if I submit an ethnographic article to Educational Evaluation and Policy Analysis, the editor is likely to swiftly drop-kick it back to me. Similarly, I know my quantitative-oriented friends will not be submitting to Qualitative Inquiry any time soon.

Without getting into nuances like impact factors, tenure decisions, and research standards, most qualitative researchers understand the landscape. To be honest, when I read the BMJ rejection note, I thought it was quite civil. I have received a few rejections that would make that rejection blush. I once submitted a life history on a late Friday evening. Within an hour, I received a rambling three-paragraph response from the editor, perhaps assisted by a nightcap or two, stating that anything with an N of one is neither research nor policy-relevant. Okie dokie. Thanks for the quality feedback. On to the next journal.

2.

Last week, Stephen Porter, a professor of higher education at North Carolina State, published a strongly-worded blog denouncing the #BMJnoQual incident, in particular, and qualitative research, in general.

I do not know Porter. He is a senior scholar with a fine record and numerous accomplishments. He is far more accomplished than I—so, take everything with a grain of salt. I am sure he is a reasoned, thoughtful person. However, after even a generous interpretation, the blog demonstrates a provincial understanding of qualitative research and a paternalistic and mean-spirited tone towards qualitative scholars.

Consider a few points:

(1) The title of the blog—“Speaking Truth to Power about Qualitative Research”—is ironic. Whether intentional or not, the blogger alludes to Aaron Wildavsky’s classic Speaking Truth to Power: The Art and Craft of Policy Analysis. Wildavsky, a hugely influential policy scholar, argued that policy analysts need to account for the interpretive nature of policy-making—something that qualitative work is particularly well-suited to accomplish.

(2) Porter argues that qualitative research has little impact and will have less. I agree that qualitative researchers need to redouble their efforts to collaborate with scholars and practitioners across disciplines and methodologies in an effort to produce and advocate for rigorous policy-relevant research. NSF isn’t funding many six- and seven-figure qual studies. But, many examples exist—as models for early career faculty to follow—of scholars who have achieved wide impact with well-designed studies that include qual methods. I’m fortunate to have two of them as mentors: Bill and Yvonna. Even more, look at the list of past AERA presidents over the last few decades. That’s an awful lot of impactful and innovative “dinosaurs,” a term Porter uses to describe qualitative researchers.

The blogger also introduces the technology argument: tech will enable scholars to create, gather, and analyze larger and larger datasets. The argument works both ways. Technology and social media will provide new opportunities for qualitative researchers to collect, synthesize, analyze, present, and share data. Big data will magnify, not lessen, the need for interpretive and site-based inquiry. As we’ve learned from previous examples, policies based on one-sided approaches are often ineffective. Some even reinscribe the same inequities they seek to remedy.

(3) The idea that qualitative research does not appear in well-read publications is fiction. I have read numerous articles and blogs at the NY Times and Washington Post. I am a sociologist of education who examines neighborhood-level issues. For a recent example of impact, search Google for MacArthur Fellow Matthew Desmond’s newest book Evicted, based on ethnographic research. And, while federally-funded qualitative studies in education are rare, there are numerous foundations who do fund multiple methodologies. I’m thinking of the Russell Sage Foundation, which has funded, published, and promoted significant projects that have reached beyond academia and into public and policy discourses.

(4) Porter presents a straw-man argument about generalization and qualitative research. Of course, qualitative research can’t (and shouldn’t) generalize—while a conversation for another time, most quantitative work shouldn’t either. And yet, from rigorous, well-designed qualitative studies, scholars can and have provided actionable findings and policy implications. At school and community levels, researchers and participants creatively and meaningfully employ strategies like action research and participatory action research to improve practice, research, and policy.

(5) The blogger writes, “Some qual researchers insist there are multiple realities. What do you think the average person, who lives in a single reality like most of us, thinks of this idea?” The answer is—while probably not using the pedantic words of academics—they would agree. As accomplished scholars like Gloria Ladson-Billings, Rich Milner, Luis Moll, Michelle Fine and numerous others have demonstrated, education research and policies often propagate white, privileged perspectives. Deficit-based policies represent dominant beliefs and assumptions about whose knowledge matters and how it is defined and measured. The “average” person who has experienced persistent poverty and endured racist policies would probably argue that their voices are not heard and their experiences are not represented in policy discussions and designs.

Critical (and necessary) exchanges about epistemologies and ontologies and axiologies and other fancy words and concepts are easy targets. From a policy perspective, overly theoretical arguments unmoored from practice—Latour refers to these stances as “fairy positions”—become counterproductive. I agree that navel-gazing rarely influences policy design. But, skilled academics have the ability to connect theory and practice, something the aforementioned scholars (who have employed qualitative methods) have done to great and consequential success.

I could go on, but I won’t. Again, I do not know Porter. I’m sure he has reasons for his fervent and seemingly unyielding opinions about qualitative research. He certainly has years of experiences to inform his perspective. But, a narrow approach to research—and, by extension, knowledge, beliefs, values, assumptions, etc.—is misguided, at best, and harmful, at worst. When has a one-size-fits-all approach to education ever worked, particularly for underrepresented populations? Social inquiry and policy design require a plurality of approaches. Each has strengths and limitations. We have a large toolbox of methods to examine complex, intractable issues. Why would we limit ourselves to just one?

So you want to be a qualitative researcher in the 21st century

Originally posted at www.21stcenturyscholar.org

A tension exists between old and new. In The Anxiety of Influence, Harold Bloom explains the generational process among writers: Old poets inspire young poets. The apprentice learns to love form by reading the work of a skilled master. The beginner writes derivative verse. Anxiety stirs as she realizes the only way to establish a legacy is to break from tradition. And, that’s the rub.

The charm of Bloom’s theory is that it extends to numerous fields. Consider the myriad movies in which young protagonists ignore the sage advice of their battled-scarred mentors. Characters fail, fail, and fail again. And then, after sweaty and bruised adversity, they triumph. Hello, Karate Kid. Or, think about athletes. Young basketball phenoms like LeBron battle the legacies of legends like Jordan and Magic. Musicians provide yet another example—thankfully, Bird inspired Coltrane. The theory extends to more quotidian examples too. Children clash with parents. Students argue with teachers. The young fight for a trophy, the ability to say, “I did things my own way, a better way.” The trophy, of course, proves elusive.

As qualitative research enters an exciting moment, apprentice and master researchers are reenacting similar clashes in classrooms and research labs across the globe. “The methods are quaint,” the initiate says, “but I think they’re a little dusty. I can do better.” The mentor winces: How many times has she heard similar boasts?

Innovative technologies and digital media are providing new tools and venues. Consider the possibilities of research-based digital media. They can reveal complex processes that contribute to elusive opportunities for low-income students in ways that peer-reviewed articles cannot. Policymakers often grimace at pedantic and esoteric research. A digital short provides fertile ground for conveying the sorts of thick description qualitative researchers seek and also improving the relevance of research for policy stakeholders.

Novel methods are alluring, an opportunity for novice researchers to shape their legacies. But, like the young poet who privately spends thousands of hours mastering rhyme and rhythm or the basketball phenom who quietly practices drills in the gym, the innovative researcher is the product of hours and hours of unheralded work: planning, collecting, analyzing, producing, experimenting, revising, and repeating.

Rigorous designs depend on the ability of a scholar to undergird the process and product with traditional methods, all the while embracing emerging opportunities. A two-minute film excites. It also requires a complex set of skills. The researcher has to be well versed in fundamentals like interviewing and analyzing as well unconventional techniques like filming and editing. She has to understand triangulation and color theory, parallelism and the rule of thirds, NVivo and Final Cut. The challenge is formidable. But, the chance to experience that inventive moment, the next adjacent possibility, is worth the work.

How do scholars produce policy relevant research?

As I mentioned in my last post, I was named as an Emerging Education Policy Scholar (EEPS). The program, a collaboration between The Thomas B. Fordham Institute and the American Enterprise Institute, provides the opportunity for young, policy-minded scholars to collaborate with and learn from a range of key policy players.

Our first meeting took place a few weeks ago. The long list of impressive participants included advisors on Capital Hill and in the White House, representatives from large foundations, researchers working in think tanks and the government, and reporters from a variety of publications. 

The conversations were always candid, often refreshing, and sometimes daunting. We discussed the relationship between research and policy. Yes, research influences policy. However, the path from research question to legislation is often circuitous and involves a mixture of concerted effort and unpredictable chance.

The meeting was enlightening and rewarding. Here are a few themes—many of which complement the emerging criteria Bill and I present in our article “Qualitative Research and Public Policy: The Challenges of Relevance and Trustworthiness”:

Relevance

How do scholars engage with the most pressing policy issues? Policy windows open and close quickly. During those times, politicians use relevant research to either shape or support policy decisions. Importantly, individuals during the meeting frequently cited the need for research to both address contemporary issues and also say something specific about policy implications. For instance, while a study may find that more low-income students need to obtain postsecondary degrees, such a finding will likely not guide legislation. Policy makers want to know about the effectiveness of specific interventions; they worry about what to do and how to fund it. There was a heavy premium on experimental and quasi-experimental studies. I will discuss the role of qualitative research in another blog.

Audience

How do scholars appeal to multiple audiences? Several participants stated the need for scholars to produce dual publications. If you publish in an academic journal, translate the findings for a blog at Huffington Post or Edweek. Why? Policymakers receive stacks and stacks of research. They rarely read studies, but they do read blogs.

Visibility

How visible are scholars among multiple channels? Especially due to social media, academics have the opportunity to engage multiple audiences. The tenure process still relies on publications. However, when thinking about a scholar’s ability to influence policy, publicity (or “klout points,” using Rick Hess’s term) matters.

Relationships

With whom do scholars engage? Academics who influence policy often have connections with key change agents. Get to know policymakers. Talk to reporters. Engage funders. Informal networks provide predictable and unpredictable opportunities.

These four interrelated themes—relevance, audience, visibility, and relationships—often undergirded our discussions. We will reconvene in June; I’ll keep you updated.

Sitting at some new tables in the academic cafeteria

As Murray Milner documents in Freaks, Geeks, and Cool Kids, teenagers often order themselves using status symbols. Take a stroll around a high school cafeteria. With few exceptions, students clump neatly into groups: cool kids, jocks, band geeks, drama nerds, emos (not to be confused with emus), and any number of other categories. Each group has unique norms, values, and symbols. While monumental decisions loom—like what to do for the rest of their lives—teenagers spend inordinate amounts of time wondering about very different questions, like what to wear, where to sit, and with whom to talk.

Fast-forward to academic life. How different are scholars? What do we wear? Where do we sit? And, with whom do we talk? Cliques and status symbols—albeit drastically different—are very much part of the daily experiences of graduate students and faculty members. However, instead of figuring out what to do on Friday night, we talk about epistemologies.

A few weeks ago, I was named as an Emerging Education Policy Scholar (EEPS). The program, a collaboration between The Thomas B. Fordham Institute and the American Enterprise Institute, two conservative organizations, provides the opportunity for young, policy-minded scholars to collaborate with and learn from a range of key policy players. I know colleagues who have participated in previous cohorts, and I know some of my cohort-mates. I’m thrilled to have the opportunity. 

As an early career faculty member, I didn’t think too much about how others would react. Why would I? I have the opportunity to talk to really smart people about critical issues. Even more, while no one would mistake Bill as a neocon, as an advisor, he has always encouraged me to communicate with and learn from individuals from all perspectives, even if they belong to a different political party or tax bracket. That’s part of being a reasoned, thoughtful scholar, right?

Reactions have been interesting. One university colleague congratulated me and also provided a few tips for “talking to the other side.” Another stood speechless. I saw the wheels turning in her head: Is Randy a liberal or conservative? To diffuse, I made a joke about brushing up on supply-side economics. Both people were sincerely congratulatory and well meaning. But, the interactions revealed the ways in which politics—among other factors—act as a sorting mechanism among academics.

Most will agree that improving education is important. And yet, our actions do not always demonstrate the same joint commitment. I skim vitriolic, all-or-nothing arguments from both liberal and conservative scholars on Twitter. I then read supposedly objective and rigorous research and wonder how to separate reality from ideology, research from propaganda. A methods section—if there even is one—can only do so much; social media is turning out to be just as telling for the trustworthiness of a researcher. 

Some seem most concerned about being right. “We know what works,” they argue and then provide a checklist of reforms in which the research is decidedly mixed. Likewise, intransigent liberals attack and vilify corporate reformers and then, in the same breath, argue for deliberative democracy and social justice. If people scream and yell long enough, others—even if they share similar beliefs—start to wonder about things like credibility, ulterior motives, and a true desire to have a cooperative and mutually beneficial discussion.

I’m going to sit at some new tables in the academic cafeteria. For the few worried liberals or qualitative researchers out there, I still support universal pre-k programs, neighborhood schools, and more inclusive standards for government-funded research. But, we place too much emphasis on who’s sitting where. I’m interested in discussing innovative solutions and improving education. Our problems are far too complex to be solved by anchoring to a set of ideals and then yelling louder than the other person.

Rush to relevance: Conducting research to improve policy and practice

“We need research to be more relevant” is a common clarion call in education. Most recently, John Easton, Director of IES, released a video for AERA in which he talks about different initiatives to improve relevancy.

During one of my first Ph.D. courses, Bill asked us about the three major responsibilities of academics: research, teaching, and service. In particular, he wondered about teaching, something that is too often overlooked at prestigious universities. As someone who just left the high school classroom, I boldly proclaimed—in a way only a first-year Ph.D. can—that all professors should be required to teach, and to teach well. Students pay a lot of money and deserve more than someone who views teaching as a chore to be completed before doing important work. As Bill pointed out then, and I have thought about since, academics have different skill sets. Some are unbelievably talented teachers, others researchers. Let’s propose the best possible scenario: Young scholars receive effective mentorship and professional development and work at universities that support and reward teaching and learning. Even then, some will never be effective teachers. What’s my point? Bold proclamations often fail to account for important nuances.

Just as I wanted all celebrity researchers to be all-star teachers, I think, at a certain level, all research should be relevant. But, the rush to relevancy has a few troubling side effects. First, it ignores the value of various forms of research. A new set of guidelines illustrates the types of methodologies—namely experimental and quasi-experimental—that IES favors (and funds) to produce relevant research. Large-scale, experimental studies are important. If I want to understand how growing up in a low-income neighborhood effects social mobility, neighborhood effects studies provide compelling, significant findings. What else do they provide? A lot of unanswered questions. The Moving to Opportunity (MTO) study, funded by HUD in the 1990s, was an innovative randomized experiment. Researchers sought to understand what happened to low-income families when they received housing vouchers. Scholars are still arguing about the study’s effects on variables like educational achievement, employment, and health. Even the best studies fail to capture the complex and dynamic social processes at work in low-income neighborhoods. While a life history of one or an ethnography of sixty may not generalize, they can provide valuable findings. Understanding social issues requires sophisticated, complimentary methodologies.

Second, the rush to relevance may diminish academic freedom. Foundations and think tanks like and support trendy issues. Schools of education like grant funding. Inevitably, these factors lead to subtle (or not-so-subtle) nudges to Ph.D. students and early career faculty: “Yes, you are free to study whatever you want, but we hope it will be relevant and also generate funding.” Considering fewer and fewer tenure-track positions, the pressures are magnified. Some scholars maintain balanced research portfolios, examining both mainstream and non-mainstream topics. But, even in those circumstances, we diminish the vibrancy of national debates by settling into familiar paths. 

Third, and related to the above point, relevance relies on ever-changing interests. Last year, we talked about MOOCs. This year, we are talking about Common Core. Too often, scholars sacrifice their own interests to pursue those of others. I want to be clear here: I am not arguing against forward-thinking scholars or the study of emerging, innovative issues. Rather, policy windows open and close quickly. Thinking of the most pressing, deeply rooted issues in education, neither scholarship nor meaningful reform benefit from that same hectic agenda. Instead of racing to relevance and the allure of fame and funding, researchers ought to use their own experiences and expertise to help define what is and is not important.