Knowledge in a blender: Why we're so mixed up about evidence in public policy

There is a lot of concern about recent trends that appear to be undermining the perceived role and credibility of science and evidence in policy making processes. Many factors contribute to these perspectives and their potential impacts on public decision-making are immense. 

The public is inundated with information every day. There is much talk of the current “post-truth era” with its “fake news” and “alternative facts” that are all too easily distributed to large audiences via the web and social media. You can now consult Dr. Google regarding just about any medical issue or beauty product and obtain all sorts of quasi-scientific advice and information on how this or that product will make your life better. This can easily undermine people’s confidence, given some information may emanate from more credible and reliable scientific research while other advice may be blatantly inaccurate and motivated by nefarious objectives. Alternatively, this uncertainty may lead some to abandon the search for evidence altogether, viewing it as all relative to one’s perspective, without regard for the reliability, trustworthiness and/or biases of the sources in question.

Of course, we must also recognize that within the public sector, evidence may be just one consideration at play. And sometimes, it is a difficult one at that. As the British economist, John Maynard Keynes, stated so eloquently, “there is nothing the government hates more than to be well-informed; for it makes the process of arriving at decisions much more complicated and difficult” (as cited in Davies, 2004). In the face of numerous pressures, including public perception, values, political habits, competing evidence from potentially equally reliable sources, not to mention limited time and resources (Cappe 2011; Davies 2004), decision-makers often find themselves caught between doing what may be right versus what is most appropriate or achievable amidst a multitude of competing issues. As policy experts, Michael Howlett, M. Ramesh and Anthony Perl (2009) have observed, governments often prove “resistant to ‘expert’ advice…In the real world of public policy, technically superior analysis [is] often subordinated to political necessity.”

Faced with the skepticism surrounding so-called informed policy-making, it is all too easy to find ourselves retrenching or “doubling down” on the importance and enduring value of science and evidence (Rayner 2020). But alas, there too, we face uncertainty. As many in the research community are prepared to admit, factors like political influence impact not only what and how certain issues are studied but also what information may be conveyed for public consideration. A study by the Professional Institute of the Public Service of Canada (2013) found that 86% of federal scientists, when faced with “a departmental decision or action that could harm public health, safety or the environment,” did not believe that “they could share their concerns with the public or media without censure or retaliation from their department.” Moreover, 50% reported “being aware of actual cases in which the health and safety of Canadians or environmental sustainability has been compromised because of political interference with their scientific work.” 

Meanwhile the so-called “irreproducibility crisis,” highlighted through a recent study covered in Nature has shone light on the tenable nature of scientific findings (Baker 2016). While 70% of researchers surveyed indicated that “they have tried and failed to reproduce another scientist’s experiments,” more than 50% have “failed to reproduce their own.” Public awareness of this troubling issue was heightened earlier this year when the “totally bummed” Nobel Laureate Frances Arnold had to retract a scientific publication because the “work has not been reproducible” (O’Brien 2020).

As the above examples illustrate, even seemingly sound science from presumably reliable sources has admitted fault lines. And as others have pointed out (Rayner 2020), even where we find some consensus on the quality and viability of certain findings, there is evidence (there’s that word again…) that it may do little to persuade individuals who have already formed opinions based on other factors (Kahan et al. 2012). 

So, where does this leave those concerned about evidence in the world of policymaking? Are we resigned to despair and should we select another area for our own research? 

At the end of the day, I am heartened that there is a wealth of existing and emerging literature addressing what some describe as decision-making or judgment amidst uncertainty. (Amos Tversky and Daniel Kahneman’s 1974 classic work provides just one well established model to consider.) And while (as my limited title suggests) it’s beyond the scope of this particular post to do any justice to reviewing that body of scholarship and I do apologize for taking readers through a diagnosis without full consideration of a remedy, I am happy to share a couple tentative perspectives from my own vantage point on what might form part of our considerations as we move forward.

First and taking into account what has been revealed about the tenuous nature of science and by consequence, I believe, so-called facts, I still support taking into account the science and evidence put forward by those researchers whose investigative methods and communication of findings we trust. It doesn’t mean the findings will always be accurate nor perfect (whatever those terms mean in this context…). But it does mean we can hopefully expect to have publically accessible dialogue and debate not only about findings, but also with regards to the questions asked, the discourse at play and the methods utilized to arrive at certain conclusions.

Secondly, I believe we have to broaden our research and avoid excluding particular voices from within and beyond the academy. For decades, we’ve realized that what we may perceive as “reality” is always constructed and mitigated in particular social and cultural contexts (Berger and Luckmann 1967). The currently in-vogue expression of “wicked problems” to describe large-scale global challenges like climate change and food security is nearing its 50th year in parlance and reminds us that the unwieldiness of policy dilemmas is hardly new. This perpetual uncertainty about what is really going on and what really needs to happen has never permitted complacency. It should embolden those of us with the time and resources for research to ask ourselves tough questions and to listen to others whose perspectives might challenge our own. We need a full roster of perspectives that is only achievable through convergent and collaborative research. 

I can’t think of a better model for research teamwork than one that would incorporate the input of scientists, social scientists and humanists who work together to deepen comprehension and make holistic and usable information available to public decision-makers. This would be particularly true in the case of crises like COVID-19 and climate change with their multiple scientific, social and cultural dimensions. It is abundantly clear to all of us now that these challenges impact every aspect of our lives. A corresponding breadth of expertise would be our best strategy for finding appropriate and workable policy solutions that are themselves adaptable as we continue to deepen our own understanding and knowledge.

Download the blog post.

Baker, Monya. 2016. “1500 scientists lift the lid on reproducibility.” Nature. May 25. Available at: https://www.nature.com/news/1-500-scientists-lift-the-lid-onreproducibility-1.19970

Berger, Peter and Thomas Luckmann. 1967. The Social Construction of Reality: A Treatise in the Sociology of Science. Anchor Press. 

Cappe, M. 2011. “Analysis and Evidence for Good Public Policy: The Demand and Supply Equation.” Tansley Lecture. Available at: https://ourspace.uregina.ca/handle/10294/7567

Davies, Philip. 2004 “Is Evidence-Based Government Possible?” The Jerry Lee Lecture. Available at: https://webarchive.nationalarchives.gov.uk/20091013084422/http:/www.nationalschool.gov.uk/policyhub/downloads/JerryLeeLecture1202041.pdf 

Howlett, Michael, M. Ramesh and Anthony Perl. 2009. Studying Public Policy: Policy Cycles and Policy Subsystems, 3rd Edition. Don Mills, Ontario: Oxford University Press.

Kahan, D., Peters, E., Wittlin, M. et al. 2012. “The Polarizing Impact of Science Literacy and Numeracy on Perceived Climate Change Risks.” Nature Clim Change 2, 732–735. Available at: https://doi.org/10.1038/nclimate1547

O’Brien, Cillian. 2020. ‘Totally bummed’ Nobel Prize winner admits a ‘painful’ mistake.” January 3. Available at https://www.ctvnews.ca/sci-tech/totally-bummed-nobel-prizewinner-admits-a-painful-mistake-1.4751683 

Professional Institute of the Public Service of Canada. 2013. The Big Chill: Silencing Public Interest Science, a Survey. Available at: https://www.pipsc.ca/portal/page/portal/website/issues/science/bigchill    

Rayner, Jeremy. CSIP Research Forum presentation. Policy advice in the post-truth era: What role now for evidence-informed policy analysis? November 6, 2019. 

Tversky, Amos and Daniel Kahneman. 1974. “Judgment Under Uncertainty: Heuristics and Biases.” Science. September 27. Available at https://science.sciencemag.org/content/185/4157/1124