The art, science and mystery of nonprofit news assessment

This report seeks to answer the two-pronged question, “What is ‘impact,’ and how can it be measured consistently across nonprofit newsrooms?” A review of recent, relevant literature and our informal conversations with experts in the field reveal growing ambitions toward the goal of developing a common framework for assessing journalism’s impact, yet few definitive conclusions about how exactly to reach that framework. This is especially the case when journalism’s “impact” is defined by its ultimate social outcomes — not merely the familiar metrics of audience reach and website traffic. As with all journalism, the frame defines the story, and audience is all-important. Defining “impact” as a social outcome proves a complicated proposition that generally evolves according to the constituency attempting to define it. Because various stakeholders have their own reasons for wanting to measure the impact of news, understanding those interests is an essential step in crafting measurement tools and interpreting the metrics they produce. Limitations of impact assessment arise from several sources: the assumptions invariably made about the product and its outcome; the divergent and overlapping categories into which nonprofit journalism falls in the digital age; and the intractable problem of attempting to quantify “quality.” These formidable challenges, though, don’t seem to deter people from posing and attempting to find answers to the impact question. Various models for assessing impact are continually being tinkered with, and lessons from similar efforts in other fields offer useful insight for this journalistic endeavor. And past research has pointed to specific needs and suggestions for ways to advance the effort. From all of this collective wisdom, several principles emerge as the cornerstones upon which to build a common framework for impact assessment.