Chalkbeat and RJI heard from 58 U.S. journalists involved in source tracking

Two years ago, Chalkbeat joined a growing list of newsrooms attempting to track the diversity of our sources so we could better understand whose perspectives we’re elevating in our journalism.

We started with a small swipe — two recurring series — and quickly realized we had work to do. We also learned that many, many newsrooms or individual journalists are doing this auditing work alone in silos. We wanted to learn from one another and to build a better system.

Over the last few months, Chalkbeat and Reynolds Journalism Institute student staffers Tatyana Monnay and Sara Dingmann collected source tracking information from 58 journalists and interviewed 14 of them.

The journalists interviewed represent newsrooms ranging from small startups to legacy radio stations. Some have audited sources for years, some were only a few months into their process, and some were running periodic spot checks every few months.

The majority of interviewees were from public media, which has dominated the field so far in building source audits. All newsrooms were evaluating the representation of the people they were interviewing and comparing those findings with their intended audience and communities.

The collected information will guide Chalkbeat and RJI in building a source diversity tracker that is efficient, sustainable, and replicable. Want to get involved or recieve updates on this project? Let us know here. You can also join our ONA discussion this week here. 

What we learned

Lesson 1: Newsrooms need clear direction at the beginning on goals for source tracking, data collection, and how to use the results

An effective source audit system should fit into a newsroom’s broader plan to diversify its staff and coverage to better reflect the  communities it serves. Many newsrooms launched source diversity audits in the past year (10 of the 14 newsrooms  surveyed) — and many audits are part of larger newsroom efforts in diversity, equity, inclusion, and belonging, or DEIB.

KUER in Utah, for example, launched its audit this year as part of a larger newsroom culture project. And WHYY in Philadelphia has paired staff training on cultural competency with expanding its source audits over three years.

But many audits lacked that kind of clear direction or tie to broader organizational initiatives, and subsequently, felt like one-off projects that didn’t have a meaningful impact on the newsroom  Source tracking work can be tedious and only matters if it helps us understand our own patterns and blind spots as journalists and helps hold ourselves accountable to more inclusive coverage.

Only a few of the newsrooms interviewed had been tracking sources for more than a year — and most were still struggling with how to benchmark their data — for instance, should newsrooms use Census data to assess findings? What’s the right benchmark?

Many newsrooms also are still developing systems for how to use the results internally and publicly:  When do you review data with staff? Do you set goals for individual reporters and hold them to it? Will you share your results publicly?   

Key takeaways: Articulating the why and how of any source audit project is critical to its success. Reporters should have input into designing audits or tracking systems and in analyzing the results. Newsrooms that had these conversations with reporters early in the process seemed to get more staff buy-in and support.

At the same time, newsroom leaders should develop a clear plan for how to incorporate findings into the culture and systems (for example, will results play into reporters’ goal-setting? Who is responsible for driving results? How and when will newsrooms post publicly to inform their audiences?)

Lesson 2: Surveys should be short and auditing should be continuous

These interviews confirmed that the shorter the survey, the better. Many newsrooms, including those working with America Amplified, started with long surveys then cut down to five to eight key multiple-choice questions. Retroactive audits took more time than did ongoing tracking, and usually provided less reliable data.

Some newsrooms enlisted interns or college journalism programs to design the audit. KQED worked with Impact Architects to establish a baseline, and The Philadelphia Inquirer worked with Temple University to analyze 3,000 pieces of journalism. While enlisting outside or temporary help can make retroactive audits less overwhelming, it can be difficult for newsrooms to continue work started by an outside entity after the initial audit.

Spot checks — or tracking sources for random periods — drew worse participation from reporters and spottier data. Because the checks weren’t baked into the workflow and regularly enforced, reporters would forget. And one news cycle could skew the data.

Key takeaway: A brief  survey that tracks name, age, gender identity, race/ethnicity, source title or category, ZIP Code, and an option to opt into a source database provides enough data. (Note: Many newsrooms struggled with sources’ titles, which should be defined before launching a survey. For examples, check out the appendix of this study by the Institute for Nonprofit News). 

Key takeaway: Continuously track and audit as part of the reporting process. If newsrooms believe source diversity is important, they should strive to get the best data. 

Lesson 3: Automation saves time (but doesn’t mean reporters are less invested)

There was a clear divide among interviewees of who should be in charge of this work and to what extent newsrooms should invest in automating their diversity tracking systems.

The vast majority of newsrooms interviewed used Google Forms to record survey data. Resolve Philly used Airtable. Most, like Colorado Public Radio, require reporters to collect sources’ demographic information during interviews then record it into the form.

Some interviewees felt strongly that reporters should ask demographic questions and fill out the survey. Rather than just providing email addresses, journalists then can take more ownership in diversifying their sources.

All the newsrooms relying on journalists to fill out Google Forms or separately query sources acknowledged that neither could last long term. Both take time, can be forgotten, and raise data security questions.

Wisconsin Public Radio’s demographic project embeds source data into its content management system. WPR said that doesn’t strain reporters and makes it harder for them to forget. Still, WPR staff spend time every month figuring out which sources have and have not been quoted before. The data comes from emailed surveys that about half of sources fill out.

Both manual and more automated processes raised security questions. Some newsrooms are moving away from Google Forms because they don’t trust Google to keep sensitive information safe. Others reported concerns about adding sources’ personal information into their CMS — the system that publishes stories — could that information accidently be published online. 

Key takeaway:  Newsrooms that rely on journalists to collect the data reported more issues with buy-in and with data collection — often reporters would forget or put it off. This speaks to the heavy load of reporters and why automation can help. 

Key takeaway: WPR has come the closest to building an automated system that’s far more effective than manually updating Google Forms. But it isn’t seamless. It could be further automated to lighten the load on staff. Newsrooms also need to better assess security risks of collecting and storing personal data. 

Lesson 4: Newsrooms should design with a living source database in mind

Very few newsrooms initially developed a source audit with the idea of also building a source database for future stories. Thanks to WPR’s influence, that’s becoming more of a trend. WPR uses Google Sheets for its newsroom Rolodex.

One WPR audit manager oversees the database and helps reporters find sources. The database prioritizes sources that identify in an area underrepresented in WPR’s coverage, but doesn’t include sources’ demographic information to protect privacy.

News organizations designing a lasting source audit might consider how to create a source list that elevates people historically underrepresented in that newsroom. In the same way audits help newsrooms understand who they are amplifying, shared databases will make it easier for journalists to expand beyond their usual sources.

Key takeaway: Newsrooms should consider investing in a source database for the long haul. Having one person or one team manage the source database could make it easier for journalists to use this resource, but editors and reporters would also need training and access to ensure the database is used well. 

Want to read more?

Here are helpful field guides from across the industry, and some surveys you can copy: Wisconsin Public Radio, America Amplified, WHYY , KQED, INN — case study

What’s next?

Chalkbeat and RJI are taking all of these learnings into our next phase: Building a long-term system that will work for Chalkbeat’s newsrooms, and hopefully, for any other newsroom interested in this work. Questions? Suggestions? Want to get involved or say hello? Reach us at community@chalkbeat.org and duncank@rjionline.org