Experimenting by adding links and generating AI summaries for newsletters

Sophie Rentschler is a 2025 RJI Student Innovation Fellow partnered with the Source. The RJI Innovation Student Fellows will be sharing their innovative work throughout the summer through Innovation in Focus.

While the Source’s flagship daily newsletter, the Cascades Reader, has maintained a competitive click rate of 4.3% and open rate of 47.6% — with about 12,000 subscribers — newsletter A/B testing is important to conduct routinely. Mailchimp’s online resources describe the importance of routine A/B tests as a “scientific way to solve marketing mysteries” rather than using anecdotal evidence as a crutch for measuring a newsletter’s success. 

As the Source moves to a WordPress site, its content will automate with Mailchimp to produce this newsletter. Newspack, a publishing platform built on WordPress, will pull headlines and copy from stories posted on the website and synchronize that information with Mailchimp. This automation creates a draft for the newsletter that our editors can revise. As the Source continues to experiment, the results of these A/B tests will guide the staff on where they should be putting their energy.

The tests we conducted on Mailchimp gleaned insight into whether adding links into the newsletter introduction space increases the click-through rate of our campaign and if AI can assist in generating subject lines to achieve a higher open rate. 

Here’s what just about two weeks of testing concluded and how we did it

This newsletter goes through both the sales and editorial team, so we developed an informal process to facilitate collaboration. This process also allowed us to easily jump into Mailchimp and make the A/B test without interfering with other staff’s daily workflows. 

Our workflow for building the newsletter while A/B testing: 

1. The sales team updates the advertisements in the Cascades Reader newsletter.
2. The newsletter then goes to a member of the editorial team to input the content.
3. Our managing editor gives the newsletter another look and sends out a test email to our publisher, sales team and editorial team for any final edits.
4. After final edits, we went into Mailchimp and scheduled it. 

Think strategically about timing

Conducting an A/B test begins by allocating ample time to run the test. Mailchimp suggests not to send an A/B test during odd times of year, for example holidays like Fourth of July. We decided to run each test for at least seven full days to get results

Create a template of the email you want to test

For Mailchimp users, you can start building the test by creating a template of your draft email campaign. Do this by clicking “Edit Design” on the newsletter preview page.

Then click “Save as a template” in the upper right corner.

The latest template created should appear at the top of the email templates section of the “Content” page, which can be navigated to the left. In this case, the template is labelled “Cascades Reader Wednesday July 23 Test”. 

Once we created the template, we set it aside momentarily until we set up the actual test. 

Set up the A/B test in Mailchimp

1. Select “Campaigns” on the main sidebar on the left side.
2. Click “Create” on the top right of the “Campaigns” page.
3. Under “create multivariate test,” click “create test.

4. Select the desired newsletter audience, if applicable. We chose the entire audience for the Source, since a few subdivisions of the publication house their newsletters on this Mailchimp account as well. 

5. Choose the metrics to test (subject line, name, contents or send time). We selected our test variable as “Contents” because we were experimenting with the copy in our newsletter’s introductions. 

6. We selected 100% percent of our email recipients to split our audience directly in half. So, roughly 6,000 people received the variant and roughly 6,000 people got the control email in their inbox. This is what made our newsletter an A/B test rather than a multivariate test. We only tested one variable, whereas the latter describes newsletter tests that test “Subject line”, “From name” and “Contents”, for example. 

7. To edit subject lines, there is a separate display page. A user can insert their variable headline and control headline there. 

8. Next, select “Edit content” on each newsletter preview and the user will be directed to the templates page. Click the template that was saved earlier and modify only one template. For our test, we added links into the second variation. 

9. Finally, schedule the template. We schedule our Cascades Reader for 6 a.m. daily. 

Sifting through test results 

While we assumed that including more links would increase the click-through rate, the results from this test fluctuated and didn’t cause a noticeable change in the newsletter’s click rate. 

Though there wasn’t a measurable impact derived from this test, the results informed us that including links in each individual story’s preview copy on the newsletter is a better use of time and yields more engagement. We determined this because the clickrate didn’t increase as we added links. 

We also conducted A/B tests comparing AI-generated and human-generated subject lines. In the first few days of testing Mailchimp’s generative AI feature in newsletter subject lines, the results have been surprising. In our experience, the use of AI in generating subject lines can be largely helpful in developing different types of summaries and brainstorming ideas. 

Users can modify the generated subject lines based on its tone, length and even creativity. 

We usually inserted the lead story’s headline as the generative AI prompt, which creates captivating subject lines more often than not. This feature also includes options to change the tone and shorten the length of the AI-generated subject line. In two out of the three days of testing, the headline using AI assistance received a higher open rate than the human-generated one. 

Next, we’ll be carefully looking at how open rates change as a result of this second test. The complete results will be wrapped into a formal newsletter audit, including audience demographic information, that will later be presented to the Source editorial team. 

Sign up for the Innovation in Focus Newsletter to get our articles, tips, guides and more in your inbox each month!

---

Cite this article

Rentschler, Sophie (2025, Aug. 19). Newsletter A/B tests you can try — and how to set it up. Reynolds Journalism Institute. Retrieved from: https://rjionline.org/news/newsletter-a-b-tests-you-can-try-and-how-to-set-it-up/