Humans appear to have the upper hand. For now.

There’s been a surge of interest among news publishers around offering audiences quizzes to boost engagement on their content.

Some of the more prominent examples mainly focus on helping readers digest current events in a fun, gamified way. But quizzes have the potential to help audience members not only understand information but retain more from what they read. This makes them more efficient news consumers as they maximize the value of their time. For news publishers, this translates to a shift away from the fickle engagement of skimmers gorging on news. Research even shows shallow mass consumption is a source of agitation and stress for readers.

Passing a comprehension quiz suggests a deeper understanding of news content, so publishers have some assurance that their content was thoughtfully read and absorbed.

Hallmarks of a good quiz

Note: We’re intentional about not naming news as text or articles because quizzes can be built around other content e.g. audio or video stories. However, we do refer to news quiz takers as readers for continuity.

When it comes to producing high-quality comprehension quizzes for news content, there are a number of considerations that the editor should keep in mind. Here’s what we learned from experimenting with our own quiz. 

1. Accuracy

For each question, all wrong answer options should be marked as “incorrect” and the right answer as “correct.”

2. Self-contained

Only use information that can be found within the news item or that the reader could logically infer from it. No further reading or expertise on the topic should be required to pass a quiz.

3. Coverage

A quiz should have questions covering every part of the content, from beginning to end. This increases the likelihood that passing a quiz signifies the thoughtful consumption of the news content in its entirety.

4. Testing for holistic understanding

The most effective quizzes test the reader for a cohesive understanding of the content instead of merely testing recall of individual details. Thorough understanding means not just remembering details in isolation, but also how they relate to each other.

💡Tip: Use details from distinct but connected parts of the text when formulating the questions and options.

5. Plausible wrong answers

Quizzes should be written in a way that quiz takers must carefully and thoughtfully read the content to pass the quiz. It’s best practice to write answer options so that they sound equally plausible to anyone who hasn’t actually read the content, yet fairly easy to distinguish between for someone who has.

💡Tip: Rely on nuances. Formulate answer options with subtle (yet substantive) differences between them.

6. Answers not easily searchable

Coming up with the correct answer should take more than a quick text search (good ol’ Command + F) or even a Google search. Avoid pulling idiosyncratic words, specific numbers and isolated details from the content that is easy for bots to exploit. Try to use synonyms for rare words, and instead of numbers, ask for simple calculations.

7. Make it fun!

Avoid producing quizzes that are bogged down in facts or nuances. A quiz shouldn’t feel like a chore. Accessible language and some levity will add fun that keeps quiz takers engaged.

💡Tip: Infuse quizzes with your brand’s personality and quirkiness. Use visuals to make them feel snappy.

While writing quality quizzes that follow these guidelines undoubtedly demands some additional work for news publishers, it could be integrated into the production process. In the case of larger, better-resourced newsrooms, dedicated quiz editors could oversee the process.

Publishers might also consider using one of the many AI quiz-generators as an alternative (or complement) to manually edited quizzes. However, this comes with its own challenges.

AI presents opportunities and challenges

While AI makes it much easier for publishers to auto-generate and deploy quizzes, it also makes it easier for audiences to cheat on them by using AI tools to answer questions.

Opportunity: Auto-generating quizzes

As much as quiz-generating AI tools enable publishers to produce quizzes with ease, today’s Large Language Models (LLMs) have a fundamental limitation in that they have no real understanding of story content (e.g. nuance, context, etc.) in the way humans do.

There are many tools available to newsrooms for generating quizzes on their content.

Challenge: Making them AI-resistant

The goal is for the AI solver (e.g. an AI chatbot trying to solve the quiz) to have a much harder time passing the quiz (compared to a human who has read the content thoroughly). This is to say that quizzes should be AI-resistant.

The measure of AI resistance is the rate of questions the AI solver (e.g. ChatGPT) gets wrong. A high level of AI resistance helps protect quizzes from bad actors like bots. Newsrooms should always aspire to produce quizzes that are AI resistant without making them unduly hard for readers to complete. This balancing act necessitates some involvement from a human quiz editor.

Better at writing quizzes: Humans or AI?

AfroLA’s summer UI/UX design intern Wajiha Moin manually produced a 10-question multiple choice comprehension quiz based on an AfroLA article, “Opioid addiction recovery providers favor individual-centered treatment options for Black women.” We gave the same article to three popular AI quiz generators. Then, we asked ChatGPT-4o to solve the four quizzes to test their AI resistance. (See detailed results of our experiment in our github here.)

We gave ChatGPT-4o (te AI solver) our test article and asked it to solve 10-question multiple choice quizzes generated by quiz apps QuizGecko, Arlinear, and QuestGen.ai, as well as our human-made one.

Because of the stochastic nature of LLMs, we needed to eliminate the role of chance. Therefore, for each quiz, we ran 40 rounds of tests with ChatGPT-4o trying to solve it and took the average number of incorrectly answered questions.

Since in all of our quizzes, each question has four answer options with only one of them being correct, the blind guess strategy (random selection of an answer, like on a standardized test), on average, yields a 75% error rate or AI resistance. This rate means that the quiz is so hard for an AI solver that it would have gotten the same result by blindly guessing the answers.

“None of these”

In the second part of our experiment, we changed each question’s correct answer option to “None of these answers.” In all but one case, this improved AI resistance. This strategy may have been successful because LLMs are inherently biased toward finding a positive answer.

To clarify, a quiz editor shouldn’t just mindlessly swap out all the correct answers using this technique as it would make the correct answers predictable to audience members (humans). Instead, a quiz editor should use the “None of these answers” approach strategy sparingly as distractors, and employ additional strategies to make quizzes more resistant to AI.

Improving AI resistance

Our small experiment shows how human editing on quizzes can improve their AI resistance. There are likely many other “smart editing” strategies that editors could employ to strengthen their quizzes against AI solvers.

In addition to semantic methods, there are other ways to improve AI resistance and counteract bad actors like AI-enabled bots, including:

• Rasterizing (turning chunks of text into images) questions and answers, though this does present accessibility challenges (e.g. ensuring alt text properly conveys the content for a screen reader user)

• Deploying CAPTCHAs and other humanity filters upon quiz submission

• Inlaying “meta-games” into quizzes for an added layer of defense like requiring the reader to solve a small word game to reveal parts of the question

The big caveat to all of this, of course, is that LLMs (and AI in general) are a fast-moving landscape. So all of these results are subject to change, and so is the success of any AI resistance strategy.

This is a taste of how current LLMs are unable to comprehend a piece of content in the way humans typically do. Additionally, the success of our manual quiz reinforced our assumptions by demonstrating the value of standardizing quiz-editing practices. Our findings point to some (much-needed) good news for newsrooms as they (our newsroom AfroLA is no exception) grapple with the new relationship between AI and quality, ethical journalism…at least for now.

---

Cite this article

Csernatony, Zoli; and Amihere, Dana (2024, Sept. 5). Building AI-resistant news quizzes. Reynolds Journalism Institute. Retrieved from: https://rjionline.org/news/building-ai-resistant-news-quizzes/