CNN vs. The Onion - Beta Binomial

Authors

Affiliations

Laurie Baker

Bates College

Jim Scott

Colby College

Mine Dogucu

UC Irvine

Activity Introduction

The goal of this activity is to explore how prior beliefs, what we think is likely before seeing any data, can influence the conclusions we draw after seeing new evidence: our posterior. To make things interesting, we will use a quiz where you’ll try to tell whether a headline came from CNN (a real news site) or The Onion (a fake, satirical news site).

Before taking the quiz, you’ll think about how many headlines you expect a person might guess correctly, and you’ll turn that guess into a prior, or starting point for your beliefs. Then, you’ll update your beliefs using data from actual quiz results and compare your updated beliefs, posterior, to priors representing other beliefs: someone expecting a person to guess most answers correctly (optimistic), someone undecided (no prior knowledge) about whether a person will guess many right or wrong (undecided), and someone expecting to guess most answers incorrectly (pessimistic) about how many answers they will get correct in the game.

Learning objectives

By the end of the activity you’ll be able to:

• Understand what a prior and posterior are

• See how beliefs change when we get new information

• Make and interpret plots of different prior and posterior distributions

• Calculate and compare summary statistics like the mean, mode, and standard deviation of these distributions.

Let’s dive in and see how well we can guess the news from fiction and learn about Bayesian thinking along the way!

CNN vs The Onion

CNN (the Cable News Network) is widely considered a reputable news source. The Onion, on the other hand, is (according to Wikipedia) “an American news satire organization. It is an entertainment newspaper and a website featuring satirical articles reporting on international, national, and local news.” Another way of putting it - The Onion is “fake news” for entertainment purposes.

In this exercise you will assess people’s ability to determine real news stories published on cnn.com from fake news stories published on theonion.com.

Frontpage of the Onion. Photo: Casey Bisson, via Flickr

Packages

library(tidyverse)
library(bayesrules)
library(gridExtra)
library(googlesheets4)
library(googledrive)

Code Set Up

Run this set up code to load the functions from Bayes Rule package first

The package Bayes Rule is not available in the webr server at this time, so we will include the functions here.

Priors

The CNN vs. The Onion quiz consists of 15 questions. Each question has the same possible answers: CNN or The Onion. Before we take the quiz, predict how many headlines a person will guess correctly out of 15. You might think about a person’s ability to determine fact from fiction or their familiarity with CNN and The Onion.

Let \(\pi\) be the proportion of correct answers a person guesses right in the CNN vs the Onion quiz. Keeping that number in mind, let’s explore in the table below, three different priors for \(\pi\)

Optimistic	Undecided	Pessimistic
Beta(14, 1)	Beta(1, 1)	Beta(5, 10)

Why do we use the Beta distribution?

The Beta distribution is commonly used as a prior for probabilities because it is defined on the interval \([0, 1]\), just like any probability.

Its two shape parameters \(\alpha\) and \(\beta\) can be interpreted as representing prior pseudo-observations:

• \(\alpha - 1\): prior successes
• \(\beta - 1\): prior failures

This means a Beta\((\alpha, \beta)\) distribution is like saying:

Before seeing any new data, I’ve seen about \(\alpha - 1\) cases where this went right, and \(\beta - 1\) where it didn’t.

This interpretation helps ground priors in real-world expectations.

When you construct your own prior, you can use your guess for how many headlines you think a person will get right as your “prior successes” and how many they’ll get wrong as your “prior failures”. For example, if you are pessimistic about the number people might get correct and think they’ll get 5 out of 15 correct, you might choose:

\[ \text{Beta}(5, 10) \]

Plotting the Priors

Where does your prediction fall?

When we construct our priors from the Beta distribution, the shape parameters \(\alpha\) and \(\beta\) can be interpreted as the approximate number of successes and the approximate number of failures. In constructing your prior, you can derive your alpha and beta parameters into how many questions out of 15 you expect a person to get correct: Beta(approx_num_correct, approx_num_wrong).

Returning to your own prediction, replace approx_num_correct and approx_num_wrong with your predictions.

Looking at the graph of your prior, which guesser is your prior most similar to: Optimistic, Undecided or Pessimistic?

Vocabulary

We often describe priors in terms of how much information they give about the unknown variable. Priors are often described as:

• Informative prior: An informative prior reflects specific information about the unknown variable with high certainty (i.e. low variability).

• Vague (diffuse) prior: A vague or diffuse prior reflects little specific information about the unknown variable. A flat prior, which assigns equal prior plausibility to all possible values of the variable, is a special case.

Let’s take a look at a very informative prior from someone who is very optimistic about individuals getting answers correct.

Reflection:

• What do you notice about the very optimistic prior. Is it more informative? What do you notice about the plot?
• How would you classify your prior? Is it informative or vague? Why?

Updating with data

• Based on observed data, we will update the posterior for our four priors and our own prior. We will do this with our own score, our neighbor’s scores, and the class or dataset score.

• Next, we calculate the summary statistics for the prior and posterior for all four priors using the function summarize_beta_binomial:

summarize_beta_binomial

• summarize_beta_binomial(alpha, beta, y = NULL, n = NULL) function summarizes the mean, mode, and variance of the prior and posterior Beta models of \(\pi\)

• Arguments:

  • alpha, beta: positive shape parameters of the prior Beta model
  • y: number of successes
  • n: number of trials

• Next, we plot the prior, likelihood, and the posterior for all four.

• Lastly, we examine the effect of different priors on the posterior.

Take the quiz

Let’s take the quiz and add our data to the dataset of trials and successes for a person taking this quiz.

Each of you will take a quiz consisting of 15 questions. Each question has the same possible answers: CNN or The Onion. You can take the quiz through our google form:

Note down how many you get correct. We will compare this with our neighbors and pool data from the class.

Adding data

Choose the priors that interest you and we will update them as follows below:

1. Add your quiz data

Let’s update the prior with some data. Add the number you got correct and enter your prior from before. Your n column should be 15 to reflect that you answered 15 questions.

2. Compare with your classmates

• Let’s now update your prior with your classmate to your left’s score. Your column should now be 30 to reflect the two quizzes.

• Let’s now update your prior with your classmate to your right’s score. Your column should now be 45 to reflect the three quizzes.

3. Finally, we will compare to the full class data or quiz dataset.

Calculating the summary statistics and plotting the distribution

The Optimist

Undecided

The Pessimist

Very Optimistic

Your prior

Reflection

• How does observing more data affect the shape of the posterior?

• What happens to the posterior mean as you observe more correct or incorrect outcomes?

• In what ways does the posterior reflect a compromise between prior belief and observed data?

Compare to your class or to our dataset

  student question correct year institution
1       1        1       1 2010       Colby
2       1        2       1 2010       Colby
3       1        3       1 2010       Colby
4       1        4       0 2010       Colby
5       1        5       1 2010       Colby
6       1        6       1 2010       Colby

Let’s take a look at the student results from 2010.

The Optimist

Undecided

The Pessimist

Very Optimistic

Comparison of the priors

Fill in the gaps to add your alpha and beta shape parameters with your guess:

Reflection

• Inspecting the updated posteriors, how did the posteriors change or differ based on the respective priors?

• What happened when you had a more informative prior, e.g. The Very Optimistic?

• Look back at the summary statistics that you calculated for the distributions. Choose one of the posteriors “Optimistic, Pessimistic, Undecided, Very Optimistic” and compare it with your own. What do each of the summary statistics (mean, mode, standard deviation) tell you about the probability of success after seeing the data?

Recap

In this activity, you:

• Made a prediction about how well you’d do on a quiz and turned that into a prior distribution

• Learned about different types of priors: optimistic, undecided, and pessimistic

• Updated your prior using data from the quiz to get a posterior distribution

• Compared how different priors affect the posterior, even when we see the same data

• Practiced reading and interpreting plots of priors and posteriors

• Calculated key summary statistics like the mean, mode, and standard deviation

This exercise shows how Bayesian thinking helps us combine what we already believe with new evidence to make better, more informed decisions. And sometimes, it reminds us that even when we think we’re great at telling real news from fake news, the data might say otherwise!

Reflection

Think about your prior prediction and how it compared to the data.

• How similar or different was your prior to the actual results?

• Did updating your beliefs with data change your thinking? In what way?

• If you had chosen a different prior (e.g., more vague or more informative), how would your posterior have changed?

• What does this activity show about the role of prior knowledge or assumptions in data analysis?

Take a few minutes to jot down your thoughts before share your ideas with a partner or group.