library(tidyverse)
library(bayesrules)
library(gridExtra)
library(googlesheets4)
library(googledrive)
CNN vs. The Onion - Beta Binomial
Activity Introduction
The goal of this activity is to explore how prior beliefs, what we think is likely before seeing any data, can influence the conclusions we draw after seeing new evidence: our posterior. To make things interesting, we will use a quiz where you’ll try to tell whether a headline came from CNN (a real news site) or The Onion (a fake, satirical news site).
Before taking the quiz, you’ll think about how many headlines you expect a person might guess correctly, and you’ll turn that guess into a prior, or starting point for your beliefs. Then, you’ll update your beliefs using data from actual quiz results and compare your updated beliefs, posterior, to priors representing other beliefs: someone expecting a person to guess most answers correctly (optimistic), someone undecided (no prior knowledge) about whether a person will guess many right or wrong (undecided), and someone expecting to guess most answers incorrectly (pessimistic) about how many answers they will get correct in the game.
By the end of the activity you’ll be able to:
Understand what a prior and posterior are
See how beliefs change when we get new information
Make and interpret plots of different prior and posterior distributions
Calculate and compare summary statistics like the mean, mode, and standard deviation of these distributions.
Let’s dive in and see how well we can guess the news from fiction and learn about Bayesian thinking along the way!
CNN vs The Onion
CNN (the Cable News Network) is widely considered a reputable news source. The Onion, on the other hand, is (according to Wikipedia) “an American news satire organization. It is an entertainment newspaper and a website featuring satirical articles reporting on international, national, and local news.” Another way of putting it - The Onion is “fake news” for entertainment purposes.
In this exercise you will assess people’s ability to determine real news stories published on cnn.com from fake news stories published on theonion.com.
Packages
Code Set Up
The package Bayes Rule is not available in the webr server at this time, so we will include the functions here.
Priors
The CNN vs. The Onion quiz consists of 15 questions. Each question has the same possible answers: CNN or The Onion. Before we take the quiz, predict how many headlines a person will guess correctly out of 15. You might think about your ability to determine fact from fiction or your familiarity with CNN and The Onion.
Let \(\pi\) be the proportion of correct answers a person guesses right in the CNN vs the Onion quiz. Keeping that number in mind, let’s explore in the table below, three different priors for \(\pi\)
Optimistic | Undecided | Pessimistic |
---|---|---|
Beta(14, 1) | Beta(1, 1) | Beta(5, 10) |
Why do we use the Beta distribution?
The Beta distribution is commonly used as a prior for probabilities because it is defined on the interval \([0, 1]\), just like any probability.
Its two shape parameters \(\alpha\) and \(\beta\) can be interpreted as representing prior pseudo-observations:
- \(\alpha - 1\): prior successes
- \(\beta - 1\): prior failures
This means a Beta\((\alpha, \beta)\) distribution is like saying:
Before seeing any new data, I’ve seen about \(\alpha - 1\) cases where this went right, and \(\beta - 1\) where it didn’t.
This interpretation helps ground priors in real-world expectations.
When you construct your own prior, you can use your guess for how many headlines you think a person will get right as your “prior successes” and how many they’ll get wrong as your “prior failures”. For example, if you are pessimistic about the number people might get correct and think they’ll get 5 out of 15 correct, you might choose:
\[ \text{Beta}(5, 10) \]
Plotting the Priors
Where does your prediction fall?
When we construct our priors from the Beta distribution, the shape parameters \(\alpha\) and \(\beta\) can be interpreted as the approximate number of successes and the approximate number of failures. In constructing your prior, you can derive your alpha and beta parameters into how many questions out of 15 you expect a person to get correct: Beta(approx_num_correct, approx_num_wrong)
.
Returning to your own prediction, replace approx_num_correct
and approx_num_wrong
with your predictions.
Looking at the graph of your prior, which guesser is your prior most similar to: Optimistic, Undecided or Pessimistic?
We often describe priors in terms of how much information they give about the unknown variable. Priors are often described as:
Informative prior: An informative prior reflects specific information about the unknown variable with high certainty (i.e. low variability).
Vague (diffuse) prior: A vague or diffuse prior reflects little specific information about the unknown variable. A flat prior, which assigns equal prior plausibility to all possible values of the variable, is a special case.
Let’s take a look at a very informative prior from someone who is very optimistic about individuals getting answers correct.
- What do you notice about the very optimistic prior. Is it more informative? What do you notice about the plot?
- How would you classify your prior? Is it informative or vague? Why?
Updating with data
Based on observed data, we will update the posterior for our four priors and our own prior. We will do this with our own score, our neighbor’s scores, and the class or dataset score.
Next, we calculate the summary statistics for the prior and posterior for all four priors using the function
summarize_beta_binomial
:
summarize_beta_binomial(alpha, beta, y = NULL, n = NULL)
function summarizes the mean, mode, and variance of the prior and posterior Beta models of \(\pi\)Arguments:
alpha, beta
: positive shape parameters of the prior Beta modely
: number of successesn
: number of trials
Next, we plot the prior, likelihood, and the posterior for all four.
Lastly, we examine the effect of different priors on the posterior.
Take the quiz
Let’s take the quiz and add our data to the dataset of trials and successes for a person taking this quiz.
Each of you will take a quiz consisting of 15 questions. Each question has the same possible answers: CNN or The Onion. You can take the quiz through our google form:
Note down how many you get correct. We will compare this with our neighbors and pool data from the class.
Adding data
Choose the priors that interest you and we will update them as follows below:
- Add your quiz data
Let’s update the prior with some data. Add the number you got correct and enter your prior from before. Your n column should be 15 to reflect that you answered 15 questions.
- Compare with your classmates
Let’s now update your prior with your classmate to your left’s score. Your column should now be 30 to reflect the two quizzes.
Let’s now update your prior with your classmate to your right’s score. Your column should now be 45 to reflect the three quizzes.
- Finally, we will compare to the full class data or quiz dataset.
Calculating the summary statistics and plotting the distribution
Compare to your class or to our dataset
student question correct year institution
1 1 1 1 2010 Colby
2 1 2 1 2010 Colby
3 1 3 1 2010 Colby
4 1 4 0 2010 Colby
5 1 5 1 2010 Colby
6 1 6 1 2010 Colby
Let’s take a look at the student results from 2010.
The Optimist
Undecided
The Pessimist
Very Optimistic
Comparison of the priors
Fill in the gaps to add your alpha and beta shape parameters with your guess:
Inspecting the updated posteriors, how did the posteriors change or differ based on the respective priors?
What happened when you had a more informative prior, e.g. The Very Optimistic?
Look back at the summary statistics that you calculated for the distributions. Choose one of the posteriors “Optimistic, Pessimistic, Undecided, Very Optimistic” and compare it with your own. What do each of the summary statistics (mean, mode, standard deviation) tell you about the probability of success after seeing the data?
In this activity, you:
Made a prediction about how well you’d do on a quiz and turned that into a prior distribution
Learned about different types of priors: confident, undecided, and pessimistic
Updated your prior using data from the quiz to get a posterior distribution
Compared how different priors affect the posterior, even when we see the same data
Practiced reading and interpreting plots of priors and posteriors
Calculated key summary statistics like the mean, mode, and standard deviation
This exercise shows how Bayesian thinking helps us combine what we already believe with new evidence to make better, more informed decisions. And sometimes, it reminds us that even when we think we’re great at telling real news from fake news, the data might say otherwise!
Think about your prior prediction and how it compared to the data.
How similar or different was your prior to the actual results?
Did updating your beliefs with data change your thinking? In what way?
If you had chosen a different prior (e.g., more vague or more confident), how would your posterior have changed?
What does this activity show about the role of prior knowledge or assumptions in data analysis?
Take a few minutes to jot down your thoughts before share your ideas with a partner or group.