List Experiments

Uncover real opinions about sensitive topics

3 List experiment without bg

Get honest answers to sensitive questions

Why use a list experiment

Some questions are too sensitive to ask directly, even in an anonymous online survey. The more sensitive the question, the more likely respondents will feel uncomfortable and withhold their true beliefs and opinions. List experiments capitalize on indirect questioning methods to alleviate concerns of preference falsification and social desirability bias.

When you really need some honesty on a sensitive topic, list experiments are the right tool for the job. Not only that, but accurately measuring private opinion means you can estimate the gap between privately held and publicly-stated opinion.

magnifier

Guarantee privacy to your audience

reduce

Reduce bias and preference falsification

priority

Reveal the gap between public and private opinion

What are list experiments?

How it works

A list experiment (also called the item-count technique) is a method for indirectly measuring private opinion for issues where individuals might otherwise be likely to publicly withhold their true opinion. 

List experiments work by guaranteeing privacy. Unlike traditional survey methods, respondents are never asked directly to share their opinion about individual statements. Instead, respondents are asked to read a list of statements and choose the number with which they agree.

By comparing a group of people who see a list that includes the sensitive statement (i.e., the treatment group) to a group of people who see a list without the sensitive statement (i.e., the control group), we can make inferences about the prevalence of that private opinion in a population—or simply the difference in means between the number of endorsed statements by the treatment and control group.

Recent advancements in statistics and multivariate modeling have led to more sophisticated analyses allowing for more detailed investigation of the data.

answer

We build custom survey questions to fit your particular business needs

data_chain

We use state-of-the-art Bayesian models to estimate true, private opinion

benchmark

With easy-to-understand charts, we reveal private opinions in a visually engaging format

You get the real opinions on sensitive topics

What you get

Frequently asked questions (FAQ)

When would you use a list experiment?

There are several common situations that might demand a list experiment:

  • When you want to learn about the prevalence of sensitive attitudes, perceptions, or behaviors
  • When privacy is imperative for honest self-reporting
  • When there’s reason to believe respondents are incentivized to withhold their honest opinion(s)
  • When you want to know if there’s a gap between public and private opinions
  • When you want to know who is more likely to withhold their public opinion


Is there a limit to the number of list experiments you can add to a survey?

While there isn’t a hard and fast limit, list experiments are cognitively demanding tasks. For every list experiment, respondents must weigh their opinion on 4-5 separate statements all while counting. It’s best to limit surveys to a handful of list experiment tasks.



What kind of results come from a list experiment?

The output from a single list experiment is a percentage that indicates the probability of the sample agreeing with the sensitive statement of interest. If the sensitive statement is asked as a direct question, the increase or decrease in likelihood of a respondent answering the question privately, instead of publicly, can also be calculated.

How do list experiments guarantee privacy?

No single list experiment can guarantee complete privacy (some people agree with all or none of the statements you show them). But to minimize bias and maximize privacy, we carefully craft lists of statements to meet a few important standards. First, we design statements that are thematically similar to the sensitive statement. Second, we assemble lists of statements to minimize the percentage of people who agree with all or none of the statements.

Stories from the field

List Experiment 3

Assessing Fears in the Early Days of COVID-19

Trendlines Opinion Survey

In March of 2020, the spread of COVID-19 was just ramping up. In such a chaotic time, directly asking respondents about their fears may have led to inaccurate findings. To get around this, Gradient conducted a list experiment to uncover how truly afraid Americans were about knowing someone who contracted COVID-19.

Using a list experiment was valuable as Americans were twice as likely to publicly say they were worried about someone they knew contracting COVID-19 than they privately, truly, felt. 

While 66% of Americans publicly stated they were worried, only 33% were actually worried when asked through an experimental condition where their privacy was guaranteed.

Read the full story →
List Experiment 2

Private Versus Public Opinions about Diversity in the Judicial System

Trendlines Public Opinion Survey

The Supreme Court confirmation of Judge Ketanji Brown Jackson brought with it questions about America’s opinions on adding diversity to major political positions. If asked directly most individuals would say they support increases in diversity as to not seem racist. Thus, to better gauge how Americans really feel about diversity, a list experiment is needed.

The fruitfulness of the list experiment was evident when comparing the public and private opinions across political parties. Roughly one in four Republicans (24%) publicly admitted it's important to have more Black Americans in positions of power in the judicial system—the same proportion who agreed with the statement, even when guaranteed privacy (25%). 

In contrast, while 69% of Democrats publicly agreed it’s important to have more Black Americans in positions of power in the judicial system, that number decreased to under half (47%) when responding to the statement privately.

Read the full story →
List Experiment 6

Populace Insights: Private Opinion in America

Client Case Study

Americans don’t feel like they live in a democracy of opinion—that is, Americans don’t feel comfortable voicing their true thoughts when they are counter to perceived societal consensus. This lack of public honesty creates widespread preference falsification to the point that systems and institutions are designed with a purpose with which many Americans actually disagree

We designed a landmark study, using several list experiments, to measure the difference between publicly stated and privately held opinions for several sensitive statements about politics, culture, and society ranging from abortion to COVID-19 to what’s taught in public schools.

Americans of all different backgrounds feel pressure to self-censor. Across all demographics, every subgroup of Americans had large gaps between private and public opinion—oftentimes large enough to drive a bus (hopefully not full of migrants) through—depending on the topic and sensitive statement.

Download the case study →

Who we’ve partnered with

The world's most forward-looking organizations trust Gradient

framebridge
oscar
bacardi
estee-lauder
harrys
populace
nike

Want to learn more?

Contact us now!