Sign-up to receive Trendlines.
Our bi-monthly email with research insights on trending topics.
Copyright © Gradient Metrics 2020
Some questions are too sensitive to ask directly, even in an anonymous online survey. The more sensitive the question, the more likely respondents will feel uncomfortable and withhold their true beliefs and opinions. List experiments capitalize on indirect questioning methods to alleviate concerns of preference falsification and social desirability bias.
When you really need some honesty on a sensitive topic, list experiments are the right tool for the job. Not only that, but accurately measuring private opinion means you can estimate the gap between privately held and publicly-stated opinion.
A list experiment (also called the item-count technique) is a method for indirectly measuring private opinion for issues where individuals might otherwise be likely to publicly withhold their true opinion.
List experiments work by guaranteeing privacy. Unlike traditional survey methods, respondents are never asked directly to share their opinion about individual statements. Instead, respondents are asked to read a list of statements and choose the number with which they agree.
By comparing a group of people who see a list that includes the sensitive statement (i.e., the treatment group) to a group of people who see a list without the sensitive statement (i.e., the control group), we can make inferences about the prevalence of that private opinion in a population—or simply the difference in means between the number of endorsed statements by the treatment and control group.
Recent advancements in statistics and multivariate modeling have led to more sophisticated analyses allowing for more detailed investigation of the data.
Trendlines Opinion Survey
In March of 2020, the spread of COVID-19 was just ramping up. In such a chaotic time, directly asking respondents about their fears may have led to inaccurate findings. To get around this, Gradient conducted a list experiment to uncover how truly afraid Americans were about knowing someone who contracted COVID-19.
Using a list experiment was valuable as Americans were twice as likely to publicly say they were worried about someone they knew contracting COVID-19 than they privately, truly, felt.
While 66% of Americans publicly stated they were worried, only 33% were actually worried when asked through an experimental condition where their privacy was guaranteed.Read the full story →
Trendlines Public Opinion Survey
The Supreme Court confirmation of Judge Ketanji Brown Jackson brought with it questions about America’s opinions on adding diversity to major political positions. If asked directly most individuals would say they support increases in diversity as to not seem racist. Thus, to better gauge how Americans really feel about diversity, a list experiment is needed.
The fruitfulness of the list experiment was evident when comparing the public and private opinions across political parties. Roughly one in four Republicans (24%) publicly admitted it's important to have more Black Americans in positions of power in the judicial system—the same proportion who agreed with the statement, even when guaranteed privacy (25%).
In contrast, while 69% of Democrats publicly agreed it’s important to have more Black Americans in positions of power in the judicial system, that number decreased to under half (47%) when responding to the statement privately.Read the full story →
Client Case Study
Americans don’t feel like they live in a democracy of opinion—that is, Americans don’t feel comfortable voicing their true thoughts when they are counter to perceived societal consensus. This lack of public honesty creates widespread preference falsification to the point that systems and institutions are designed with a purpose with which many Americans actually disagree.
We designed a landmark study, using several list experiments, to measure the difference between publicly stated and privately held opinions for several sensitive statements about politics, culture, and society ranging from abortion to COVID-19 to what’s taught in public schools.
Americans of all different backgrounds feel pressure to self-censor. Across all demographics, every subgroup of Americans had large gaps between private and public opinion—oftentimes large enough to drive a bus (hopefully not full of migrants) through—depending on the topic and sensitive statement.Download the case study →
The world's most forward-looking organizations trust Gradient