Skip to content

Your partner in decision science

We uncover objective truths through custom quantitative research programs built for impact. Our team blends curiosity, rigor, and advanced methods to help you make smarter, more confident decisions.

Meet the team behind the insights—curious, rigorous, and driven by purpose.

Our values shape how we work—with each other and with every client.

Join our remote-first team and help shape the future of decision science.

Let’s talk through your big questions and figure out the evidence that matters.

How we do research differently

For us, research isn’t about numbers in a spreadsheet—it’s about answering your most important questions with clarity, purpose, and real-world impact.

We begin with the decision you need to make—then design lean, focused research to get you there.

Spotting trends is simple. Uncovering why takes rigor. Our statistical models reveal hidden drivers that turn data into confident action.

AI is powerful, but it can’t replicate lived experience. We use it to streamline our process—not to replace the human responses that are central to our research.

See what insight can do

We believe sharing knowledge is as important as creating it. That’s why we’ve built a library of resources designed to inform, inspire, and empower decision-makers like you.

See how we’ve answered complex questions with research backed insights and creative problem-solving.

Our point of view on what’s happening in research, strategy, and human behavior.

Stay ahead with our regular roundup of fresh ideas, actionable advice, and the stories shaping the future of market research.
bg_default_6
Brendon EllisMar 19, 2024 2:06:06 PM4 min read

Best Practices for Analyzing and Interpreting List Experiment Results

Navigating the landscape of private opinion on sensitive topics can be challenging, with individuals often reluctant to openly express their true beliefs. This is where list experiments emerge as a potent tool for researchers and policymakers alike. A list experiment is a method designed to indirectly measure private opinions, safeguarding respondent anonymity and encouraging candid responses. This approach protects privacy and yields insights that may otherwise remain obscured. In this post, we delve into the intricacies of analyzing and interpreting list experiments, offering a step-by-step guide to unlock the wealth of information they hold and empowering decision-makers to navigate complex societal issues with greater clarity and understanding.

Analyzing list experiment data

Difference-in-means estimation

Historically, list experiments have been analyzed using a simple difference-in-means estimator. Specifically, the mean number of items reported by the control group subtracted from the mean number of items reported by the treatment group is equal to the proportion of the population that privately supports the sensitive item. 

There is a bias-variance tradeoff to using the list experiment. The primary advantage of using the list experiment is lower bias in revealing private opinion – historically, the primary disadvantage is higher variance. By design, we increase the noise, making it harder to find a signal. However, multivariate regression models allow us to recoup some of the variance costs of the standard difference-in-means estimator by more efficiently using the data collected from list experiments.

Multivariate regression techniques

Recent advancements in statistics and multivariate modeling have led to more sophisticated analyses (Blair & Imai, 2012; Imai, 2014). Multivariate regression techniques are now freely available through open-source platforms that allow for the following advantages over the standard difference-in-means estimator:

  1. Investigation of individual characteristics differences to understand who is most likely to endorse sensitive statements
  2. Analysis of counterfactual probabilities – i.e., which sub-groups are more likely to answer the question differently if asked directly (using traditional public opinion polling methods) or indirectly (using the list experiment)
  3. Test of the design effect – does inclusion of a sensitive item change the way people respond to the control items?
  4. Adjustments of results for the presence of ceiling or floor effects

Using statistical software, we can calculate the joint distribution of responses to the sensitive item asked both directly and indirectly. Assuming there are no liars and no design effect, we can use the maximum and minimum responses in the treatment group to calculate the population proportion for each respondent type. The table below depicts all respondent types.

 

Response to list experiment

Treatment

Control

5

(4, 1)

 

4

(3, 1) (4, 0)

(4, 1) (4, 0)

3

(2, 1) (3, 0)

(3, 1) (3, 0)

2

(1, 1) (2, 0)

(2, 1) (2, 0)

1

(0, 1) (1, 0)

(1, 1) (1, 0)

0

(0, 0)

(0, 1) (0, 0)

 

Note: The first number in the parentheses refers to the number of control items a participant agrees with and the second number refers to whether or not they agree with the sensitive item.

Case study: Private Opinion in America

In Gradient's journey to uncover how to accurately measure private opinion in America, we often partner with think tank Populace. One of Populace's core beliefs is that Americans don’t feel like they are living in a democracy of opinion—that Americans don’t feel comfortable voicing their true thoughts because they are counter to society’s. This lack of public honesty creates widespread preference falsification to the point that systems and institutions are designed with a purpose with which many Americans actually disagree. 

Social desirability bias is a problem in all self-report research, but it is especially common in research on sensitive topics. Respondents feel pressure to conceal the truth due to a variety of factors, including social pressures (e.g., reporting private opinions expected to be dissonant with the collective attitude), fear of reprisal, or lack of privacy. It is often difficult or impossible to determine the size and the direction of misreporting bias.

In the Private Opinion in America project, Gradient deployed a list experiment to estimate how Americans privately feel on a variety of social issues and compare that against public opinion on the same issues.

The results of the project revealed the pressure to misrepresent our private views—to offer answers on politically and socially sensitive questions that are out of sync with our true beliefs—is pervasive in society today. Across all demographics every subgroup had multiple issues with at least a double-digit gap between public and private opinion.

For example, when it comes to abortion, men are not as supportive as their public opinions would suggest. In public, a majority of men (60%) agree the decision to have an abortion should be left to a woman and her doctor; however, in private that number is only 45%. In addition, most men (52%) publicly say that “abortion should be legal in most cases,” but in private that is not the majority view (48%).

Conclusion

Using a list experiments offers a transformative approach to uncovering private opinions on sensitive topics, illuminating the stark disparities between public and private stances. Gradient's sophisticated approach to analyzing and interpreting list experiment data underscores the significance of safeguarding respondent anonymity and employing rigorous statistical techniques to extract nuanced—and actionable—insights.

Through this methodology, decision-makers at think tank Populace and other organizations can gain a clearer understanding of societal dynamics, enabling them to navigate complex issues with heightened sensitivity and informed decision-making. Results from list experiments can reveal the pervasive pressure on the public to misrepresent private views, thereby underscoring the imperative of embracing innovative methodologies to capture the true pulse of private opinion.

Does your organization need to capture the private opinion of your audience? Gradient can help! Let us know your research needs. 

 

avatar
Brendon Ellis
Brendon earned his master’s degree in Positive Organizational Psychology & Evaluation from Claremont Graduate University and is currently pursuing his PhD in Positive Organizational Psychology. He applies his background in survey research methods on a daily basis to help clients develop instruments that measure anything from attitudes and preferences to intentions and behavior.

RELATED ARTICLES