Society · Human trial
Behavioral interventions have modest effects on climate beliefs and actions
- Researchers tested eleven strategies to change people's climate beliefs and behaviors across 59,440 participants in 63 countries.
- Most strategies produced small improvements in beliefs or policy support, but none successfully encouraged people to do something effortful like plant trees.
- The effectiveness of each strategy depended heavily on whether people were already skeptical about climate change.
Science Advances · 2024 — https://doi.org/10.1126/sciadv.adj5778
Addressing climate change with behavioral science: A global intervention tournament in 63 countries
INTRODUCTION
The climate crisis is one of humanity’s most consequential and challenging problems. Successfully rising to the challenge depends on both “top-down” structural changes (e.g., regulation and investment) and “bottom-up” changes (e.g., individuals’ and collectives’ beliefs and behaviors). These bottom-up processes require widespread belief in climate change, support for climate change policy, and willingness to engage in climate action (–). The behavioral sciences have been seen as a crucial component in promoting bottom-up change, through the development of large-scale interventions that can shift public opinion and enable and support top-down governmental climate policies (–). However, it is unclear which strategies are most likely to motivate people to change their climate change beliefs and climate mitigation behaviors. Here, we assess the effectiveness of expert-crowdsourced, theoretically derived interventions at promoting a range of climate change mitigation behaviors in a large and diverse global sample.
A growing body of research across the behavioral sciences has been investigating intervention strategies aimed at boosting sustainable intentions and behaviors such as recycling, public transportation use, and household energy saving. For instance, communications aimed at reducing the psychological distance of climate change, by making it feel more geographically, socially, and temporally close, were effective at increasing climate concern and amplifying self-reported intentions to engage in mitigating behaviors, such as reducing energy consumption. Similarly, normative appeals that include an invitation to work together and “join in” were found effective at influencing behaviors such as charitable giving. These are only two examples in a growing list of behavioral interventions designed to mitigate climate change. Hence, there are numerous competing theories in the behavioral sciences about how to stimulate climate change beliefs and proenvironmental behaviors.
While many of these theories, as well as their corresponding interventions, are promising, they have been tested independently with different samples and, on separate outcomes, making it impossible to directly compare their effectiveness. In addition, assessing interventions on a single outcome renders it difficult to understand their effects on multiple facets of climate mitigation, which are all necessary to substantially reduce climate change (e.g., support for climate mitigation policy and sustainable behavior). These limitations are a major barrier to resolving theoretical debates within the scientific community and to translating scientific findings into impactful policies. Moreover, traditional attempts to compare interventions (e.g., meta-analyses) are limited by differences in experimental protocols, outcome variables, samples, and operationalizations. These differences hinder evaluations of the relative effectiveness of different theories and interventions. To address these concerns, we used the megastudy approach—an experimental paradigm similar to a randomized controlled trial but designed to evaluate the efficacy of many interventions on several outcome variables, in the same large-scale experiment. This provides a rigorous direct comparison of competing approaches to climate change mitigation.
Another challenge is that most prior work across the behavioral sciences (including the megastudy approach) has been mainly conducted on Western, educated samples from industrialized, rich, and developed countries (i.e., WEIRD). Results from these samples may not generalize to other nations, restricting the ability to apply findings beyond WEIRD populations. This is a particular problem for a topic like climate change where the social and political dynamics, as well as exposure to the impacts of climate change, vary across countries. While wealthier nations are disproportionately responsible for causing climate change, it is still important to understand which interventions work across a diversity of cultures since the most effective mitigation strategies will likely require global cooperation. Accordingly, we leveraged the many labs approach, in which the same study is being conducted by many research laboratories around the world, aggregating the results in the same international dataset.
In this global megastudy, we crowdsourced interventions previously found to stimulate climate mitigation from behavioral science experts (fig. S5). We used a crowdsourcing approach to determine which interventions to test, given recent evidence that crowdsourcing can improve the quality of scientific investigations by promoting ideation, inclusiveness, transparency, rigor, and reliability. This resulted in the identification of 11 behavioral interventions based on competing theoretical frameworks in the behavioral sciences.
Fig. 1.. Interventions, theoretical frameworks, and brief descriptions.
We tested these interventions in a global tournament spanning 63 countries on four outcome variables, which were also crowdsourced and selected on the basis of their theoretical and practical relevance to climate mitigation. The first outcome on which we assessed each intervention was belief in climate change (four items; e.g., “Climate change poses a serious threat to humanity”). Given that belief is a key antecedent of proenvironmental intentions, behavior, and policy support, we examined how the interventions would affect these outcomes for different people along the belief continuum ranging from skeptics to true believers.
The second outcome was support for climate change mitigation policy (nine items; e.g., “I support raising carbon taxes on gas/fossil fuels/coal”). Given that successful climate mitigation requires large-scale policy reform and the public’s support for climate policies is the top predictor of policy adoption, this outcome variable reflects the importance of impactful systemic change, rather than private mitigation efforts based on individual decision-making (–). Recent work argues that individual-level behaviors should be targeted alongside structural changes, especially since framing climate change as an individual level problem can backfire, leading to feelings of helplessness and concerns about free riding.
To target more ecologically valid behavior and climate activism, the third outcome was willingness to share climate mitigation information on social media (i.e., “Did you know that removing meat and dairy for only two out of three meals per day could decrease food-related carbon emissions by 60%?”). While this behavior is relatively low effort, recent work suggests climate information sharing with one’s community as an essential step in addressing the climate crisis.
Last, given the large gap between self-reported measures and objective proenvironmental behavior, the fourth outcome we targeted was a more effortful behavior of contributing to a real tree planting initiative by engaging in a cognitively demanding task (i.e., a modified version of the work for environmental protection task or WEPT). The WEPT is a multitrial, web-based procedure in which participants choose to exert voluntary effort screening stimuli for specific numerical combinations (i.e., an even first digit and odd second digit) in exchange for donations to a tree-planting environmental organization. Thus, they had the opportunity to produce actual environmental benefits at actual behavioral costs, mimicking classic sustainable behavior trade-offs (–).
Participants (N = 59,440, from 63 counties;) were mostly recruited through online data collection platforms (80.8%) or via convenience/snowball sampling (19.1%;). They were randomly assigned to 1 of 11 experimental interventions or a no intervention control condition in which they read a passage from a literary text. Then, in a randomized order, participants indicated their climate beliefs, climate policy support, and willingness to share climate-related information on social media. Last, participants were able to opt into completing up to eight pages of a tree-planting task, each completed page resulting in the real planting of a tree through a donation to The Eden Reforestation Project. As a result of participants’ behavior, our team actually planted 333,333 trees. Assuming that the average fully grown tree absorbs between 10 and 40 kg of carbon dioxide per year in 5 to 10 years when all trees are fully grown, the efforts from this project will result in ~9,999,990 kg of carbon dioxide sequestered per year, which is the equivalent amount of carbon dioxide used to produce energy for 1260 U.S. homes.
Table 1.. Variables on which the samples in each country were matched to the population.Countries in which no demographic variable was census matched are marked as “N/A” in the “matched variables” column. SES, socioeconomic status.
Main effects of intervention
First, we examined the effect of each intervention on each of the four outcomes, estimated using a series of Bayesian regressions (see Materials and Methods). As the goal of this study is to estimate the relative effectiveness of treatments, in contrast to establishing non-null effects or differences, Bayesian estimation is preferable to classical null hypothesis significance testing. Bayesian techniques produce posterior distributions for parameters (here, treatment effects) that characterize their magnitude and associated uncertainty. We summarize this distribution in using a point estimate corresponding to the mean and a 94% credible region, which differs from a confidence interval in which it indicates a region with a 94% chance of containing the unobserved parameter value. Moreover, we also conducted similar frequentist analyses (hierarchical mixed models) and found converging results (see the Supplementary Materials for details).
Fig. 2.. Average effects (i.e., posterior estimates using Bayesian regressions) by intervention for each outcome.Dots indicate the mean, with error bars indicating the 94% credible region. Thicker error bars indicate the interquartile range. Vertical lines indicate control average. (A) Belief, (B) support for policy, (C) willingness to share climate change information on social media, and (D) number of trees planted in the WEPT. Estimates are reported in tables S1 to S4.
We began by assessing the main intervention effects on each outcome. For belief in climate change (measured on a scale from 0 to 100), the top performing intervention, decreasing psychological distance, increased beliefs by an absolute effect size of 2.3% (1.6 to 2.9) (94% credible region) compared to the control condition. Consistent with prior work, some interventions slightly increased beliefs. However, other interventions had near-zero effect, suggesting that findings of some prior research did not extend to this context.
For climate policy support (measured on a scale from 0 to 100), the intervention with the largest average effect was writing a letter to a member of the future generation, which increased policy support by 2.6% (2.0 to 3.2). Similar to belief, all interventions produced either more policy support or no discernible differences from the control condition.
For willingness to share climate change information on social media (measured as a binary choice), all interventions generally increased intentions to share. The largest gains were exhibited in the negative emotion induction condition, which led to 12.1% (9.8 to 14.6) more sharing compared to the control condition.
For the number of pages completed on the WEPT tree-planting task (from 0 to 8), no intervention was better than the control condition, and some interventions (i.e., decreasing psychological distance, inducing negative emotions, work-together normative appeals, and writing a letter to a future-generation member) appeared to reduce tree planting. These results held regardless of the operationalization of a tree planted as participants’ confirmation that they wanted to complete another WEPT page or their accuracy in the task (table S24).
The interventions that produced negative effects on the WEPT were also those that took the most time to complete (see the Supplementary Materials). Assuming that participants have a limited budget of time for completing surveys and given that the tree-planting task requires time, it is expected that we observed a trade-off between the time spent on the intervention and on the outcome task. Therefore, in an exploratory analysis (tables S22 and S23), we assessed the effects of the interventions when adjusting for the time spent on each intervention. While we still observed the negative effects of some interventions on tree planting, we now also observed positive effects of five interventions. That is, when controlling for intervention length, binding moral foundations, scientific consensus, dynamic norms, pluralistic ignorance, and system justification all increased the number of trees planted compared to the control condition. Thus, in the absence of time constraints, these interventions might increase proenvironmental behavior. However, the degree to which these findings actually generalize to proenvironmental behaviors that do not hinge on time (e.g., donations) should be assessed in future studies.
For further assessing the average effects of each intervention on each outcome within any subsample of interest varying along demographics such as nationality, political ideology, age, gender, education, or income level, we provide an easy to use and disseminate web tool: https://climate-interventions.shinyapps.io/climate-interventions/.
Heterogeneous intervention effects along initial belief continuum
We found a high level of belief in climate change [i.e., 85.7% (85.2 to 86.2), an estimate computed using the ratings of belief in the control participants and estimated preintervention levels of belief from all other participants]. This could raise two potential concerns when evaluating the main effects of the interventions mentioned above: On the one hand, at this high level of belief, participants may be particularly receptive to interventions. As a result, average effects may tend to overestimate the effectiveness of interventions in applied contexts where the aim is to increase belief or policy support in skeptical participants that do not already believe in climate change. On the other hand, as our outcomes are bounded, these high levels of belief may lead to ceiling effects in the estimation of the average effects, which may undervalue the true effectiveness of the interventions. To address this concern, we conducted an additional analysis where we modeled heterogeneous effects as a function of unobserved preintervention belief (see Materials and Methods and the Supplementary Materials). This analysis allowed us to visualize how effective interventions were across the continuum from climate change skeptics (i.e., those with initial beliefs less than 35%) to true believers (i.e., those with initial beliefs higher than 65%;).
Fig. 3.. Marginal effects (as the difference between interventions and control) as a function of estimated preintervention belief in climate change.Lines indicate the average effect size, with shaded regions indicating the 94% credible region. For visual clarity, regions in which the 94% credible region overlap zero are omitted from the figure. Where interventions have positive and negative effects that meet these criteria, a dashed line is used to connect these regions. The dashed vertical line indicates average belief, where effects in Fig. 1 are estimated. (A) Climate change belief, (B) policy support, (C) sharing information on social media, and (D) trees planted via the WEPT.
For the impact of interventions on belief, we found clear indications of ceiling effects with many interventions being maximally impactful among uncertain participants, even those with low to moderate levels of initial belief. Even in participants with low levels of preexisting climate change belief (i.e., less than 35%), interventions such as reducing psychological distance, future self-continuity, and effective collective action are all viable ways to increase belief in climate change.
For policy support, a different pattern emerged. Interventions such as writing a letter to a member of the future generation, collective action efficacy, future self-continuity, and decreasing psychological distance all increased support for climate policy. Those same interventions appear to function well on individuals with modest to high levels of initial climate change belief (i.e., at ~35 to 90%;). However, they were relatively ineffectual among those that were low in initial belief (i.e., climate skeptics). The main exception is in writing a letter to a member of the future-generation intervention, which worked across nearly the entire spectrum of initial belief. In addition, for those that were very low to moderate (i.e., 0 to 65%) on initial belief, the negative emotion intervention appeared to backfire, reducing support for climate change policies. Similar to belief, the work-together normative appeal also slightly backfired in participants with moderate levels of initial belief.
Regarding social media sharing, nearly all interventions (i.e., 9 of 11) increased willingness to share even at moderate levels of initial belief (i.e., those greater than ~35 to 60%). Moreover, the increase in willingness to share by inducing negative emotions extended into individuals who generally do not believe in climate change. Last, the work-together normative appeal intervention backfired among those who are very low on initial belief (i.e., ~0 to 15%), reducing their willingness to share information on social media by up to 12%. Last, for the tree-planting task, more than half of the interventions decreased the number of pages completed on the WEPT across all levels of initial belief.
Country-level main effects
Last, we examined the country-level main effects for each of our key outcome variables. We found that average belief in climate change, across all countries surveyed, was high [85.7% (85.2 to 86.2); this includes both ratings of belief in the control participants and estimated preintervention levels of belief from all other participants]. There was a very little variation between countries (, fig. S4A, and table S5) indicating a clear majority belief in climate change. Similar patterns were observed for policy support, with all countries indicating clear majority support for a variety of climate change policies [72.2% (71.6 to 72.8)]. These results suggest that there is clear and consistent global consensus regarding the dangers posed by climate change and the importance of enacting climate change mitigation.
Fig. 4.. Country-level means of each outcome variable.Countries without available data are shown in gray. Statistics are shown in tables S5 to S8. (A) Climate change belief, (B) policy support, (C) sharing information on social media, and (D) trees planted via the WEPT.
Other outcome variables exhibited larger variation across countries. Willingness to share climate change–related information on social media was more modest [56.9 (56.4 to 57.5)] and variable, ranging from low in Latvia of 17.6% (14.3 to 21.4) to high of 93.3% (90.4 to 95.7) in Kenya. These results suggest that observations of climate change discussion online may not accurately reflect global sentiments about the reality of climate change but rather different local norms. Last, half of all participants (50.7% of total sample and 53.1% of control condition sample) completed all eight pages of the WEPT, earning the maximum number of trees possible, with an overall average of 5.2 (5.1 to 5.3) pages completed.
DISCUSSION
In a global megastudy conducted on a sample of 59,440 people from 63 countries, we empirically assessed the relative effectiveness of 11 expert-crowdsourced, theoretically derived behavioral interventions at stimulating climate mitigation beliefs and behaviors (i.e., climate change beliefs, policy support, willingness to share information, and tree-planting contributions). We found that different interventions tended to have small global effects, which varied across outcomes and largely affected nonskeptics, emphasizing the importance of examining the impact of climate interventions on a range of outcomes before drawing conclusions regarding their overarching relative efficacy. These findings suggest that the impact of behavioral climate interventions varies across audiences’ characteristics and target behaviors.
Here, climate change beliefs were strengthened most by decreasing the psychological distance of climate change. Support for climate change mitigation policy was increased mostly by writing a letter to be read in the future by a socially close child, describing one’s current climate change mitigation actions. Willingness to share climate change information on social media was increased most by inducing negative emotions through “doom and gloom”–styled messaging about the consequences of climate change. Last, while half of the tested interventions had no effect on the effortful tree-planting behavior, the other half of the interventions reduced the number of trees participants planted. Beyond revealing the utility of harnessing a multioutcome approach, these results also highlight the need for tailoring interventions to target outcomes.
Our findings extend prior work and are theoretically informative in several ways. Notably, these findings help reconcile several theoretical debates in the literature. For example, some have argued in favor of using a doom and gloom messaging style in climate communications (i.e., induce negative emotions) as a way to stimulate climate mitigation behaviors. For instance, recent work found that online news consumption is largely driven by the negative content of the news. However, others have warned that this messaging may have no impact on behavior or, worse, that it may depress and demoralize the public into inaction. Here, we found empirical support for both accounts on different outcomes: While negative emotion messaging was highly effective at stimulating climate information–sharing intentions (a relatively low-effort behavior), it decreased tree-planting efforts. Further, the negative emotion induction intervention appeared to backfire on policy support among participants with low initial climate beliefs. These results suggest that climate scientists should carefully consider the differential effects of the prevalent fear-inducing writing styles on different proclimate outcomes. Moreover, it suggests that theoretical models need to explain divergent patterns across outcomes.
The results also indicate that the impact of the interventions on each outcome depends on peoples’ preexisting belief in climate change, supporting the claim that interventions need to be tailored to the characteristics of their audience. For belief, the effectiveness of several interventions (e.g., decreasing the psychological distance and collective action efficacy) was maximized among the uncertain, with lesser effects among believers and skeptics. For policy support, however, interventions were generally only effective among those with high initial levels of belief, with negative emotions backfiring among skeptics. Similarly, the robust increases in willingness to share on social media were largely restricted to people who already believed in climate change—with negative emotions increasing sharing intentions even among skeptics. For the higher effort behavior, however, interventions appeared to uniformly reduce tree planting across all levels of initial belief.
Given the heterogeneity of these results across outcomes, we created a web tool resource (https://climate-interventions.shinyapps.io/climate-interventions/) that can easily and rapidly assess intervention efficacy across each of the four outcomes and across a range of variables, including country, political ideology, gender, age, socioeconomic status, income, and education. While we caution that users must take into account the sample sizes when exploring subsamples of the data and the fact that they are looking at percentage of change compared to the control condition, this web tool can be used as a rapid and intuitive way to query intervention efficacy within subsamples of interest. For example, for highly educated conservatives in the United States, the top intervention to increase climate policy support was the future self-continuity intervention, increasing support by 18%. This intervention also increased climate beliefs in Russian participants by 9%. The scientific consensus intervention increased climate policy support by 9% in Romania but decreased it by 5% in Canada. The binding moral foundations intervention increased the number of trees planted by Australians under the age of 40 by 40%, and by Gambians by 35%, but this intervention decreased the number of trees planted by wealthy Japanese participants by 24%. These results can inform the development of local intervention strategies, which should then be empirically validated. Critically, these results also bolster the message that interventions need to be tailored to the characteristics of the target audience, nationality being an important factor. The accompanying data exploration web tool and the open-source raw dataset contribute to the data-as-public-good trend emerging in the spirit of open science, thus facilitating the testing of additional hypotheses and advancement of science.
In a linked forecasting experiment, academics (e.g., behavioral scientists) and the general public were asked to predict how each intervention would affect belief, policy support, and the tree-planting behavior in a subset of participants from this study (i.e., those from the United States). While academics were better than the general public at predicting the efficacy of these interventions on beliefs and policy support, when compared to statistical models using simple heuristics such as “interventions would have no effect,” no group was able to accurately predict how interventions would affect behavior. These results suggest that our findings here reflect an important departure from the expectations within the academic community.
There are also several limitations and future directions that should be emphasized. First, the sampling procedures differed between countries (e.g., the U.S. and Israel samples matched the census on age, gender, region, ethnicity; and the Norway sample matched on age, gender, ethnicity; etc.;). It should be noted that 73.6% of the entire sample was matched for at least one variable. However, despite these differences, recent work has found that representative samples are not required to obtain generalizable estimates of effect sizes within countries. Various analyses have highlighted that convenience samples are adequate for estimating treatment effects. Hence, given that our paper is primarily concerned with the effects of these interventions rather than with estimating levels of opinion within each country, our sampling procedures were appropriate for the analyses and conclusions drawn here. However, while realizing that it will be a challenge, we encourage future work to examine these processes using larger, more representative samples from an even broader sample of countries.
Second, we leveraged an online survey–based approach, which means that we were able to capture a limited set of contextual factors that may have influenced our results. This approach was the most effective way to measure and compare intervention efficacy in such a diverse global sample. However, one important and potentially impactful avenue for future research could be to leverage these findings to conduct local field experimentation in targeted samples.
One of the major strengths of our tournament was testing 11 different interventions simultaneously in a large global sample across multiple outcomes. Given the heterogeneity in the effectiveness of the interventions across the outcomes, future work should likewise prioritize testing promising interventions on even more climate-relevant antecedents and outcomes for a more comprehensive assessment of climate interventions and their underlying theoretical frameworks. One constraint we faced when attempting to test additional theories was the decision to not use deception in our interventions. For example, descriptive or injunctive norm–based interventions would have needed to be based on deception to be included in and deployed at this global scale, given the unavailability of the empirical information critical to creating these interventions. We hope that the current dataset can provide this information for future research in international contexts. Future work should also investigate additional proenvironmental behaviors, such as investment decisions, activism, advocacy, or civic participation, critical to climate change mitigation.
Future research should also assess the processes behind the negative effects we observed on the tree-planting task. Here, we find evidence for a trade-off between time spent on the intervention and in the behavioral task, but additional processes may also be at play. For instance, the negative effects observed might suggest a negative spillover process, by which increasing some mitigation actions (e.g., policy support, social media sharing, etc.) could have decreased other mitigation actions (e.g., contributing to tree planting). Given that the tree-planting task was also the last outcome variable completed by participants, such a process could be plausible. However, each of the first three outcomes (i.e., climate belief, climate policy support, and information-sharing willingness) was positively associated with the last outcome (i.e., WEPT; fig. S2 and tables S13 to S15). These positive associations at the study level also held within each of the 12 conditions (tables S16 to S18). That is, the more a participant supported climate policy, the more trees they planted, a pattern found under each condition (table S17). Similarly, participants who were willing to share climate information on social media also planted more trees, again a pattern found under each condition (table S18). These positive associations are more consistent with a positive spillover.
An alternative explanation for the intervention effects on the tree-planting task could be that current behavioral science theories and their corresponding interventions are more effective at targeting conceptual processes compared to more effortful and time-consuming behavioral signatures, especially in such a heterogeneous global sample. However, another explanation could be that interventions that made the negative consequences of climate change more salient (e.g., negative emotions, decreasing in psychological distance, and future self-continuity), triggered the perception that individual-level solutions (e.g., planting trees) may be futile in the face of such an insurmountable phenomenon, in line with the learned helplessness hypothesis. On the other hand, perhaps, a combination of these explanations gave rise to the effects observed. Future research is needed to clarify these processes and identify interventions that increase more effortful climate actions around the world, as well as actions that are more effective solutions to the climate crisis.
Last, while, in this global study, we tested the effects of several theoretically derived behavioral interventions on people’s beliefs and actions in the context of climate change, our findings provide meaningful insights to the broader fields of social and behavioral sciences. For instance, the average global effects of the interventions tested ranged from effectively zero to very small in the conceptual outcomes (beliefs and policy support) and near zero to negative in the behavioral outcome (tree planting). These findings point to critical limitations in these theories’ utility and generalizability beyond the contexts in which they were developed. The most extreme example is the correcting pluralistic ignorance intervention, which had no effect on beliefs, policy support, or willingness to share information on social media and even reduced tree-planting efforts. Theories are often tested and evaluated mainly on their ability to account for decontextualized patterns of data in laboratory settings, rather than their ability to help solve societal problems. In response to this limitation, researchers have recently proposed reverting the scientific paradigm to an impact-oriented theoretical and empirical research agenda.
The small effect sizes we observed in this global sample might also be partly interpreted through the lens of recent work reporting that over 60% of studies in the most prestigious journals in psychology have only focused on 11% of the world’s population. In our data collected in the United States or other WEIRD nations, the effects of the top interventions on belief and policy support were much stronger than at the global level. The skewed representation in the field may pose another notable obstacle in addressing societal problems that depend on global cooperation and a diversity of solutions for different cultural contexts, as is the case in climate change among numerous others global crises. One promising solution to these generalizability and practicality limitations in the behavioral sciences relies on embracing international collaborative science. Large global scientific projects can benefit from access to not only a wider range of populations but also from a diversity of scientific perspectives. For example, crowdsourcing has been found to improve the quality of scientific investigations by promoting ideation, inclusiveness, transparency, rigor, and reliability among other factors. Thus, crowdsourcing decisions related to the experimental design from experts more widely representative of the global scientific community might increase the impact and generalizability of scientific investigations. For example, the crowdsourcing of the theories tested from our large international team has led us to include less established interventions, such as “letter to future generation,” which ended up being one of the top interventions tested. Future work could also consider extending this crowdsourcing paradigm to include nonexperts (e.g., lay audiences), as recent work suggests that there may be unique benefits (e.g., increased interdisciplinarity), sometimes even producing research questions that outperform experts’ suggestions. Last, combining this “many labs” approach with the megastudy approach promises to push the limits of conventional scientific practices and overcome some of the main barriers of science generalization and implementation.
Overall, we tested the effectiveness of 11 expert-crowdsourced behavioral interventions at increasing climate awareness and action in 63 countries. Our findings provide theoretical support for many of the tested interventions. However, variation in effectiveness across outcomes, between countries, and along the spectrum of climate beliefs, suggests substantial gaps in our current theoretical understanding of climate change behavior. Moreover, the high preexisting levels of belief and policy support, alongside the small effect sizes observed here, raise critical questions about the practical capacity to facilitate bottom-up change at a global level, suggesting that top-down change might need to be prioritized to achieve the emissions reduction necessary to stay within safe planetary limits for human civilization. Practically, these findings provide critical information to policymakers considering climate solution implementations, streamlining the behavioral sciences’ response to the climate crisis.
Participants
The data were collected between July 2022 and July 2023 (see “note added in proof”). A total of 83,927 completed the study. Of them, 59,440 participants (Mage = 39.13, SDage = 15.76; 50% women and 46% men) from 63 countries who passed the two attention checks (i.e., “Please select the color “purple” from the list below” and “To indicate you are reading this paragraph, please type the word sixty in the text box below”) and correctly completed the WEPT demo were included in the analyses. Although removing participants who failed these preregistered attention checks risks contributing to a selection bias in the sample, we a priori determined we would screen participants according to these criteria to ensure data quality.
Fig. 5.. The number of participants in each of the 63 countries represented in the sample (Ntotal = 59,440).
Ethics approval was obtained independently by each data collection team from their corresponding Institutional Review Board. Only datasets submitted along with Institutional Review Board approval were included in the analysis.
Collaboration procedure
Following procedures from Van Bavel and colleagues, the organizational team submitted a call for collaboration (https://manylabsclimate.wordpress.com/call-for-collaboration/) in November 2021 on social media (i.e., Twitter), via personal networks and by posting on various mailing lists. We asked researchers from around the world to join our project by contributing in one of three ways: (i) collecting data (i.e., >500 responses) from a country in which data had not already been collected, (ii) propose an intervention that becomes included in the final study, and/or (iii) fund data collection (i.e., >500 responses) from a country in which data had not already been collected and support a local team who lacks funding. The collaborators who proposed an intervention were asked to keep in mind time constraints (i.e., each intervention had to take on average at most 5 min) and the targeted outcome variables (i.e., climate beliefs, policy support, social media sharing, and tree-planting contributions). We received a total of 36 proposed interventions, which were coded by the first authors (who were blinded to the intervention authors). The coding procedure involved screening the proposed interventions for feasibility in an international context, relevance for the dependent variables, and theoretical support from prior work (quantified by previously reported effect sizes). We also aggregated similar interventions and duplicates. Following this procedure, we identified 11 unique and feasible interventions, which we then asked all collaborators to rate on perceived efficacy (practical support) and theoretical value (theoretical support), initially aiming to select the top five interventions. We obtained 188 responses from our collaborators in January 2022 (fig. S5). Given high support for all interventions, we decided to test all 11 interventions in the main study. We then contacted the collaborators whose interventions had been selected to be included in the main study, to coordinate the intervention implementation and programming on the Qualtrics survey platform (www.qualtrics.com/). After obtaining the programmed interventions, we gave our collaborators feedback on their submissions and allowed them time to address our comments. After receiving the revised interventions, we contacted expert researchers who had published theoretical work relevant to each intervention, asking them to critically review each intervention’s implementation. For example, Jost reviewed the system justification intervention, and van der Linden et al. reviewed the scientific consensus intervention. This process was iterated for each of the 11 interventions. After receiving critical suggestions from these experts, we engaged in another round of revisions. Last, in an attempt to reduce American-centric researcher biases, we asked all collaborators from around the world for additional feedback on the entire survey, including all interventions, demographics, and independent variables. This process lasted until the end of May 2022, when we started piloting the final version of the study, on a sample of 723 participants (Mage = 43.6; SDage = 15.7; 52% women and 46% men, <2% nonbinary), collected in the United States. Using the pilot data, we wrote our analysis scripts and the preregistration (available at https://aspredicted.org/blind.php?x=W83_WTL). After the piloting was completed (July 2022), we sent our collaborators the final version of the study in Qualtrics along with an in-depth instructions manual (https://osf.io/ytf89/files/osfstorage/6454f8e3b30b49156cb9dd79/) on how to translate and adapt the study to each country. We also instructed our collaborators to obtain ethics approval from their International Review Boards before launching data collection. All collaborators were given 10 months (until May 2023) to submit their data.
Experimental design
Participants signing up to complete the study (expected to take 15 min to complete) were first asked to read and sign the informed consent. They were then exposed to the first attention check (“Please select the color “purple” from the list below. We would like to make sure that you are reading these questions carefully.”), which removed from the experiment any participants choosing an incorrect answer. Then, participants were then given a definition of climate change: “Climate change is the phenomenon describing the fact that the world’s average temperature has been increasing over the past 150 years and will likely be increasing more in the future.” After reading this definition, participants were randomly assigned to 1 of 12 conditions: 11 experimental interventions or a no intervention control condition, in a between-subjects design. Participants under the control condition were then exposed to a short, thematically unrelated text from the novel Great Expectations by Charles Dickens, while participants under the experimental conditions were exposed to an intervention. Then, all participants were directed to the outcome variable phase, in which they rated (in random order) their (i) climate beliefs, (ii) climate policy support, and (iii) willingness to share climate information on social media. Last, participants were given the chance to contribute to tree-planting efforts by completing the WEPT. Then, participants under the control condition were asked to complete an additional set of variables. Last, all participants were asked to fill out a series of demographic variables, which included another attention check (“In the previous section, you viewed some information about climate change. To indicate you are reading this paragraph, please type the word sixty in the text box below.”). Notably, participants filled out the entire survey in the primary language of their country of residence.
Climate beliefs
Climate beliefs were measured by participants’ answer to the question “How accurate do you think these statements are?” from 0 = not at all accurate to 100 = extremely accurate. The four statements were as follows: “Taking action to fight climate change is necessary to avoid a global catastrophe,” “Human activities are causing climate change,” “Climate change poses a serious threat to humanity,” and “Climate change is a global emergency.” The Cronbach’s alpha measure of internal consistency of this four-item scale in this dataset was 0.934.
Climate policy support
This dependent variable consisted of participants’ level of agreement from 0 = not at all to 100 = very much so, with the following nine statements: “I support raising carbon taxes on gas/fossil fuels/coal?,” “I support significantly expanding infrastructure for public transportation,” “I support increasing the number of charging stations for electric vehicles,” “I support increasing the use of sustainable energy such as wind and solar energy,” “I support increasing taxes on airline companies to offset carbon emissions,” “I support protecting forested and land areas,” “I support investing more in green jobs and businesses,” “I support introducing laws to keep waterways and oceans clean,” and “I support increasing taxes on carbon intense foods (for example meat and dairy).” The Cronbach’s alpha measure of internal consistency of this nine-item scale in this dataset was 0.876.
Social media sharing
Participants were first presented with the text, “Did you know that removing meat and dairy for only two of three meals per day could decrease food-related carbon emissions by 60%? It is an easy way to fight #ClimateChange #ManyLabsClimate${e://Field/cond} source: https://econ.st/3qjvOnn” (where “{e://Field/cond}” was replaced with the condition code for each group). Participants were then asked “Are you willing to share this information on your social media?,” the answer options being “Yes, I am willing to share this information,” “I am not willing to share this information,” and “I do not use social media.” Participants who indicated that they do not use social media were excluded from this analysis (i.e., a third of the sample). Moreover, participants were asked to indicate the platform (e.g., Facebook, Twitter, and Instagram) on which they posted the information.
WEPT tree-planting efforts
To measure an action with a real-world impact performed at an actual cost to participants, we used a modified version of the WEPT. This task is a multitrial web-based procedure that detects consequential proenvironmental behavior by allowing participants the opportunity of engaging in voluntary cognitive effort (i.e., screen numerical stimuli) in exchange for donations to an environmental organization. This measure has been validated and has been found to correlate with well-established scales for the assessing proenvironmental behavioral intentions (e.g., general ecological behavior scale) and with direct donation behaviors (e.g., the donation of a part of their payment to an environmental organization).
Participants were first exposed to a demonstration of the WEPT, in which they were instructed to identify all target numbers for which the first digit is even and the second digit is odd (4 of 18 numbers were target numbers on the demonstration page). Participants were not allowed to advance the page until they correctly completed the WEPT demonstration. Then, they were told that planting trees is one of the best ways to combat climate change and that they would have the opportunity to plant up to eight trees if they chose to engage in additional pages of the item identification task (one tree per page of WEPT completed). These pages contained 60 numbers per page, where participants had to screen for target numbers. Alongside these instructions, participants were shown a pictogram of eight trees, one of which was colored green to mark their progress in the task. Participants were allowed to exit the task at any point with no penalty.
Demographics
Participants were asked to indicate their gender, age, education level, political orientation for economic and social issues, and household income.
Working-together norms
The working-together norms intervention was submitted by M. Vlasceanu and J. Van Bavel. This intervention was adapted from Howe et al., and it combines referencing a social norm with an invitation to work with others toward a common goal. This working-together normative appeal invites people to join in and “do it together” and has been found to increase interest in and actual charitable giving, reduce paper-towel use in public restrooms, and increase interest in reducing personal carbon emissions. Mediation analyses in prior work also suggested that working-together normative appeals are effective because they foster a feeling in participants that they are working together with others, which can increase motivation while reducing social pressure. Participants under this condition were exposed to a flier adapted from Howe and colleagues, after which they were asked 15 questions about the flier, serving as manipulation checks that were also meant to reinforce the manipulation [e.g., “If you are taking steps toward reducing your carbon footprint, to what extent would you feel like you are doing so together with other Americans (or participants’ group, adapted for each country)?” on a scale from 0 = not at all to 100 = extremely or “How strongly do you identify with your fellow Americans (or participants’ group, adapted for each country)?” on a scale from 0 = not at all to 100 = extremely].
System justification
SThe sstem justification intervention was submitted by O. Buchel, M. Tyrala, and A. Findor. This intervention is situated at the intersection of social identity, collective narcissism, and system justification approaches [based on] and consists of framing climate change as uniquely threatening the way of life of participants’ nationality (e.g., the American way of life). Participants were asked to read a text emphasizing the importance of nature and the environment to one’s life [e.g., “ the food you eat, the sports you enjoy, the customs you observe, how you spend your free time, or even how you imagine growing old, all are likely impacted by where you live”], followed by examples of the effect of climate change on the local environment of participants’ nation [e.g., “ we can already see the consequences of climate change in the United States. For example, floods are becoming more and more frequent, putting a quarter of Americans at risk of losing their homes. Similarly, wildfires are becoming more frequent and more intense, threatening millions of Americans.”]. The text ends with an appeal to being proenvironmental as a patriotic gesture that will protect one’s way of life (e.g., “Being proenvironmental allows us to protect and preserve the American way of life. It is patriotic to conserve the country’s natural resources. It is important to protect and preserve our environment so that the United States remains the United States.”). This narrative was also intertwined with representative images of participants’ country of residence.
Binding moral foundations
The binding moral foundations intervention were submitted by B. Douglas and M. Brauer. This intervention relies on evoking ingroup-loyalty and authority moral foundations, which has been shown to increase support for proenvironmental behavior and attitudes. Participants were asked to read the following text “We are Americans (or participants’ nationality, adapted for each country). This means we can rise to any challenge that faces our country. From scientists to experts in the military, there is near universal agreement that climate change is real. The time to act is now. Using clean energy will help to keep our air, water, and land pure. It is the American (or participants’ nationality, adapted for each country) solution to the climate crisis,” after which they were exposed to an image of a person holding the national flag of participants’ country of residence.
Exposure to effective collective action
The exposure to effective collection action intervention was submitted by E. Shuman and A. Goldenberg. This intervention features examples of successful collective action that have had meaningful effects on climate policies, building on prior work showing that exposure to nonviolent action can increase willingness to join and maintain support. In addition, prior work also found that highlighting the possibility of making real concrete changes through collective action can increase hope, efficacy, and collective action. Participants were exposed to a text explaining that the impact people’s actions can have on curbing the effects of climate change, citing research indicating there is still “a window of opportunity” to make a difference. Then, participants were informed that the effectiveness of people’s actions to fight climate change depends on their ability to “come together and demand systemic change.” Participants were then exposed to several successful examples in which people solved global issues, such as the restoration of the ozone layer in 1987. Then, participants were exposed to examples of climate activism initialized by individual people and leading to large-scale movements or policy implementation (e.g., protests by locals from the American Midwest against fossil fuels pressured the governors of Illinois, Indiana, Michigan, Minnesota, and Wisconsin to build a new network for charging electric vehicles.). Images of concepts described in the text were displayed throughout.
Future self-continuity
The future self-continuity intervention was submitted by V. Ponizovskiy, L. Grigoryan, S. Grelle, and W. Hofmann. This intervention consists of emphasizing the future self that has been found in prior work to motivate future-oriented behaviors, such as academic performance, ethical decision making, and proenvironmental behavior (–). Participants were asked to read a text emphasizing the importance of engaging in climate action [i.e., “If no changes are made, the average temperature can increase by up to 6.5°C (12 F) by the year 2100 (IPCC, 2022). This would be extremely dangerous as super hurricanes, gigantic wildfires, and extreme food, and water shortages would become commonplace.”]. They were then presented with a series of causes for this phenomenon (i.e., “Human behaviors like energy production from fossil fuels, excessive meat consumption, and car driving increase the concentrations of greenhouse gasses in Earth’s atmosphere. Over 90% of the increase in the world’s temperature is caused by human activity.”). Last, participants were asked to imagine that their 2030 self is writing a letter to their present self, in which their future self is describing the actions they would have wanted to take regarding climate change [i.e., “Please put yourself in the year 2030—8 years from now. Take a few moments to imagine your life in that future. Imagine how you will look, where you will be, and who you are with. In the year 2030, it will be clear whether keeping climate change under 2°C is still possible. It will be clear whether the necessary change occurred fast enough to match the speed of the changing climate. As the Earth’s atmosphere continues to heat up, the effects of climate change will be more apparent: The “highest observed temperature” records will keep being updated, heatwaves and the draughts will become more common, species will continue to become extinct. Now please write yourself a “letter from the future.” This should be a letter you are writing in the year 2030, to your past self. As the person that you will be in 2030, what role would you think would be appropriate for you in respect to climate change? What would you want to tell yourself in the past? What would you like your past self to do? Please spend a bit of time on this task and try to write at least 100 words (five sentences), or more, if possible.”].
Scientific consensus
The scientific consensus intervention was submitted by A. van Stekelenburg, C. Klöckner, S. Vesely, and D. Bleize. This intervention consists of a message suggesting that climate scientists are in agreement with each other that climate change is real and primarily caused by human action. This messaging has been found to increase people’s belief in climate change and support for climate mitigation policy. Participants were exposed to the following text “Did you know that 99% of expert climate scientists agree that the Earth is warming and climate change is happening, mainly because of human activity (for example, burning fossil fuels)? [Myers et al., Environmental Research Letters; Lynas et al., Environmental Research Letters; Doran and Zimmerman, EOS]”. The text was accompanied by a pie chart with 99% of the surface area shaded.
Decreasing psychological distance
The decreasing psychological distance intervention was designed by S. Chamberlain, D. Hine, and G. Huang. This intervention is based on prior work finding that many perceive climate change as psychologically distant (i.e., “as a set of uncertain events that may occur far in the future, impacting distant places and affecting people dissimilar to themselves”). Thus, framing climate change as a psychologically proximal risk issue (e.g., geographic) is expected to reduce psychological distance and increase public engagement. Participants were exposed to a paragraph emphasizing the impact of climate change (i.e., “There is no doubt that humans are the main driver of climate change. Human influence has warmed the atmosphere, ocean, and land. Climate change is already affecting every region across the world. It has resulted in more frequent and intense extreme weather events, causing widespread harm and damage to people, wildlife, and ecosystems. Human systems are being pushed beyond their ability to cope and adapt.”). They were then exposed to two examples of recent natural disasters caused by climate change in participants’ region (e.g., U.S. participants will be exposed to information about the 2021 record-breaking heat wave in North America causing the Lytton wildlife and to information about the 2017 Hurricane Harvey in Texas and Hurricane Irma in Florida, killing 232 people and causing $175 billion in damage). Participants were then asked to select the aspects of their lives affected by climate change from a list including: food production, farming and crop production, health and wellbeing, infectious disease, heat related harm and deaths, lack of, mental health issues, flooding and storms, changed land, freshwater and ocean environments, damaged infrastructure, and economy. After making the selections, participants were provided the correct answers based on current scientific estimates (i.e., all the possible options). Last, participants were asked to write about how climate change will affect them and their community (i.e., “Please write in a few sentences: How those climate consequences will affect you, your friends and family, and your community. Try to imagine these things happening today so you can be specific and describe what it will be like.”).
Dynamic social norms
The dynamic social norms intervention were submitted by O. Genschow, D. Loschelder, G. Sparkman, and K. C. Doell). This intervention is based on work showing that dynamic norms (i.e., how other people’s behavior is changing over time) are even more impactful at changing behavior than static social norms. Participants in this intervention first read a paragraph emphasizing that “People in the United States and around the world are changing: More and more people are concerned about climate change and are now taking action across multiple fronts,” accompanied by an image featuring relevant data in support of this claim. Then, participants were given examples of actions people are starting to take to mitigate the changing climate [i.e., “Since 2013, concerns about climate change have increased in most countries surveyed. What kinds of actions are people taking right now? More than ever before, people are making changes to their lifestyles, supporting policies to address climate change, and are giving the issue more time and attention. For example, more and more people from around the world are now cutting back on personal consumption, especially meat and dairy products, spending time, effort, and money on initiatives to mitigate climate change (for example, planting trees, offsetting carbon emissions), switching to low carbon modes of transportation (for example, taking bicycles). There’s also been a notable increase in support for climate change mitigation policy—some of the most popular policies include attempting to conserve forests and land, transitioning to solar, wind, and other renewable energy sources, creating/raising carbon taxes on fossil fuels, coal, gas, etc.”].
Correcting pluralistic ignorance
The correcting pluralistic ignorance intervention was submitted by M. Schmitt, A. Lutz, and J. Lees. This intervention builds on work reporting that people substantially underestimate the climate change concern of others, a phenomenon labeled as “pluralistic ignorance”. Accordingly, collective action might be limited by people’s misperception that not many people are concerned. This intervention presented real public opinion data, which show that majorities around the world are concerned about climate change. Participants were first asked to predict the percent of people in their country who hold the belief that climate change is a global emergency [i.e., Researchers recently conducted the “People’s Climate Vote,” which is the World’s largest survey of public opinion on climate change (“global warming”). A total of 1.2 million people completed the survey from 50 different countries around the globe. The survey included people from the United States. Think for a moment about Americans and their views on climate change. How many Americans do you think would agree with the statement “Climate change is a global emergency”?]. After providing a prediction, participants were shown the actual percentage of people in their country who hold the belief in question, according to the Peoples’ Climate Vote. For example, participants in the United States will be told that “The People’s Climate Vote found that 65% of Americans agree that climate change is a global emergency”. For countries where the People’s Climate Vote does not report national level results, participants were presented with the climate opinion of people in their region.
Letter to future generations
The letter to future generations intervention was submitted by S. Syropoulos and E. Markowitz. This intervention involves writing a letter to a member of the future generation, which has been shown to reduce the psychological distance between one’s current choices and their consequences on future generations. Participants were asked to write a letter to a child who will read it in the future [i.e., “Please think of a child that is currently less than 5 years old Now imagine that child is a 30-year-old adult. It is approximately the year 2055, they have started a family of their own, and they are finding their own way in the world. Whether they recognize it or not, they live in a world that is powerfully shaped by the decisions we are all making now, in 2022. One day, they find a letter written today, in 2022, which is a message from you.”]. In this letter, participants are encouraged to write about their actions toward ensuring an inhabitable plant (i.e., “In it, you tell this family about all of the things you have done and want to do in the future to ensure that they will inherit a healthy, inhabitable planet. You tell them about your own personal efforts—however small or large—to confront the complex environmental problems of your time, from habitat loss to water pollution to climate change. In this letter, you also tell this family in 2055 about how you want to be remembered by them and future generations as someone who did their best to ensure a safe, flourishing world.”). Participants were allowed to write for 3 min and encouraged to write at least 100 words or 5 sentences.
Negative emotion
The negative emotion intervention was submitted by K. Doell and C. Pretus. This intervention involves exposure to scientific facts regarding the impacts of climate change in a doom and gloom messaging style typically used by climate communicators to induce negative emotions as a way of stimulating mitigation behaviors. Participants were first asked to report their baseline levels of emotions related to climate change, (e.g., hopeful, anxious, depressed, scared, indifferent, angry, helpless, and guilty). They were then exposed to information about the consequences of climate change alongside representative images [e.g., “Climate change is happening much more quickly and will have a much greater impact than climate scientists previously thought, according to the latest report by the Intergovernmental Panel on Climate Change (IPCC, 2022). If your anxiety about climate change is dominated by fears of starving polar bears, glaciers melting, and sea levels rising, you are barely scratching the surface of what terrors are possible, even within the lifetime of a young adult today. And yet the swelling seas—and the cities they will drown—have so dominated the picture of climate change/global warming that they have blinded us to other threats, many much closer at hand and much more catastrophic ”]. Last, participants were asked to report their levels of emotions related to climate change again.
Control condition
Participants in the control condition were instructed to read a text retrieved from the novel Great Expectations by Charles Dickens [i.e., “As soon as the great black velvet pall outside my little window was shot with gray, I got up and went downstairs; every board upon the way and every crack in every board calling after me (…) I took it in the hope that it was not intended for early use and would not be missed for some time.”]. Participants were required to spend at least 10 s reading this text. This was to ensure that participants exerted some level of cognitive effort before being exposed to the dependent variable phase, to mirror the experience of participants in the experimental conditions. We chose a fiction text to prevent priming participants in any relevant way that could influence the dependent variables. After reading the excerpt, participants under the control condition were directed to the dependent variable phase, followed by the demographics phase. Last, participants under the control condition were also directed to an additional independent variable phase, exclusive to participants under this condition.
Additional variables collected
These variables were only displayed to participants under the control condition, after they completed all dependent variables. First, participants were asked to rate the competence of climate scientists (“On average, how competent are climate change research scientists?” on a scale from 0 = not at all to 100 = very much so), their trust in scientific research about climate change (“On average, how much do you trust scientific research about climate change?” on a scale from 0 = not at all to 100 = very much so), their trust in their government (“On average, how much do you trust your government?” on a scale from 0 = not at all to 100 = very much so), their attitudes toward human welfare (“To what degree do you see yourself as someone who cares about human welfare?” on a scale from 0 = not at all to 100 = very much so), their global citizenship identity (“To what degree do you see yourself as a global citizen?” on a scale from 0 = not at all to 100 = very much so), their environmental identification (e.g., “To what degree do you see yourself as someone who cares about the natural environment?” on a scale from 0 = not at all to 100 = very much so), and their extrinsic environmental motivation (e.g., “Because of today’s politically correct standards, I try to appear proenvironmental.” on a scale from 0 = strongly disagree to 100 = strongly agree). Then, they were asked to estimate the percentage of people in their country who believe that climate change is a global emergency.
Statistical methods
Our dependent variables have distributional properties (fig. S6) that preclude unbiased estimation with common, off-the-shelf, regression tools (such as the preregistered analyses). To address this, estimates presented in Fig. 2 relied on Bayesian methods and custom likelihood functions. Full mathematical descriptions of all models can be found in the supplied code (https://github.com/josephbb/ManyLabsClimate). Additional analyses can be found at https://github.com/mvlasceanu/ClimateTournament.
Belief was estimated using a hierarchical Zero-One-Inflated Beta (ZOIB) model. This model was further used to derive adjusted participant-level estimates of preintervention belief, to avoid postintervention bias in subsequent models. Sharing on social media was evaluated with a logistic regression. For WEPT, we used a geometric regression with a customized likelihood function to account for truncation and overinflation for the maximum number of trees planted. Priors were selected using prior-predictive simulation, with model structure iteratively developed through analysis of the prior predictive distribution and validated through model comparison using posterior predictive simulation. Posteriors were sampled using a No-U-Turn Sampler implemented on a graphics processing unit (GPU) with PyMC/NumPyro.
We note that these modeling choices are different from our preregistered analysis, which specified linear (belief and policy), ordinal (WEPT), and logistic (sharing) mixed-effects models. Plots of residuals from preregistered models suggested moderate to severe violations of distributional assumptions. For this reason, P values and estimates of effect sizes for these models may be unreliable. Despite these issues, we note that the findings from preregistered analyses are qualitatively similar to those from the Bayesian analyses. Overall, similarities between the preregistered and Bayesian analyses suggest effects that are remarkably robust to analysis decisions.
For completeness, we include the results as preregistered in tables S9 to S12 and fig. S1. Belief and policy support were modeled using a linear mixed-effects model with climate policy support as the dependent variable, condition as the fixed effect, including item (nine policies), participant, and country as random effects. WEPT was modeled using an ordinal mixed-effects model with climate action (WEPT) as the dependent variable and condition as the fixed effect, including country as random effects. Sharing was modeled using an ordinal mixed-effects model with climate action (WEPT) as the dependent variable and condition as the fixed effect, including country as random effects.
To develop and evaluate our Bayesian models, we adapted an established Principle Bayesian Workflow. This process begins by identifying inference goals, domain knowledge, and features of the dataset. Candidate statistical models are proposed, with prior predictive checks that are used to identify reasonable priors. Data are simulated from the prior predictive distribution, and the statistical model is fit to this simulated data. This allows for evaluation of computational properties of the model, tuning of the sampler, adjustment of the model or priors, and refinement. Key insight was gained through visual inspection of the posterior z-score versus posterior contraction, which can indicate issues with overfit, underfit, bad prior models, or poorly identified model specification. This process was iterated on until a suitable candidate model and priors were identified. Last, posterior predictive checks were used to verify that models adequately reconstructed broad properties of the data without regard to the estimands of interest (i.e., country/treatment effects). Failures here lead to adjustment of the underlying model. Once all model development criteria were satisfied, final analysis of the dataset was used to generate estimates of treatment and country level effects as well as all relevant figures. We note that priors for similar parameters across models may differ as a result of this iterative process, owing to distinct link functions and differing computational constraints. However, the impact of the prior on posterior samples is unlikely to be meaningful, given the volume of data.
We fit the selected model to the study data using PYMC with a No U-Turn Sampler implemented on the GPU in NumPyro. We evaluated the model fit, ensuring the absence of divergent transitions, sufficient mixing of the (four) Markov chains, a large enough effective sample size, and an acceptable Estimated Bayesian Fraction of Missing Information. Last, data were simulated from the posterior distribution and visual inspection of these posterior retrodictive checks that were used to assess model fit. Sampling parameters were largely default and can be found in the supplied code.
Belief
Belief was indicated for four items on a scale from 0 to 100, inclusive. We scaled the outcome variable for each item to 0 to 1 to facilitate the use of common bound distributions. However, as both 0 and 1 were possible values, our likelihood function needed to account for possible inflation. Hence, we implemented a hierarchical ZOIB regression. We developed a generative model in which participants were estimated to have an unobserved preintervention belief, defined by their observed belief minus the estimated preintervention effect for their level of belief (i.e., as though they had been in the control condition) that was partially pooled by country, which, in turn, was partially pooled via a hyperparameter for average belief. Interventions were modeled with an intercept, corresponding to the average effect, and an effect of the estimated preintervention belief. The intervention effect and intercept for the control condition were fixed at zero. Otherwise, we modeled intervention effects using a multivariate normal distribution to account for covariance between intercepts and interventions. Further, we included partially pooled intercepts for item-specific effects. Where necessary, noncentered parameterizations were used to improve model fit.
Last, we extracted the posterior average preintervention belief for each participant to use in modeling policy support, social media sharing, and WEPT. This reflects the observed level of belief after adjusting for intervention effects on belief. As the treatment effects are small, these adjustments are minimal. Ideally, one would jointly model belief and other outcomes; however, the large sample sizes inherent to a megastudy impose computational constraints, a particular issue with model development and evaluation. Extracting intervention-adjusted estimates of initial belief enables us to examine heterogeneous intervention effects for each of these outcomes, at a tractable degree of model complexity. We chose to focus on belief for evaluating heterogeneous intervention effects under the assumption that belief is more likely to be a cause of support for policy, social media sharing, and investment in tree-planting activities than a consequence. Full mathematical descriptions of the model can be found in the supplied code.
Policy support
Support for policy was indicated for nine items on a scale from 0 to 100, inclusive. Because of computational constraints with the full dataset, we examined the average of these items. As with belief, this outcome was scaled from 0 to 1, and a ZOIB was used to model the data. Policy support was modeled with an intercept, an effect of adjusted belief, with intercept and belief effects modeled for interventions and countries. Intervention and country effects were modeled as separate zero-centered normal distributions.
Social media sharing
Sharing was a binary outcome, restricted to users who used social media. To analyze the impact on sharing, we relied on a Bayesian logistic regression. The probability of sharing was modeled with an intercept, an effect of adjusted belief, with intercept and belief effects modeled for interventions and countries. Intervention and country effects were modeled as separate zero-centered normal distributions.
Work for environmental protection task
Participants were able to plant between one and eight trees. We began by modeling this as a truncated geometric distribution, assuming that participants have a per–time step chance of giving up and are forced to stop at 8. However, we noticed an overabundance of planting eight trees consistent with some participants committing to planting all eight. Accordingly, we modified our likelihood to include inflation at eight trees. Posterior predictive fits confirmed adequate model fit. With this likelihood, we constructed a Bayesian hierarchical with an intercept, an effect of adjusted belief, and intercepts and belief effects modeled for interventions and countries.
Note added in proof: After this manuscript was accepted for publication, the authors alerted the editorial office to a paper they recently finalized that includes data used in this paper. This data can be found at: K. C. Doell, et al. The International Climate Psychology Collaboration: Climate change-related data collected from 63 countries. (2024). https://doi.org/10.31234/osf.io/7fy2g