Anchoring the Experts: Using Vignettes to Compare Party Ideology Across Countries - Sciences Po Access content directly
Journal Articles Research & Politics Year : 2014

Anchoring the Experts: Using Vignettes to Compare Party Ideology Across Countries

Ryan Bakker
  • Function : Author
Seth Jolly
Jonathan Polk
  • Function : Author
Marco Steenbergen


Expert surveys are a valuable, commonly used instrument to measure party positions. Some critics question the cross-national comparability of these measures, though, suggesting that experts may lack a common anchor for fundamental concepts such as economic left–right. Using anchoring vignettes in the 2010 Chapel Hill Expert Survey, we examine the extent of cross-national difference in expert ideological placements. We find limited evidence of cross-national differences; on the whole, our findings further establish expert surveys as a rigorous instrument for measuring party positions in a cross-national context. Scholars of party politics need to measure the positions parties take on a variety of policy domains. They have developed a diverse set of data collection tools – including, most notably, manifestos, roll call data, voter judgment, and expert surveys – in response to this challenge. While different sources of data may be appropriate for different kinds of questions related to party positions, we argue that expert surveys offer several advantages over other alternatives. Expert surveys allow researchers to obtain positions for many different types of parties across a range of contexts (see, for example, Benoit and Laver, 2006; Hooghe et al., 2010; Rohrschneider and Whitefield, 2009). Expert surveys’ virtues include that they are inexpensive to administer, that they draw on broad knowledge about parties by tapping into information about what parties say and do, and that they allow for a high degree of flexibility as researchers can gather information on any topic for which there are enough competent experts (Marks et al., 2007). As a result, expert placements of parties are widely used within political science today. Yet important questions pertaining to the expert evaluations remain. For instance, Budge (2000) is concerned that expert-specific differences contribute to errors in the data, including, for example, expert political preferences leading to biased placements of right-wing parties (Curini, 2010) or knowledge differences among experts contributing to overly centrist placements (Gemenis and Van Ham, 2014). Here, we focus on a separate but related question that concerns expert survey data: whether expert placements of parties are comparable across countries. Put differently, does an expert on the French or Swedish party system think of concepts such as left and right the same way as an expert on British parties when both are asked to place the parties of their respective countries on an economic left–right scale? As Benoit and Laver (2007: 94) argue, “what experts do have in mind when they talk about left and right, in terms of substantive policy dimensions, varies in intuitively plausible ways from country to country.” The political space underlying the substantive policy dimension may differ across individual experts, countries or regions. Benoit and Laver (2007) stress the importance of comparing expert survey-based estimates of party positions with other independent left–right scales. However, even if expert-based estimates correlate highly with other measures of left–right, these correlations do not necessarily mean that the expert surveys are free of expert, country, or other context-specific bias in their responses. Thus, we need to examine the cross-national comparability of expert placements. Research on European Parliament (EP) party group formation and durability provides a specific example for when this cross-national comparability is necessary. As EP power grows relative to other EU institutions (Hix and Høyland, 2013), it becomes increasingly important to understand party behavior in this supranational legislature. National parties join party groups in the EP that are composed of parties with similar policy positions, and when parties switch group affiliation they do so in an attempt to minimize positional incongruence (McElroy and Benoit, 2010, 2012). Scholars of the EP therefore require measurements of party positions that they can confidently assert are cross-nationally comparable. Survey vignettes, which are designed to identify and ultimately correct situations when survey respondents interpret identical questions differently, provide a method of directly measuring the incomparability of responses to survey questions with ordinal response categories (King and Wand, 2007; King et al., 2004). The 2010 Chapel Hill Expert Survey (CHES) implemented this method by presenting country specialists with three hypothetical vignette parties described by short vignettes. Experts from all survey countries placed these vignette parties on an 11 point economic left–right scale, which allows the analyst to identify potential incomparability in the experts’ placement of the actual parties in the survey. With these vignettes, we can assess, for example, whether a 7 in Sweden is a 7 in the United Kingdom. Our results indicate that experts across Europe are strikingly consistent in their ordering of the vignette parties. Although limited cross-national differences in measurement do exist, taken on the whole, our findings are consistent with those showing that expert placements provide reliable and valid measures of left–right party positions (Benoit and Laver, 2007; Hooghe et al., 2010). Our analysis shows that most experts included in this sample use “left–right” in a similar way. These findings are thus an important step forward in establishing expert surveys as a rigorous instrument for measuring party positions over time and across countries. We begin the article by summarizing the anchoring vignettes approach, and then introduce the 2010 CHES. After briefly discussing the validity and reliability of the survey, we focus our attention on expert placements of the three vignette parties. We conclude by discussing the implications of our findings for the use of expert surveys in cross-national studies of political parties.
No file

Dates and versions

hal-02186416 , version 1 (17-07-2019)



Ryan Bakker, Erica Edwards, Jan Rovny, Seth Jolly, Jonathan Polk, et al.. Anchoring the Experts: Using Vignettes to Compare Party Ideology Across Countries. Research & Politics, 2014, pp.1 - 8. ⟨hal-02186416⟩
39 View
0 Download


Gmail Facebook X LinkedIn More