Corrections to: Field Experiments and the Study of Voter Turnout

The problem

The replication program computes each pooled estimate as a weighted average $\sum_j a_jd_j$ of the contrasts $d_j$, with weights $a_j$ proportional to $1/s_j^2$, where $s_j$ is the standard error of $d_j$. It takes the variance of this average to be $1/\sum_j s_j^{-2}$. That is the variance only if the weighted covariance terms sum to zero, as they do for uncorrelated contrasts. Several pooled contrasts are arms of one experiment that share a control group. The Hawthorne, Self and Neighbors mailings of Gerber et al. (2008), which p. 36 describes with one control condition, are three of the 29 social-pressure treatments. Four Kentucky treatments of Mann (2010) share a control group of 19,561 people and hold 32% of the weight of that pool. Two contrasts with a common control group both contain the sampling error of the control mean. For unscaled differences in means under independent sampling, their covariance is the variance of that mean, which is positive. The printed intervals omit these terms. How much the intervals' widths should change for the actual experiments cannot be recovered from the dataset, which gives marginal standard errors but no covariances; the size of the omission can be computed under stated models, as below.

Evidence an exact counterexample

The evidence is a derivation, two exact counterexamples to the program’s variance formula, and a recalculation from the replication files.

Reproduction. The pooling commands of the replication program receive an effect and its standard error for each treatment and no covariance. With the variance $1/\sum_j s_j^{-2}$, all nine estimates and intervals on pp. 31–33 reproduce to the printed precision. For the social-pressure pool the standard error is 0.0838.

Derivation. Let $\delta_j=d_j-\mathbb{E}[d_j]$ and $\Sigma_{jk}=\mathbb{E}[\delta_j\delta_k]$, so that $\Sigma_{jj}=s_j^2$. For fixed weights, I have

With $a_j=s_j^{-2}/\sum_i s_i^{-2}$, the first sum equals $1/\sum_j s_j^{-2}$. Let $\bar Y_j$, $\bar Y_k$ and $\bar Y_0$ be the mean outcomes of two treatment groups and of their common control group. If the three means are uncorrelated, then

Counterexamples. Consider six voters whose turnout is $(1,1,1,0,0,0)$ whatever they are sent, and assign two of them at random to each of a control group and two mail groups. Both true treatment effects are zero. Enumerating the 90 equally likely assignments shows that each contrast has variance $3/10$ and that the two contrasts have covariance $3/20$. Their average then has variance $9/40$, and the program’s formula gives $3/20$. In the second example, each voter in three independent groups of 100 votes independently with probability $1/2$. Each of the two contrasts has variance $1/200$, and their covariance is $1/400$. Their average has variance $3/800$, where the program’s formula gives $1/400$. By exact enumeration, the nominal 95% normal interval built from $1/400$ covers the true value zero with probability 0.8887. The examples show that the program’s formula can fail for contrasts with a common control group. They do not measure the coverage of the article’s intervals.

Recalculation. The dataset records the arm sizes for some groups of contrasts with a common control group. The groups with at least two contrasts in the social-pressure pool hold 50.3% of its weight, and the four Kentucky treatments hold 32.4%. I use the shared-control model’s correlations with the stored standard errors for these groups and take all other contrasts to be independent. For the social-pressure estimate, I obtain a standard error of 0.1014, which is 21% above 0.0838. With the Kentucky group alone, the standard error is 0.0997, which is 19% above. The three mailings of Gerber et al. (2008) hold 23.4% of the weight, each with $a\approx0.0780$. If their three covariances equal $c$, then they add $6a^2c\approx0.0365\,c$ to the variance. The dataset does not determine $c$, since its sample-size fields for these rows cannot be used (P042-E003), and the model treats the three contrasts as independent.

I also computed two upper bounds. At the narrower bound, each group of contrasts that appear to come from one experiment, but lack sizes, has the largest covariance that their standard errors allow, $s_js_k$. The broader bound also allows overlaps that the dataset leaves unresolved, among them a control group common to all seven Kentucky rows. At both bounds, different groups are independent. Neither bound is the covariance of the actual experiments.

The difference between the nonadvocacy and advocacy estimates is 0.407. Its variance is $b^{\mathsf T}\Sigma b$, where $b_j=a_j$ in the first pool and $b_j=-a_j$ in the second. No group of contrasts in the model or the bounds joins the two pools.

Both tables use the estimates as stored. I did not repeat the calculation with the estimates corrected in P042-N003, P042-N005 and P042-N006.

What survives

The nine pooled estimates, their counts and their ordering do not depend on the variance and are unchanged. Under the model and at both bounds, each of the nine intervals keeps its position relative to zero. The social-pressure and gratitude normal intervals stay above zero for every covariance matrix that is compatible with the stored standard errors, with minimum lower endpoints of 2.087 and 0.257 to three decimal places. The difference between nonadvocacy and advocacy mail stays significant at the 5% level under the model and at the narrower bound. At the broader bound it does not ($z=1.88$), which shows that the printed statement could fail and not that it does. What changes is the stated precision: the printed intervals use the independence-based variance and a normal approximation.

The problem

The article states that every mail estimate is an increase in turnout per mailing. Five of the 31 advocacy estimates are not on that scale. Rows 226–230 of the dataset are the differences in turnout between a group that was sent a whole programme of mailings and a control group that was sent none: nine mailings in the Connecticut mayoral race, and four or six in the New Jersey assembly race. For a fixed programme of $m$ mailings versus none, the per-mailing regression coefficient and its standard error are the programme contrast and standard error divided by $m$. The five rows therefore entered the pooling with effect magnitudes that are too large and with weights that are too small by the factors 81, 36, 16, 36 and 16. As stored, they hold 0.5% of the weight in the estimate for conventional mail, and on the per-mailing scale they hold 18.0%. The corrected estimate for conventional mail is 18.8% below the printed one.

Evidence a recalculation from the source

The evidence is a check of the source, a derivation and a recalculation from the replication files.

Source. Gerber, Green and Green (2003) write on p. 572 that “the four mailings increase turnout from 63.7 to 65.6%, or approximately 0.5 percentage-points per mailing.” On p. 573 the Connecticut treatment group “received nine pieces of direct mail”. Their Table 3 reports differences in turnout of about 1.9, 0.0, 0.4 and 1.0 points for the groups in New Jersey that were sent 4, 6, 4 and 6 mailings, and of $-1.1$ points for Connecticut. The stored values 1.92, 0.02, 0.45, 1.04 and $-1.1$ are these programme differences. Table 3 reports rounded differences and not every standard error; the stored standard errors are close to binomial standard errors on the programme scale. For the New Jersey sample in which the number of mailings varies, the same article reports a slope of 0.16 per mailing with standard error 0.24 (p. 573), which is not among the stored values.

Derivation. Let a treatment group be assigned $m$ mailings and the control group none, let $D$ indicate assignment to treatment, and let $M=mD$. The regression of turnout on $D$ has slope $\hat b_D$, the difference in turnout between the two groups. Since $a+b_MM=a+(mb_M)D$, the regression on $M$ has the same fitted values. This implies that

A stored pair $(e_i,s_i)$ that is a programme difference therefore corresponds to the per-mailing pair $(e_i/m_i,\ s_i/m_i)$, and its weight $1/s_i^2$ becomes $m_i^2/s_i^2$. Let $W=\sum_i 1/s_i^2$ and $A=\sum_i e_i/s_i^2$ over a pool that contains the set $T$ of the five rows, so that the pooled estimate is $A/W$ with the article’s standard error $1/\sqrt{W}$. On the per-mailing scale, I obtain

and the pooled estimate $A'/W'$ with standard error $1/\sqrt{W'}$ by the same calculation.

Recalculation. The pooled estimates as printed reproduce from the dataset. Dividing the five rows by 9, 6, 4, 6 and 4 and keeping every other estimate, I find 0.1317 with standard error 0.0391 for conventional mail and $-0.0657$ with standard error 0.0797 for advocacy mail. The division is exact only for an estimate that compares a fixed number of mailings with none. The other scale checks were Niven’s (2006) single-mailing estimate and Ramirez’s (2005) two Colorado estimates, all per mailing; I did not examine the remaining estimates' units.

What survives

The counts and the classification of the treatments are unchanged, as are the estimates for nonadvocacy mail, 0.194 (0.106, 0.282), for social-pressure and gratitude mail, and for canvassing and phone calls. The interval for conventional mail stays above zero, and the interval for advocacy mail still contains zero. With the article’s variance calculation, the difference between the two types of mailings stays significant at the 5% level: it falls from 0.407 ($z=2.54$) to 0.260 ($z=2.84$), with 95% interval (0.080, 0.439). The statements on pp. 31–33 that conventional mail has weak effects and that advocacy mail has little effect on turnout stand. The pooled estimate for all 147 mail treatments in the replication program, which the article does not print, changes from 0.743 (0.669, 0.816) to 0.637 (0.568, 0.705). These intervals omit covariance between contrasts sharing controls; see P042-E001.

The problem

The sentence says that the pooled estimate for volunteer phone banks leaves out the experiments in which callers made follow-up calls to respondents who had said that they intended to vote. The 37 estimates pooled by the replication program include one from this class, the estimate for the OCAPICA phone bank in the election of June 2008. It is 11.1 with standard error 2.1, and the replication dataset lists its source as the book of García Bedolla and Michelson (2012). The campaign is described by Michelson, García Bedolla and McConnell (2009), the study of two-round phone banks that the article discusses on pp. 34–35. First-round respondents who intended to vote “were targeted for reminder calls just before Election Day” (p. 1556), and Table 7 of that study (p. 1557) reports 11.1 (2.1) as “the effect of one or more contacts”, that is, the effect of the two-round campaign as a whole. The estimate lies 9.4 points above the average of the other 36, and with it the pooled CACE is too high by 0.226 points.

The comparison drawn in the paragraph is affected as well. The article says that volunteer calls “seem to be more effective” than commercial calls, whose pooled CACE is 0.980. Under the independence model that the pooling uses, the difference between the two pooled estimates is 0.956 with $z=2.35$ when the estimate is included, and 0.730 with $z=1.78$ when it is removed. The pooled figures then no longer separate volunteer from commercial calls at the two-sided 5% level. The article prints no such test, and it words the comparison tentatively. The correction weakens the numerical support for the comparison, and for the reading on p. 33 that the effect of a live call depends on whether the message is “delivered in a routinized way by a commercial phone bank or in a more authentic manner by a volunteer phone bank”.

One point is not established. Nine other estimates among the 37 come from the same book. The replication dataset contains neither the number of rounds of calls nor the contrast behind each estimate, and therefore does not show whether any of the nine is also the overall effect of a two-round campaign. The corrected figure removes the one estimate known to belong to the excluded class.

Evidence a recalculation from the source

The evidence has two parts. The first is the text of Michelson, García Bedolla and McConnell (2009), pp. 1556–1557, quoted above, available in indexed excerpts of a repository copy; the typeset Table 7 was not inspected. The second is a recalculation from the replication files using exact rational weights and sums. Let $y_i$ denote the estimates and $s_i$ their standard errors, in percentage points. I set $w_i=s_i^{-2}$, $W=\sum_i w_i$ and $S=\sum_i w_i y_i$. The fixed effects average is $m=S/W$, its variance under independence is $1/W$, and the 95% interval is $m\pm zW^{-1/2}$ with $z\approx1.95996$. For the 37 estimates, I have $W\approx9.42573$ and $S\approx18.25079$, giving an average that rounds to the printed 1.936. The OCAPICA estimate has $y_j=11.1$ and $s_j=2.1$, so that $w_j=100/441$ and $w_jy_j=370/147$. Removing it, I obtain

with standard error $(W')^{-1/2}\approx0.32971$. The change in the average is

For the difference $d$ between two pooled estimates with weight sums $W_A$ and $W_B$, independence implies that $\operatorname{SE}(d)=(1/W_A+1/W_B)^{1/2}$, and I compare $z=d/\operatorname{SE}(d)$ with the normal distribution. The commercial calls have weight sum 16.95194.

The two differences have $z=2.353$ and $z=1.783$, with two-sided $p$-values 0.0186 and 0.0746.

What survives

The pooled estimate for volunteer calls remains positive, with a 95% interval above zero under independence, and its point estimate remains above that of commercial calls. The figures for commercial calls, canvassing, mail and pre-recorded calls do not use the removed estimate and are not changed by this correction. The experiments with follow-up calls reported on pp. 34–35 are also unchanged. If the statement on p. 35 that the CACE in those experiments is “more than twice as large as the typical CACE of a single phone call” was valid before correction, it remains valid when the typical CACE is this pooled figure, since the bound to exceed falls from 3.873 to 3.421. The text-message CACE of 4.1 on p. 33, which “rivals the effect of a volunteer phone call”, is now compared with 1.710. The count of 37 and the estimate 1.936 change, and under independence the pooled figures no longer separate volunteer from commercial calls at the two-sided 5% level.

The problem

The article declares one pooling method, a fixed effects average in which each study is weighted by its precision, and its nine other pooled figures on pp. 31–33 follow it. The figure in Note 5 does not. The two estimates for these experiments in the replication dataset are 2.32988 with standard error 0.55712 (“Robo SP1”) and 0.86596 with standard error 0.81782 (“Robo SP2”), and their fixed effects average is 1.866, with interval (0.963, 2.768). The printed estimate and both limits agree to three decimals with the DerSimonian–Laird random-effects estimate from the same two estimates. The note does not say that the method differs. The replication program has no command that pools the two estimates: its ten pooling commands all specify fixed effects, and its only mention of the two entries excludes them from the ordinary pre-recorded calls. Read alone, the words “a weighted average” admit the random-effects calculation. Read with the method declared on p. 31, the figure is wrong. Note that the numerical agreement does not identify the command that produced the printed figure, and it does not show whether the exception was intended.

Evidence an exact counterexample

The evidence is an exact calculation from the two estimates in the replication dataset. Let $x_1=2.32988$ and $x_2=0.86596$ denote the two estimates and $v_1=0.55712^2$ and $v_2=0.81782^2$ their variances, and set $w_i=1/v_i$. The fixed effects average and its variance are

Since $\widehat\mu_F>1.7205$, it cannot round to the printed 1.720. For the random-effects estimate, the heterogeneity statistic for two studies is $Q=(x_1-x_2)^2/(v_1+v_2)=2.188557$, and the DerSimonian–Laird estimate of the variance between studies is

With $u_i=1/(v_i+\widehat\tau^2)$, I obtain

The rounding of both intervals is the same for every multiplier between 1.9599 and 1.9600.

What survives

Both calculations give a positive estimate with a 95% interval above zero, so the statement of the note that these calls had stronger effects stands, as does the statement on p. 36 that such calls increase turnout. The nine other pooled figures of the article agree with the fixed effects calculation and do not change. Whether a third estimate for pre-recorded calls belongs with these two is a separate matter (P042-N010), and I do not change the count.

The problem

Each row of the dataset should describe the experiment behind its estimate. Row 7 names a St. Louis experiment and row 9 a Detroit experiment of 2001, and both carry the Newark elections of 2002 in the election field. At least one of the two fields is wrong in each row, and the dataset does not show which. In rows 256–259 the treatment group has 353,341 members in every row and the control group has 3,238, 6,307, 6,325 and 6,325 members, although the article describes one control group for the four mailings. These are the sizes stored for the 2010 study in rows 86–88 with treatment and control exchanged, and row 259 repeats row 258. The pattern is consistent with a copying slip, but it does not show what the correct sizes are.

Evidence a recalculation from the source

The evidence is a check of the replication dataset and a recalculation. I transcribed the fields above from the file. The ten pooling commands of the replication program use only the effect and standard-error columns, the classification flags and two title exclusions. In exact arithmetic, the ten pooled estimates and their variances are the same before and after the election and sample-size fields are changed. Neither Table 2 of Gerber, Green and Larimer (2008) nor Gillespie’s dissertation was available for the check, and the correct sizes and dates are not established here. The name of the file carries the date 19 March 2014, and I did not establish whether the file used for the article had the same entries.

What survives

The pooled estimates, intervals and counts on pp. 31–33 do not use these fields and are unchanged. The correction matters to anyone who uses the election or sample-size fields. The replication program also summarizes the column N, and that summary may change.

The problem

By the legend, a starred work is an input of the meta-analysis and is cited nowhere in the text. The marks depart from this rule in three ways.

• Five starred works are cited in the text: Almond and Verba (1963) on p. 28; Arceneaux and Nickerson (2009) on p. 38; Arceneaux and Nickerson (2010) on p. 35; Gerber and Green (2000a) on pp. 28, 31, 32 and 35; and Sinclair, McConnell and Green (2012) on pp. 36 and 37. Almond and Verba (1963) also matches no entry of the replication dataset.
• Six works that supply estimates to the pooled figures and are not cited in the text carry no asterisk: Arceneaux (2005); Arceneaux, Gerber and Green (2010); Barton, Castillo and Petrie (2012); Mann (2005); Mann and Mayhew (2012); and Nickerson (2007b).
• Shaw, Green, Gimpel and Gerber (2012) supplies two of the estimates for pre-recorded calls and has no entry in the reference list.

In addition, eight starred works supply no usable estimate to the pooled figures in the replication dataset. Seven match no entry: Aker, Collier and Vicente (2010); Bond et al. (2012); Gerber and Green (2000b); Gerber, Karlan and Bergan (2009); Mann and Klofstad (2011); Panagopoulos (2009b); and Panagopoulos and Green (2011). The eighth, Cann et al. (2012), matches two entries dated 2010, without standard errors.

Evidence a recalculation from the source

The evidence is a check of the reference list on pp. 40–48 and of the citations in the text, together with a comparison of the list with the 295 entries of the replication dataset and the selections of the replication program. The replication dataset is the release of March 2014, after publication, and I cannot confirm that it is the list of inputs used for the article. For this reason I give no complete set of corrected marks. The five citations in the text contradict the legend on the printed page and do not depend on the dataset.

What survives

No formula, estimate or argument of the article depends on the asterisks. The review and the pooled figures are unaffected; the defect is in the bookkeeping of the reference list.

The problem

The seven items are these. (1) The text and the reference list give different subtitles for one book. (2) The entries for 2001 and 2001b are the same article. (3) Volume 10 cannot be right for 1967, since the list has volume 50 of the same journal for 1956, 62 for 1968, 72 for 1978 and 80 for 1986, which implies volume 61 for 1967. (4) The surname is Wantchekon on pp. 38 and 48. (5) The word is “irregularities”. (6) No entry matches “Cutts et al. (2010)”; the only entry led by Cutts is dated 2009, and the article does not show which source is meant. (7) The list itself has the surname García Bedolla on p. 43.

Evidence a check of the printed page

The evidence is a check of the pages: I read each item on the page and compared it with the rest of the reference list. The book, the article of Kelley et al. (1967) and the source meant by the Cutts citation were not available. The correct subtitle, the page range and the authors' initials of the Kelley entry, and the intended Cutts reference are therefore not settled here. The entry for Bhavnani (2009) on p. 41 has a single page, 23, and I could not check whether a page range is missing.

What survives

The slips affect reference lookup and attribution. No estimate or argument depends on them, and the duplicated entry does not show that a study entered the meta-analysis twice.

The problem

For the Hawthorne, Self and Hawthorne-plus-survey mailings the dataset holds two estimates each. Rows 171–173 are labelled as estimates for people who did not vote in 2006. Rows 174–176 carry no such label. They are not estimates for the people who did vote in 2006. They are the estimates for the broader sample of the study, which contains the 2006 nonvoters. The same people therefore stand behind two estimates of each mailing, and three treatments are counted twice.

Evidence a recalculation from the source

The evidence is a check of the source and a recalculation from the replication files. I read Mann (2010) in the author’s posted manuscript of 20 April 2010, not in the journal’s typesetting. The first sample of the study has 78,441 people who did not vote in 2006. The broader sample has no further restriction (p. 12 of the manuscript), and Table 6 reports turnout of 13.2%, 16.3%, 15.8% and 15.7% in it. The differences from the first figure, 3.1, 2.6 and 2.5 points, are the values stored in rows 176, 175 and 174. The analysis of the broader sample leaves out only the 17,609 recipients of a fourth mailing from 95,788 people, so that 60,832 of the 2006 nonvoters are in both analyses. The 19,561 nonvoters in the control group are among the 25,037 controls of the broader sample. The printed 2.850 reproduces from the 29 rows. Without rows 174–176, the 26 remaining rows have a weighted average of 2.874, and without rows 171–173, of 3.069.

What survives

The effect of social-pressure mail stays large under either choice, and the pooled estimate moves by 1% to 8%. The pooled estimates for conventional and gratitude mail are unaffected, and no other pooled estimate of pp. 31–33 uses these rows. The overlap adds to the dependence between contrasts that the printed interval omits (P042-E001), and I did not establish a corrected interval.

The problem

Rows 10 and 11 of the dataset are two analyses of one experiment, the canvassing arm of the 1998 New Haven study. Row 11 is the report of 1999, “Does canvassing increase voter turnout? A field experiment”, and row 10 is the correction of 2005, “Correction to Gerber and Green (2000), Replication of Disputed Findings, and Reply to Imai (2005)”. The canvassing command of the replication program pools every row that has a canvassing estimate, and both rows enter. The experiment receives 2.31 times the weight of the corrected estimate alone, and a superseded estimate is pooled as if it were a separate treatment.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files in exact arithmetic. The 71 rows reproduce the printed 2.536 (1.817, 3.255). Without row 11, I obtain 2.4344 with standard error 0.3717 and nominal interval (1.7059, 3.1629) from the 70 remaining rows. That the two rows are one experiment rests on their election fields, on the abstracts stored with them and on p. 31 of the article, which identifies the 1998 New Haven study. I did not read the two source articles. A second pair, for a Detroit experiment of 2001 in rows 9 and 16, is not established as a duplicate, because the fields of row 9 conflict (P042-E003). Removing row 9 as well would give 2.410 (1.681, 3.139) over 69 estimates.

What survives

The estimate for canvassing stays positive, with an interval above zero, and the qualitative comparison of tactics on p. 33 is unchanged. No other pooled estimate uses these rows. The interval keeps the article’s calculation, which is the subject of P042-E001.

The problem

The paragraph sets the social-pressure calls aside and reports the average effect of the remaining pre-recorded calls, which it describes as messages recorded by politicians, celebrities or local clergy. The replication program forms this set by dropping two entries, titled “Robo SP1” and “Robo SP2”, from the thirteen estimates for pre-recorded calls. Among the eleven that remain is the estimate of Gerber, Green, Kaplan and Kern (2010), which is 1.85 with standard error 0.89 and which the replication dataset codes as a social-pressure call. This is the study cited on p. 36 for the statement that pre-recorded calls with a social-pressure message increase turnout (see P042-A002). The printed average is not the average for ordinary pre-recorded calls that the paragraph describes. The arithmetic is not at fault. From the eleven estimates, I find 0.156 and $(-0.118, 0.430)$, as printed, and it is true that two entries were excluded.

Three points are not established. The replication files do not show whether the three entries coded or titled as social pressure are three separate experiments, or whether the estimate of Gerber, Green, Kaplan and Kern repeats one of the two excluded entries. They do not show who recorded its message. They also do not show whether the stored 1.85 is an effect of assignment, as the paragraph says of the eleven, or an effect among those reached.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files in exact rational arithmetic. Let $b_i$ denote the estimates and $s_i$ their standard errors, in percentage points, and set $w_i=s_i^{-2}$. The fixed effects average and its standard error are

and the 95% interval is $\widehat b\pm z\operatorname{SE}(\widehat b)$ with $z\approx1.95996398454$. For the eleven estimates, I have $\sum_i w_i=51.25488$ and $\sum_i w_i b_i=8.00301$. Then $\widehat b=0.15614$, with standard error 0.13968 and interval $(-0.11763, 0.42991)$, which is the printed figure. The estimate of Gerber, Green, Kaplan and Kern has $w=0.89^{-2}=1.26247$ and $wb=2.33556$. Removing it, I obtain $\sum_i w_i=49.99241$ and $\sum_i w_i b_i=5.66744$, and therefore

The removed estimate holds 2.46% of the weight of the eleven, and the average falls by 0.043 points.

What survives

The conclusion of the paragraph stands: the average effect of ordinary pre-recorded calls is small, and its interval contains zero. The count, the estimate and the interval on p. 32 change, and the rough conversion to a CACE in Note 5, which divides the effect of assignment by 0.8, starts from 0.113. The pooled figures for mail, canvassing, commercial calls and volunteer calls use other estimates and do not change. I do not change the count of two experiments in Note 5, since the replication files do not show that there were three. The figure that Note 5 gives for those two is the subject of P042-E002.

The problem

A report issued in 2007 cannot report experiments conducted in 2008, so the years in the sentence and the citation do not agree. Which of the two should change is not established, since I have read neither the report nor a later source for the experiments of 2008. The statement that the experiments showed no interactions is not in question.

Evidence a check of the printed page

The evidence is a check of the two printed pages: the sentence on p. 34 and the single matching entry of the reference list on p. 46.

What survives

The conclusion of the section, that interactions between treatments tend to be weak, rests on several other studies that the paragraph discusses separately, and it is unaffected.

Intended meaning

The reference list has two entries that abbreviate to “Gerber et al., 2010”: Gerber, Green and Larimer (2010), on feelings of pride or shame, and Gerber, Green, Kaplan and Kern (2010), “Baseline, placebo, and treatment: efficient estimation for three-group experiments”. The sentence concerns pre-recorded calls, and the second entry is the study of such calls. The replication dataset codes its estimate, 1.85 with standard error 0.89, as a pre-recorded call with a social-pressure message, and the abstract stored there describes pre-recorded phone calls. The statement is correct, and only the abbreviation is ambiguous. The asterisk on the first entry points the same way, since by the legend a starred work is not cited in the text, but the asterisks are not reliable (P042-E004).

Intended meaning

The preceding sentence reads: “Other attributes, such as education or affluence, are also susceptible to experimental manipulation.” The example covers both attributes. Interventions that raise educational attainment are used to study the effect of schooling, and interventions that raise family income are used to study the effect of income. The sentence does not propose income interventions as a way to measure the effect of schooling. One point is left open. Doherty et al. (2006) is, by its title in the reference list, a study of personal income and attitudes toward redistribution, and its abstract describes effects on political attitudes. I could read only the abstract and cannot say whether the study reports a turnout outcome. This decides whether the citation fits a sentence about voter turnout.