Journal article P043

Corrections to: Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment

Donald P. Green, P. M. Aronow, Daniel E. Bergan, Pamela Greene, Celia Paris, and Beth I. Weinberger (2011). Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment. Journal of Politics 73(2):463–476.

DOI: 10.1017/s0022381611000107
Audited edition: Journal of Politics, 2011, 73(2):463–476; 14 pages in the audited file
SHA-256 of the audited file: 30d962cb6871dd7bb5811af49a11c4704b1e5e306ae7fb5a5c3a46f98b5053bd
Permanent address: https://pmaronow.github.io/llm-audit/works/P043/

Summary

More than 1000 students in 59 classrooms were randomly assigned to a civics course with or without a curriculum on constitutional rights. The curriculum raised knowledge of civil liberties, with no detected change in support for them. 23 entries are classified as errors (1 Moderate, 22 Minor). The corrected description reports the decline in the observed normalized control means by the third wave (P043-E007). The mediation argument is valid under its stated premise of positive effects through other channels; the experiment does not verify that premise.

What was and was not checked

I recalculated the following from the replication data or checked them against the pages; they agree with the paper, subject to the exceptions noted.

Table 2: every cell reproduces from the data, and the column totals add up; $588 + 627 = 1215$ is the total of the students assigned in Table 1.
The response rates 87.4% and 84.7% in 2007, and 53.0% and 52.5% in 2009 among the schools that remained.
The text that describes Table 1, and the 21 lessons of the curriculum listed in Appendix A.
Table 3: all sample sizes, means and standard deviations, and six of the nine reliability coefficients; the other three are corrected in P043-E004.
Table 4, columns without covariates: all nine coefficients and standard errors, to three decimals, with a random-effects fit on classrooms. Pooled ordinary least squares on the public data gives an approximate comparison for the columns with covariates (coefficients within about 0.03, standard errors within 0.001); top-coded birth years prevent exact reproduction of the original adjusted models (P043-E013).
The direction of scoring of all 17 support items, against the coding keys.
Tables A1–A3: every significance star agrees with the printed coefficient and standard error, and no support item is significant.
Table A6: eight of the nine coefficient and standard-error pairs without covariates reproduce to three decimals; the exception is P043-E005. The adjusted columns agree only approximately on the public data.
Table A4, apart from P043-E003, P043-N017 and P043-N018. In Table A5 the sample sizes 251 and 387 reproduce; in its upper triangle 32 of the 36 correlations agree to three decimals and four differ by 0.001, before the correction of P043-E003.
The six rescaled means in the figure on p. 11 of the Online Appendix.
The 2007 correlations on p. 471: 0.423, 0.297 and 0.157, printed as .42, .30 and .16.
Cohen’s $d = 0.49$ , as the unadjusted difference in means divided by the pooled standard deviation within groups (0.485).
Footnote 14: of the 36 interaction tests that can be run on the public data, three have $p < 0.05$ and none remains significant after the Holm–Bonferroni correction. The coding of one moderator and the coverage of these tests are the subjects of P043-E011 and P043-N015.
The argument of footnote 11, that outcome error with zero treatment–control mean difference does not bias the estimated treatment effect.
The reference list: every citation in the text has an entry, and conversely, with the one misspelling of P043-E008.

The two appendix files in the replication deposit are identical to the supplements used here.

I did not check the claim that this is the first large-scale experimental test of the proposition, which is a claim about the literature. The cited research was not read; of the court decisions, only Miranda v. Arizona was read, for P043-E006. I checked the random-effects estimates with an independent implementation of the moment estimator and with restricted maximum likelihood, since Stata was not available. The R script was not run unmodified from start to finish (P043-E012).

The public deposit lacks the original unedited data files, the CSV files that the R script reads, the original randomization records, the classroom observations and the teacher questionnaires. Since the public data top-code the year of birth, the covariate-adjusted columns of Tables 4, A1 and A6, the essay results of footnote 13, the age interactions and the randomization check (6.3%) cannot be reconstructed exactly, and the estimator behind the standard errors of the models with covariates could not be settled. These checks do not show that the paper is correct in the respects that were not examined.

The p. 474 mediation sentence expressly refers back to the positive-other-channel scenario on p. 473. Its conditional comparison is valid; random assignment does not verify its sign premise. The two constructed, exact cancellation examples violate that premise and limit only an unconditional reading. The total-effect estimates are unaffected; I did not assess the ample-power claim.

Headline claims and related corrections

More than 1,000 students in 59 classrooms at 10 high schools were randomly assigned, within 19 strata, to a civics course with or without the Bill of Rights curriculum; attrition was similar in the two arms.
Tables 1 and 2; Experimental Design, pp. 467-469
See P043-N001, P043-N004, P043-N006.
The paper reports increases in civil-liberties knowledge of $1.220$ correct answers out of 15 in 2007 (standard error $0.155$ , Cohen’s $d=0.49$ ) and $0.762$ out of 18 in 2008 (standard error $0.226$ ), both with $p<0.001$ .
Table 4, top panel; Table A2; footnote 13; abstract, pp. 465 and 473
See P043-N010.
The paper reports that the knowledge gain faded to $0.221$ in 2009 (standard error $0.269$ ), no longer statistically significant. After calibrating item difficulty with a 2010 survey, it describes control-group means as essentially unchanged across waves and the treatment group as declining to the control mean.
Table 4; Table A4; unnumbered figure on Online Appendix p. 11; pp. 465 and 473
See P043-E007, P043-E003.
The treatment had no effect, positive or negative, on general political knowledge in any wave, so learning in the targeted domain did not displace other knowledge.
Table 4, middle panel; Table A3; Table A6; pp. 465 and 473
See P043-E005.
The treatment had no detectable effect on support for civil liberties in 2007, 2008 or 2009, whether measured by the additive index or item by item, and the same holds among students present in all three waves.
Table 4, bottom panel; Table A1; Table A6; abstract, pp. 465, 473-474
See P043-E005.
The paper argues that the rise in knowledge without a detected rise in support calls into question a causal connection between knowledge and attitudes. It claims ample power to detect small effects and, under the hypothesized positive effects through other channels described on p. 473, a reduced-form test that favors finding mediation.
Figure 1; Table 4; abstract, pp. 473-475
See P043-E002.
The paper reports significant correlations among the knowledge and support indices in 2007: $r=0.42$ between the knowledge indices and $r=0.30$ and $r=0.16$ with support. It describes this as reproducing the pattern of observational studies and reports greater reliability for the knowledge scales than for the support scales.
Table 3; text p. 471; conclusion p. 474
See P043-E004.
Results are robust: covariates make little difference, the same pattern holds among students present in all three posttreatment waves, and there are no significant heterogeneous treatment effects after Holm-Bonferroni correction.
Table 4; Table A6; footnotes 12 and 14; pp. 468, 471, 473, 475
See P043-E005, P043-E011, P043-N015.
This is the first large-scale randomized experiment testing the effect of civics instruction on knowledge of and support for civil liberties.
Abstract; p. 465; conclusion p. 474; literature review pp. 463-464

Corrections

Moderate (1)

A real substantive error with limited reach.

P043-E007 Moderate

After the paper’s normalization, the control group means are 0.896, 0.884 and 0.805 in 2007, 2008 and 2009: the description “essentially unchanged” omits a decline of about 10% by the third wave, while the gap between the treatment and control means shrinks from 0.161 to 0.089 to 0.022.

p. 473 (PDF p. 11), right column, sentence beginning “After correcting for the changing difficulty of the items over time”; the figure of rescaled knowledge scores, p. 11 of the online appendix, and Table A4, p. 8 of the online appendix

I disputed this classification, but as a rule, deferred to the models’ final judgment: see my comment and the history of this entry.

As printed

After correcting for the changing difficulty of the items over time, we find that the control group means were essentially unchanged during the three postintervention waves, while the treatment group showed an initial surge followed by a gradual decline to the control group mean.

The problem

The sentence summarizes the unnumbered figure on p. 11 of the online appendix. The three survey waves used different questions to measure knowledge. For each wave, the figure takes the equivalent index of Table A4, which consists of the questions of that wave that were repeated on the 2010 survey. It divides this index by the mean of the same index on the 2010 survey. After this normalization, the control group means are 0.896 in 2007, 0.884 in 2008 and 0.805 in 2009. The control mean is nearly flat between the first two waves and then falls. Its decline from 2007 to 2009 is 0.090, which is 10.06% of its 2007 value and about 56% of the initial gap of 0.161 between the treatment and control means. The description “essentially unchanged” fits the first two waves, and the sentence extends it to the third, where it does not fit. The sentence then presents the treatment group as falling toward a control level that stayed fixed. The figure and the data show that both means fell and that the gap between them shrank. The paper states no tolerance for “essentially”, so there is no numerical criterion to refute, and I do not read the phrase as exact equality. The error is one of description: the sentence reports a stable control baseline and omits a decline of this size, which the paper’s own figure displays. The decline is a difference of observed means. It does not show that students forgot, that the normalization failed, or that a population mean changed.

Evidence a recalculation from the source

The evidence is a recalculation from the public replication data in exact arithmetic, together with a check of the appendix figure and of Table A4. Following the scoring rules of the Stata script in the replication files, I rebuilt the equivalent index of each wave (13, 17 and 9 questions) and the matching index on the 2010 survey. Each index is available for 251 of the 254 respondents in the public file of the 2010 survey, and the three totals over these respondents are $A_{2007} = 1904$ , $A_{2008} = 2128$ and $A_{2009} = 1345$ . Let $S_{ay}$ be the total of the equivalent index over the $n_{ay}$ students of group $a$ in the survey of year $y$ . The mean plotted in the figure is

M_{ay} = \frac{S_{ay}/n_{ay}}{A_y/251} = \frac{251\,S_{ay}}{n_{ay}A_y} .

For the control group in 2007, I obtain $M_{C,2007} = (3492/514)/(1904/251) = 219123/244664$ , which is 0.8956 to four decimals. A second calculation, written independently from the same files, gives the same fractions.

Rescaled means of the equivalent Civil Liberties Knowledge Index, recalculated from the public data, with the gap between the treatment and control means. The first three rows use every student whose index is available in that wave, and the last three use the 387 students whose index is available in all three waves.
Students	Year	Control $n$	Control mean	Treatment $n$	Treatment mean	Gap
All in the wave	2007	514	0.8956	531	1.0566	0.1610
All in the wave	2008	296	0.8842	295	0.9728	0.0886
All in the wave	2009	253	0.8055	255	0.8277	0.0222
The same 387	2007	200	0.9215	187	1.0772	0.1557
The same 387	2008	200	0.8958	187	0.9682	0.0724
The same 387	2009	200	0.8192	187	0.8632	0.0440

The public files are a release edited for privacy; the numbers of students in the three waves (1045, 591 and 508) agree with Table 4. The 2010 means $A_y/251$ are 7.59, 8.48 and 5.36 to two decimals, and the pooled means of the three waves are 7.41, 7.87 and 4.38; these six values agree with Table A4. The points of the appendix figure lie at the values of the first three rows: about 0.90, 0.88 and 0.81 for the control group, and about 1.06, 0.97 and 0.83 for the treatment group.

The control mean falls by 0.0114 from 2007 to 2008 and by 0.0901 from 2007 to 2009. Among the 200 control students whose index is available in all three waves, the decline from 2007 to 2009 is 0.1022, which is 11.09% of the 2007 value. Holding the set of students fixed therefore does not remove the decline. The gap is positive and decreasing in both sets of students. The treatment mean falls by 0.2289 from 2007 to 2009. Writing $G_y = M_{T,y} - M_{C,y}$ for the gap, I have

M_{T,2009} - M_{T,2007} = \bigl(M_{C,2009} - M_{C,2007}\bigr) + \bigl(G_{2009} - G_{2007}\bigr),

and the two terms on the right are $-0.0901$ and $-0.1388$ . The identity relates observed means and has no causal interpretation. I ran no test of change over time that accounts for repeated students, classrooms and the uncertainty of the 2010 means. I do not use the confidence bars of the figure as such a test.

What survives

This correction leaves the appendix figure, its means and Table A4 unchanged. The decline of the treatment advantage stands: the gap between the treatment and control means is positive and shrinks from 0.161 to 0.089 to 0.022. Each regression of Table 4 and of the appendix is fitted within one wave and does not assume a constant control mean across waves. The sentence is not an input to those estimates. I have not recalculated those estimates here. What changes is the description of the two trajectories. The control baseline was not stable after the normalization, and the data do not support describing the decline of the treatment group as a fall toward a fixed control level.

Corrected

After applying this normalization, the observed control-group means were approximately 0.896, 0.884 and 0.805 in 2007, 2008 and 2009, respectively. The treatment-control gap shrank from approximately 0.161 to 0.089 to 0.022 over these waves.

This wording replaces the whole sentence. A shorter repair keeps the sentence and replaces only its clause on the control group, which then reads “the observed rescaled control-group means declined, particularly in the third postintervention wave”. Neither wording attributes the decline of the control means to forgetting or to a failure of the normalization.

Crop of ten lines of text. The passage reads: footnote 10, which included most or all of the index questions on a single questionnaire (See Table A4). After correcting for the changing difficulty of the items over time, we find that the control group means were essentially unchanged during the three postintervention waves, while the treatment group showed an initial surge followed by a gradual decline to the control group mean. The treatment had no effect, positive or negative, on general political knowledge. The estimates hover; the crop ends in the middle of this sentence. Red outlines surround the words essentially unchanged during the three postintervention waves, which run over two lines. — p. 473 (PDF p. 11), right column. The outlines mark “essentially unchanged during the three postintervention waves”, the description of the control group means that the appendix figure does not support.

Plot of six points with vertical bars. The horizontal axis is the year, with ticks at 2007, 2008 and 2009. The vertical axis is the rescaled civil liberties knowledge score, with ticks from 0.7 to 1.1. In each year one point is labelled T and one is labelled C. The T points lie at about 1.06, 0.97 and 0.83. The C points lie at about 0.90, 0.88 and 0.81, and the third is lower than the first two. In 2009 the T and C points are close together and their bars overlap. Red outlines surround the three C points. — Online appendix, p. 11: the plot of the unnumbered figure “Rescaled Civil Liberties Knowledge Score Across 2007, 2008 and 2009 Surveys”. T is the treatment group, C is the control group, and the bars are 95% confidence intervals. The outlines mark the three control group means, which lie at about 0.90, 0.88 and 0.81.

My comment and the history of this entry

I think this is largely a subjective interpretation of wording.

Suggested classification: Minor error.

My position My comment above; I suggested a Minor error.

The models’ position The control group’s mean falls by about 0.09, around 10% of its starting level and 56% of the initial gap between the groups, so describing it as essentially unchanged materially changes how a reader understands the trajectories: Moderate.

Classification before my review (the classification when the audit was first completed, before I reviewed it): Moderate error.
I commented and suggested a Minor error.
Independent evaluation. Not adopted: the entry stays a Moderate error, not the Minor rating I suggested.

Minor (22)

Real but inconsequential, for example a misprint that does not change a result.

P043-E002 Minor

Random assignment and the exclusion restriction do not by themselves give a local effect of knowledge among the students whose knowledge the curriculum raises; monotonicity of knowledge in the assignment and a positive first stage suffice to justify this interpretation.

Statistical Model, p. 472 (PDF p. 10), foot of the left column to the top of the right column; Conclusion, pp. 474–475 (PDF pp. 12–13)

See also under Formal statements (Lean): P043-T4, P043-T5

As printed

p. 472: “If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).”

pp. 474–475: “This line of argument takes notice of the fact that the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.”

The problem

The first sentence obtains a local average treatment effect of knowledge from two conditions: the curriculum is randomly assigned, and it affects support only through knowledge. The second sentence reads that effect as one identified among the students whose knowledge the curriculum raises. The two conditions do not give this reading. Let $K_i(z)$ be the knowledge score of student $i$ under assignment $z\in\{0,1\}$ , and let $Y_i(k)$ be the support of student $i$ at knowledge score $k$ . Under the two conditions, the ratio that the instrumental-variable estimator targets is

\beta_{\mathrm{IV}}=\frac{\mathbb{E}[Y_i(K_i(1))-Y_i(K_i(0))]}{\mathbb{E}[K_i(1)-K_i(0)]}

when the denominator is positive, and nothing fixes the sign of $K_i(1)-K_i(0)$ . A student whose knowledge the curriculum lowers enters both expectations and can cancel the contribution of the students whose knowledge it raises. The reading needs a restriction on how knowledge responds to the curriculum, such as monotonicity: $K_i(1)\ge K_i(0)$ for every student. The paper cites the framework in which this condition is stated but does not state it, and a positive effect on mean knowledge does not imply it.

Evidence an exact counterexample

The evidence is an exact counterexample and a derivation. Consider a population with two types of student, in which every knowledge score lies inside the 2008 range of 0 to 18 and every support score inside the range of 0 to 7. The curriculum is assigned with probability $1/2$ , independently of type. I write $Z$ for the assignment and $K$ and $Y$ for the observed knowledge and support.

A population in which the instrumental-variable ratio is zero although knowledge raises support.
Type	Share	$K_i(0)$	$K_i(1)$	$Y_i(8)$	$Y_i(9)$
Knowledge rises	$2/3$	8	9	3	4
Knowledge falls	$1/3$	9	8	2	4

Support depends on the assignment only through knowledge, as the exclusion restriction requires. By the table, I have $\mathbb{E}[K\mid Z=0]=25/3$ and $\mathbb{E}[K\mid Z=1]=26/3$ , and the first stage is $1/3>0$ . I also have $\mathbb{E}[Y\mid Z=0]=\mathbb{E}[Y\mid Z=1]=10/3$ . Then the reduced form is $0$ and $\beta_{\mathrm{IV}}=0$ . However, every student whose knowledge rises has a causal effect of $4-3=1$ on support. The numerator vanishes by cancellation: $(2/3)(1)+(1/3)(-2)=0$ .

Other features of the observed data do not identify this effect in general either. Two further populations, each with five equally likely types, have the same joint distribution of assignment, knowledge and support, with first stage $1/5$ and reduced form $0$ . The average effect among the students whose knowledge rises is $1$ in the first population and $4/3$ in the second.

Under monotonicity, the corrected statement follows by telescoping. For an integer score,

\begin{aligned} Y_i(K_i(1))-Y_i(K_i(0))&=\sum_{j}\mathbf{1}\{K_i(0)<j\le K_i(1)\}\,[Y_i(j)-Y_i(j-1)],\\ K_i(1)-K_i(0)&=\sum_{j}\mathbf{1}\{K_i(0)<j\le K_i(1)\}. \end{aligned}

The expectation of the $j$ th term of the first sum is $\Pr(K_i(0)<j\le K_i(1))$ times the mean of $Y_i(j)-Y_i(j-1)$ among the students such that $K_i(0)<j\le K_i(1)$ . Taking expectations of both identities and dividing the first by the second, I obtain the weighted average in the corrected statement.

What survives

The paper reports no instrumental-variable estimate. It chooses the reduced-form regressions described on p. 473, and Table 4 and the online appendix report coefficients on the curriculum. Those estimates and their standard errors are unaffected, as are the findings that the curriculum raised mean knowledge and had no detected effect on support. The interpretation changes. Without monotonicity, a zero effect of the curriculum on support does not show that knowledge has no effect on support among the students whose knowledge the curriculum raised. Monotonicity is an assumption about every student, and the paper does not test it. The check does not show that the curriculum lowered any student’s knowledge.

Corrected

p. 472: “If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, and that the treatment lowers no student’s knowledge and raises knowledge on average, then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).”

pp. 474–475: “This line of argument takes notice of the fact that, if the treatment curriculum lowers no student’s knowledge and raises knowledge on average, the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.”

In full, let $Z_i\in\{0,1\}$ be the assignment of student $i$ , let $K_i(z)$ be the student’s knowledge score under assignment $z$ , and let $Y_i(k)$ be the student’s support at knowledge score $k$ . Assume that $Z_i$ is independent of $(K_i(0),K_i(1),Y_i(\cdot))$ , that $0<\Pr(Z_i=1)<1$ , that assignment affects support only through knowledge, and that the observed scores are $K_i=K_i(Z_i)$ and $Y_i=Y_i(K_i)$ . Assume in addition that $K_i(1)\ge K_i(0)$ for every student and that $\mathbb{E}[K_i(1)-K_i(0)]>0$ . Then, for an integer knowledge score,

\begin{aligned} \frac{\mathbb{E}[Y_i\mid Z_i=1]-\mathbb{E}[Y_i\mid Z_i=0]}{\mathbb{E}[K_i\mid Z_i=1]-\mathbb{E}[K_i\mid Z_i=0]} &=\sum_{j}w_j\,\mathbb{E}\bigl[Y_i(j)-Y_i(j-1)\mid K_i(0)<j\le K_i(1)\bigr],\\ w_j&=\frac{\Pr(K_i(0)<j\le K_i(1))}{\mathbb{E}[K_i(1)-K_i(0)]}, \end{aligned}

where the sum is over the score levels $j$ with $\Pr(K_i(0)<j\le K_i(1))>0$ . The weights $w_j$ are nonnegative and sum to one. Only students whose knowledge the curriculum raises enter the conditional means, and a student who crosses several levels contributes at each of them. Another sufficient restriction on the response of knowledge to the curriculum may replace monotonicity. Without such a restriction, random assignment and exclusion do not identify an effect confined to the students whose knowledge the curriculum raises. No numerical estimate is corrected.

Crop of the last seven lines of the left column of page 472. A red box marks the last line, 'then we can estimate the local average treatment'. The two lines above it read 'assume that the treatment only influences support for civil liberties indirectly by influencing knowledge,'. — Statistical Model, p. 472 (PDF p. 10), foot of the left column. The marked line begins the conclusion drawn from random assignment and the exclusion restriction; it continues at the top of the right column.

Crop of the first four lines of the right column below Table 3. Red boxes mark the words 'effect of knowledge by using randomly assigned curriculum as an instrumental variable', which are followed by the citation '(Angrist, Imbens, and Rubin 1996)'. — p. 472 (PDF p. 10), top of the right column, below Table 3. The marked words complete the sentence: the effect of knowledge is to be estimated with the curriculum as an instrument.

Crop of the last seven lines of the right column of page 474. A red box marks the last line, 'effects of increases in knowledge are identified only'. The lines above it attribute the interpretation to the work of Angrist, Imbens, and Rubin (1996). — Conclusion, p. 474 (PDF p. 12), foot of the right column. The marked line begins the statement that effects of knowledge are identified in a subgroup; it continues on p. 475.

Crop of the running head and the first seven lines of the left column of page 475. Red boxes mark the words 'among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.' The next sentence begins 'It may be that support for civil liberties among students who are affected on the margin'. — Conclusion, p. 475 (PDF p. 13), top of the left column. The marked words name the subgroup: the students whose knowledge the curriculum raises.

P043-E003 Minor

The 2009 support statistics of Table A4 and eight correlations of Table A5 are those of the seven-item index, where the notes promise the six items asked again in 2010; the six-item values are mean 3.76, standard deviation 1.25 and alpha 0.38.

Online Appendix, p. 8, Table A4, Posttreatment Survey 2009 Indices, column “Support for Civil Liberties Index”, and the note under the table; p. 9, Table A5, upper triangle, last column

As printed

Table A4, Posttreatment Survey 2009 Indices, Support for Civil Liberties Index (no asterisk). Number of Questions 6; Mean (2010 Survey) 3.77; Standard Deviation (2010 Survey) 1.33; Cronbach’s Alpha (2010 Survey) 0.43; Mean (2009 Survey) 4.06; Standard Deviation (2009 Survey) 1.27; Cronbach’s Alpha (2009 Survey) 0.31.

* Index consists of a subset of questions originally included. Notes: All questions rescaled to 0–1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007–2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey.

Table A5, upper triangle, column “Support for Civil Liberties Index 2009”, first eight rows. 0.218, 0.176, 0.232, 0.088, 0.164, 0.223, 0.346, 0.289.

Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387).

The problem

The 2009 Support for Civil Liberties Index has seven items (Table 3, p. 472). The 2010 survey did not ask one of them, the question on allowing a Ku Klux Klan rally, and the 2010 version has six items, as the row “Number of Questions” says. By the note, the 2009 statistics in this column should be those of the same six items. The printed values 4.06, 1.27 and 0.31 are instead the statistics of the seven-item index, exactly as in Table 3. The same index enters the eight correlations in the last column of the upper triangle of Table A5, which the note to that table also describes as correlations for the equivalent questions. The heading of the column in Table A4 lacks the asterisk that marks a subset of the original questions.

Evidence a recalculation from the source

The evidence is a check of the pages and of the replication code, and a recalculation from the replication data in exact arithmetic. Let $q_{11}, q_{13}, q_{15}, q_{17}, q_{18}, q_{39b}, q_{44}$ denote the responses to the seven items, after the item mean is imputed for a response coded 9 as in the replication code, and define $w_{18} = 1 + (q_{18} - 1)/2$ and $w_{39b} = 1 + (5 - q_{39b})/4$ . The seven-item index of the replication code and the six-item index that matches the 2010 survey are

\begin{aligned} D_7 &= q_{11} + q_{13} + q_{15} + q_{17} + w_{18} + w_{39b} + q_{44} - 7, \\ D_6 &= q_{11} + q_{13} + q_{15} + q_{17} + w_{18} + q_{44} - 6 = D_7 - (w_{39b} - 1). \end{aligned}

The code computes the 2009 entries of Tables A4 and A5 from $D_7$ and the 2010 entries from six items.

Support for Civil Liberties Index: the recalculated statistics. The first and third rows reproduce the printed column.
Score	$N$	Mean	Standard deviation	Alpha
2009 survey, seven items ( $D_7$ )	508	4.0570	1.2732	0.3142
2009 survey, six items ( $D_6$ )	508	3.7573	1.2512	0.3839
2010 survey, six items	251	3.7703	1.3277	0.4334

Note that the sample of 387 students with all nine indices observed is the same for $D_6$ and $D_7$ .

Table A5, upper triangle, last column: the printed correlations, the recalculation with seven items and the recalculation with six.
Row	Printed	Seven items	Six items
Civil Liberties Knowledge 2007	0.218	0.21747	0.20567
Civil Liberties Knowledge 2008	0.176	0.17635	0.16437
Civil Liberties Knowledge 2009	0.232	0.23163	0.22063
General Political Knowledge 2007	0.088	0.08813	0.09451
General Political Knowledge 2008	0.164	0.16396	0.15547
General Political Knowledge 2009	0.223	0.22294	0.20694
Support for Civil Liberties 2007	0.346	0.34610	0.37079
Support for Civil Liberties 2008	0.289	0.28927	0.26673

Seven of the eight printed correlations agree with the seven-item recalculation to three decimals. The first is printed as 0.218 and recalculated as 0.217. I could not establish the cause of this difference of 0.001; the original data, before the privacy edit of 2019, are not available.

What survives

The seven-item 2009 outcome of Table 3, the treatment effects of Table 4 and the analyses of single items are unaffected, since the correction introduces a separate score on six items for the comparison in the appendix only. The lower triangle of Table A5 and the entries that do not involve 2009 support stand, and the figure on p. 11 of the Online Appendix concerns knowledge, not support. Footnote 10 (p. 471) cites the 2010 survey in support of the reliability estimates; for 2009 support, that comparison was not made on matched items.

Corrected

Table A4, Posttreatment Survey 2009 Indices. Support for Civil Liberties Index*: Number of Questions 6; Mean (2009 Survey) 3.76; Standard Deviation (2009 Survey) 1.25; Cronbach’s Alpha (2009 Survey) 0.38. The 2010 entries are unchanged.

Table A5, upper triangle, column “Support for Civil Liberties Index 2009”, first eight rows. 0.206, 0.164, 0.221, 0.095, 0.155, 0.207, 0.371, 0.267. The diagonal entry remains 1.000.

These values use the six-item score $D_6$ defined above, on the same 387 students. The seven-item outcome of the main analysis is unchanged. Alternatively, keep the printed values and revise the item counts and both notes to say that seven items of the 2009 survey are compared with six items of the 2010 survey; the comparison then no longer claims equivalent questions.

Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index without an asterisk. The rows are Number of Questions (9, 7, 6), Mean (2010 Survey) (5.36, 3.46, 3.77), Standard Deviation (2010 Survey) (2.31, 1.90, 1.33), Cronbach's Alpha (2010 Survey) (0.70, 0.64, 0.43), Mean (2009 Survey) (4.38, 3.17, 4.06), Standard Deviation (2009 Survey) (2.28, 1.67, 1.27) and Cronbach's Alpha (2009 Survey) (0.66, 0.50, 0.31). A red outline surrounds the three entries 4.06, 1.27 and 0.31 in the last column. Under the table: asterisk, Index consists of a subset of questions originally included. Notes: All questions rescaled to 0 to 1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007 to 2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey. — Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2009 Indices”, with the notes under the table. The outline marks the three 2009 statistics of the Support for Civil Liberties Index, which are those of the seven-item index. The heading of that column has no asterisk.

Crop of Table A5, Index Correlations on 2010 Survey, a nine by nine matrix whose rows and columns are the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, each for 2007, 2008 and 2009. A red outline surrounds the first eight entries of the last column: 0.218, 0.176, 0.232, 0.088, 0.164, 0.223, 0.346 and 0.289. The ninth entry, on the diagonal, is 1.000. The note under the table reads: Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387). — Online Appendix, p. 9, Table A5 with its note. The outline marks the eight correlations of the 2009 Support for Civil Liberties Index in the upper triangle.

P043-E004 Minor

For the additive knowledge indices that the paper analyses, three reliability coefficients in Table 3 are 0.60, 0.65 and 0.67, not 0.62, 0.66 and 0.68.

Table 3, p. 472 (PDF p. 10), rows “Cronbach’s Alpha” of the 2008 and 2009 surveys

As printed

Table 3, rows “Cronbach’s Alpha”. Posttreatment Survey 2008: Civil Liberties Knowledge Index 0.62, General Political Knowledge Index 0.66. Posttreatment Survey 2009: Civil Liberties Knowledge Index 0.68.

The problem

Table 3 describes the outcome indices, and each knowledge index is the sum of items scored 1 for a correct answer and 0 otherwise. For these sums, on the samples of Table 3, Cronbach’s alpha is 0.60, 0.65 and 0.67. The printed 0.62, 0.66 and 0.68 are the values obtained when one item of each scale is reverse-scored: Cq20p, Cq41p and Dq34p, in the names of the replication files. Each of these is the only item of its scale that covaries negatively with the sum of the others. The Stata program computes the three coefficients with the alpha command and without the option asis, which keeps the items as scored. The indices add all the items as scored. This is consistent with an automatic reversal of the three items. The rounded entries do not determine every detail of the original computation.

Evidence a recalculation from the source

This is a recalculation from the replication data in exact arithmetic. For $k$ items $x_1,\dots,x_k$ with sum $y$ , on the $n$ respondents with a complete index, the coefficient is

\alpha=\frac{k}{k-1}\left(1-\frac{\sum_{j=1}^{k}\operatorname{Var}(x_j)}{\operatorname{Var}(y)}\right).

Cronbach’s alpha of the three scales: as printed, for the items as scored, and with one item reversed.
Scale	$n$	$k$	Printed	Items as scored	One item reversed
2008 civil liberties knowledge	591	18	0.62	0.601442718	0.618754678 (`Cq20p`)
2008 general political knowledge	591	22	0.66	0.654451008	0.657504233 (`Cq41p`)
2009 civil liberties knowledge	508	16	0.68	0.674405221	0.680637049 (`Dq34p`)

Reversing a binary item $x_j$ leaves every item variance unchanged and replaces $\operatorname{Var}(y)$ by $\operatorname{Var}(y)-4\operatorname{Cov}(x_j,y-x_j)$ , which raises alpha when the covariance is negative. The same calculation reproduces the other three knowledge coefficients of Table 3 (0.67, 0.48 and 0.61) with no reversal, and the sample sizes, means and standard deviations of all six knowledge indices.

What survives

The outcome indices, the sample sizes, means and standard deviations of Table 3 and the regressions of Table 4 do not use these coefficients and are unchanged. The knowledge scales still have higher alpha than the support scales (0.60 and 0.65 against 0.45 in 2008, and 0.67 against 0.31 in 2009). I also recalculated the support-scale coefficients; they reproduce the printed $0.37$ , $0.45$ and $0.31$ .

Corrected

Table 3, rows “Cronbach’s Alpha”. Posttreatment Survey 2008: Civil Liberties Knowledge Index 0.60, General Political Knowledge Index 0.65. Posttreatment Survey 2009: Civil Liberties Knowledge Index 0.67.

These are the raw coefficients of the items as scored in the indices, computed on the respondents with a complete index ( $N=591$ , $591$ and $508$ ). The outcome scores are unchanged.

Table 3, Description of Outcome Variables, with columns for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, and blocks of rows for the posttreatment surveys of 2007, 2008 and 2009. Red boxes mark three entries in the rows labelled Cronbach's Alpha: 0.62 and 0.66 in the first two columns of the 2008 block, and 0.68 in the first column of the 2009 block. — Table 3, p. 472 (PDF p. 10). The marked entries are the three reliability coefficients corrected here: 0.62 and 0.66 for the two knowledge indices of 2008, and 0.68 for the civil liberties knowledge index of 2009.

P043-E005 Minor

The coefficient $-0.308$ should read $-0.031$ , with the printed standard error $0.233$ .

Online Appendix, p. 10, Table A6, General Political Knowledge Index, 2009 Posttreatment Survey, column without covariates

As printed

Table A6, General Political Knowledge Index, 2009 Posttreatment Survey; Covariates: No; Strata: Yes; N 387. Treatment $\colorbox{#ffe58a}{$-0.308$}$ $(0.233)$ .

The problem

The cell reports the effect of treatment on the 2009 General Political Knowledge Index among the 387 students who completed all three posttreatment surveys, from a generalized least squares regression with random effects for classrooms and indicators for strata, without covariates. That specification gives $-0.0308$ , which rounds to $-0.031$ . The printed coefficient is ten times as large. The printed standard error is correct.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data. For stratum $s$ and arm $t$ (0 for control, 1 for treatment), let $n_{st}$ be the number of students among the 387 and $S_{st}$ the sum of their scores.

Students and sums of the 2009 General Political Knowledge Index among the 387 students, by stratum and arm.
Stratum $s$	$n_{s0}$	$n_{s1}$	$S_{s0}$	$S_{s1}$
A	8	9	33	34
B	13	7	78	43
I	15	30	83	148
J	16	26	64	107
K	9	23	46	94
L	6	3	25	14
M	9	5	34	19
Q	78	34	281	141
R	14	15	86	83
S	4	7	17	23
T	28	28	154	159
Total	200	187	901	865

In this cell the moment estimate of the variance between classrooms is negative ( $-0.048$ ), and the estimator sets it to zero. The random-effects estimate is then the least squares coefficient of treatment in a regression with an intercept for each stratum. Solving the normal equations for the stratum intercepts and inserting them into the equation for the treatment coefficient $b$ , I obtain

b = \frac{\sum_s (n_{s0} S_{s1} - n_{s1} S_{s0})/(n_{s0} + n_{s1})}{\sum_s n_{s0} n_{s1}/(n_{s0} + n_{s1})} = -\frac{49321068}{1600529849} \approx -0.03082 .

The residual sum of squares, with $387 - 12 = 375$ degrees of freedom, gives the standard error $0.23323$ . The same calculation reproduces the eight other coefficients without covariates in Table A6 and all nine of their standard errors to three decimals. The conclusion does not depend on how the variance between classrooms is estimated. For every nonnegative ratio of that variance to the residual variance, the coefficient from generalized least squares lies strictly between $-1/5$ and $1/5$ , which implies that no such ratio gives a value that rounds to $-0.308$ .

What survives

The standard error of the cell stands, and the estimate is not significant at the 5% level with either value. Table 4 uses larger samples and does not change. Footnote 12 (p. 473) cites Table A6 for the robustness of the results among students present in all three waves; the corrected estimate is closer to zero. This recalculation covers the nine cells of Table A6 without covariates.

Corrected

Table A6, General Political Knowledge Index, 2009 Posttreatment Survey; Covariates: No; Strata: Yes; N 387. Treatment $\colorbox{#ffe58a}{$-0.031$}$ $(0.233)$ .

Crop of one panel of a table. Under the heading General Political Knowledge Index, three pairs of columns are labelled 2007, 2008 and 2009 Posttreatment Survey. The Treatment row gives 0.201 (0.129), 0.141 (0.126), -0.180 (0.315), -0.163 (0.321), -0.308 (0.233) and -0.139 (0.237), with standard errors in parentheses. The Covariates row alternates No and Yes, the Strata row reads Yes throughout, and N is 387 in every column. A red outline surrounds the entry -0.308. — Online Appendix, p. 10, Table A6, panel “General Political Knowledge Index”. The outline marks the 2009 coefficient without covariates.

P043-E006 Minor

Two questions of the knowledge indices place the Miranda warnings at the time of arrest; the decision requires them before custodial questioning.

Mini Appendix, p. 2 (pages unnumbered), “2008 Civil Liberties Knowledge Index Questions”, item 13; p. 3, “2009 Civil Liberties Knowledge Index Questions”, item 8

As printed

2008, item 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case]

2009, item 8. As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/a reading of his or her rights at the time of arrest/protection against cruel or unusual punishment.)

The problem

In both questions the answer scored as correct concerns Miranda v. Arizona, and both describe its rule by the moment of arrest. The opinion requires the warnings before custodial interrogation, that is, before law enforcement questions a person in custody or otherwise significantly deprived of freedom of action. Arrest alone does not trigger the requirement. The two keyed statements overstate the timing and the scope of the rule. In each question the Miranda answer remains the only plausible choice.

Evidence a check of the printed page

The evidence is a check of the pages and of the opinion. The questionnaires and coding keys in the replication files confirm the wording and the keyed answers. The opinion of the Court in Miranda v. Arizona, 384 U.S. 436 (1966), says at p. 444: “By custodial interrogation, we mean questioning initiated by law enforcement officers after a person has been taken into custody or otherwise deprived of his freedom of action in any significant way.” Its summary at p. 479 says: “He must be warned prior to any questioning that he has the right to remain silent”. The quotations are from the United States Reports.

What survives

The intended answer is recognizable in both questions. No change in the responses, in the knowledge scores or in the estimated treatment effects has been shown, and none is claimed here.

Corrected

2008, item 13. Police must inform suspected criminals of their constitutional right to remain silent before custodial questioning. [Matching court case]

2009, item 8. As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/warnings about his or her rights before custodial questioning/protection against cruel or unusual punishment.)

Crop of four lines of running text. The text reads: allow the prosecutor to retry Mr. Jones in this case? (Yes/No) 12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government] 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case] 14. School districts are allowed to conduct random drug tests on any student involved in extracurricular activities. A red outline surrounds the words while being arrested. — Mini Appendix, p. 2, list of the 2008 Civil Liberties Knowledge Index questions, item 13. The outline marks the phrase that places the warning at the time of arrest.

Crop of four lines of running text. The text reads: of Rights does not safeguard individual liberties.) 8. As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/a reading of his or her rights at the time of arrest/protection against cruel or unusual punishment.) 9. The clear and present danger ruling of the Supreme Court in Schenck v. United States illustrates the continuing. A red outline in two parts surrounds the words a reading of his or her rights at the time of arrest. — Mini Appendix, p. 3, list of the 2009 Civil Liberties Knowledge Index questions, item 8. The outline marks the answer scored as correct; it runs over a line break.

P043-E008 Minor

The cited surname is Stehlik-Barry, and the stratum indicators of equation (1), written $D_k$ in the display and $D_i$ in the sentence after it, should read $D_{ik}$ in both places.

Introduction, p. 464 (PDF p. 2), left column, first paragraph; Statistical Model, p. 471 (PDF p. 9), right column, equation (1) and the sentence after it

As printed

p. 464. … even studies that emphasize the importance of cognitive skills or the experience of diversity acknowledge the mediating role of political knowledge (Nie, Junn, and Stehlik-Berry 1996; Stouffer 1955). $^{3}$

p. 471. Thus, our model has the form:

Y_i = \beta_0 + \beta_1 X_i + \sum_{k=1}^{K-1} \gamma_k \colorbox{#ffe58a}{$\displaystyle D_k$} + u_i, \qquad (1)

where $X_i$ is scored 0 for control and 1 for treatment, the $K$ -1 dummy variables marking each of the $K$ strata are denoted $\colorbox{#ffe58a}{$D_i$}$ , and $u_i$ denotes the unobserved disturbance term.

The problem

There are two misprints. On p. 464 the surname of the third author is spelled “Stehlik-Berry”. A second citation on the same page and the entry in the reference list on p. 476 have “Stehlik-Barry”.

In equation (1) the stratum indicator is written $D_k$ , with no index for the student, and the next sentence calls the same indicators $D_i$ , with no index for the stratum. An indicator of membership in a stratum depends on both. The same sentence says that the $K-1$ indicators mark “each of the $K$ strata”. They mark all strata except one, as the sentence before the display says (“adding dummy variables for all strata but one”).

Evidence a check of the printed page

The evidence is a check of the pages: pp. 464 and 476 for the surname, and p. 471 for the display and the two sentences around it. The replication code enters the stratum as a categorical variable in the model, which is the specification written below. No recalculation was needed.

What survives

Nothing depends on either misprint. The intended model is clear from the sentence before the display, and no estimate or conclusion changes.

Corrected

p. 464. (Nie, Junn, and Stehlik-Barry 1996; Stouffer 1955).

p. 471. Thus, our model has the form:

Y_i = \beta_0 + \beta_1 X_i + \sum_{k=1}^{K-1} \gamma_k \colorbox{#ffe58a}{$\displaystyle D_{ik}$} + u_i, \qquad (1)

where $X_i$ is scored 0 for control and 1 for treatment, the $K-1$ dummy variables for all strata except the reference stratum are denoted $D_{ik}$ , where $D_{ik} = 1$ if student $i$ belongs to stratum $k$ and $D_{ik} = 0$ otherwise, and $u_i$ denotes the unobserved disturbance term.

Here $Y_i$ is one of the three outcome measures for student $i$ , as the paper says after the display, and $K$ is the number of strata.

Crop of five lines of the left column. The text reads: ies that emphasize the importance of cognitive skills or the experience of diversity acknowledge the mediating role of political knowledge (Nie, Junn, and Stehlik-Berry 1996; Stouffer 1955), followed by a footnote mark 3 and the first line of the next paragraph. A red outline surrounds the word Stehlik-Berry. — Introduction, p. 464 (PDF p. 2), left column, first paragraph. The outline marks the misspelled surname in the citation.

Crop of the right column around equation (1). Two lines end with: adding dummy variables for all strata but one. Then: Thus, our model has the form, and the display Y sub i equals beta sub 0 plus beta sub 1 X sub i plus the sum over k from 1 to K minus 1 of gamma sub k D sub k plus u sub i, numbered (1). A red outline surrounds D sub k. Below, the text reads: where X sub i is scored 0 for control and 1 for treatment, the K-1 dummy variables marking each of the K strata are denoted D sub i, and u sub i denotes the unobserved disturbance term. A second red outline surrounds D sub i. — Statistical Model, p. 471 (PDF p. 9), right column. The outlines mark the stratum indicator $D_k$ in equation (1) and $D_i$ in the sentence that explains it. The sentence above the display says that there is an indicator for all strata but one.

P043-E009 Minor

The R script computes the 2007–2009 correlations of Table A5 with pairwise deletion, where the appendix specifies listwise deletion ( $N = 387$ ); the option should be `complete.obs`.

Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, line 698; the rule that the line should implement is in the note to Table A5, Online Appendix, p. 9

As printed

R script, line 698. cor(A, use="pairwise.complete.obs")

Note to Table A5. Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387).

The problem

The data frame A holds the nine indices of the 2007, 2008 and 2009 surveys whose correlations fill the upper triangle of Table A5. With the option pairwise.complete.obs, each correlation is computed on the students observed on that pair of indices. These samples range from 413 to 1045 students. The common sample with all nine indices observed has 387. The printed table follows the listwise rule of its note, and the Stata script computes the same matrix with a command that deletes listwise. The defect is therefore in the R script alone. With the indices built as in the script, line 698 returns a matrix in which all 36 correlations differ from the listwise ones, by as much as 0.101.

Evidence an exact counterexample

The evidence is an exact counterexample from the replication data. Let $x$ and $y$ denote the Civil Liberties Knowledge indices of 2007 and 2008. For a sample of size $N$ , define $U = N \sum xy - \sum x \sum y$ , $V = N \sum x^2 - (\sum x)^2$ and $W = N \sum y^2 - (\sum y)^2$ . Then the correlation is $r = U/\sqrt{VW}$ .

Civil Liberties Knowledge 2007 and 2008: the correlation under the two deletion rules.
Quantity	Pairwise deletion	Listwise deletion
$N$	551	387
$\sum x$	4124	2926
$\sum y$	4330	3054
$\sum x^2$	34292	24456
$\sum y^2$	38220	27286
$\sum xy$	34202	24401
$U$	988382	507183
$V$	1887516	902996
$W$	2310320	1232766
$r$	0.4733	0.4807

The table prints 0.481, the listwise value. With complete.obs, 32 of the 36 printed correlations of the upper triangle agree with the recalculation to three decimals, against none with pairwise deletion. The four others differ by 0.001, and the cause of these differences is not established.

What survives

The sample size $N = 387$ in Table A5 agrees with the stated listwise rule. Line 698 only displays the matrix. Its result is not stored or used again, and the treatment-effect models do not depend on it. The correction concerns the deletion rule alone; the composition of the 2009 support index in the same matrix is the subject of P043-E003.

Corrected

R script, line 698. cor(A, use="complete.obs")

Listing of lines 696 to 698 in monospaced type with line numbers. Line 696 builds the data frame A from nine columns: BBRIknowE, CBRIknowE, DBRIknowE, BGENknowE, CGENknowE, DGENknowE, Btolerance, Ctolerance and Dtolerance. Line 697 names the nine columns. Line 698 reads cor(A, use="pairwise.complete.obs"). A red outline surrounds use="pairwise.complete.obs". — Replication files, R script `Green_et_al_JP_2011_Public_Use_EDITED.R`, lines 696–698, typeset from the file with line numbers added and long lines wrapped. The outline marks the option that selects pairwise deletion.

Crop of three lines. The text reads: Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387). A red outline surrounds the words using listwise deletion (N = 387). — Online Appendix, p. 9, the note under Table A5. The outline marks the deletion rule and the sample size that the R script should implement.

P043-E010 Minor

The R program in the replication files omits the subtraction of 7 when it builds the seven-item support score of the 2010 survey, and its score runs from 7 to 14 where the paper’s index runs from 0 to 7; the mean of 3.95 printed in Table A4 is correct.

Replication files, R program Green_et_al_JP_2011_Public_Use_EDITED.R, lines 847–849 (2010 survey, score Ctolerance); in the paper, Table 3, p. 472 (PDF p. 10), 2008 support index with range 0–7, and Table A4 of the online appendix, p. 8, mean 3.95

As printed

R program, lines 847–849:

vars <- paste("seq", c("28pW", "29p", "30pW","34p","35p","36pW", "37p"), sep="")

command <- paste("data.E$Ctolerance <-", paste("data.E$",vars,sep="",collapse="+"))

eval(parse(text=command))

Table A4 of the online appendix, p. 8, Posttreatment Survey 2008 Indices, Support for Civil Liberties Index: Number of Questions 7; Mean (2010 Survey) 3.95; Standard Deviation (2010 Survey) 1.44; Cronbach’s Alpha (2010 Survey) 0.44.

The problem

The paper’s support index for 2008 is a sum of seven items, each rescaled to lie between 0 and 1, and it runs from 0 to 7 (Table 3). In this part of the R program the seven items keep their original codes of 1 and 2. Three of them are reversed by $x\mapsto 3-x$ , which preserves these codes, and the program then adds the seven items without subtracting 7. The score that it computes is the documented score plus 7. The Stata program in the same files subtracts 7 (line 777), and so does the R program where it builds the same index for the main experiment (line 150).

Evidence an exact counterexample

The evidence is an exact identity and a recalculation. Let $u_1,\dots,u_7$ be the item codes after the three reversals. Each lies in $[1,2]$ . It is 1 or 2 for an answered item, and a value between 1 and 2 where a non-response coded 9 was replaced by the mean of the item. The R score is $S_R=\sum_{j=1}^{7}u_j$ , and the documented score is

S_0=\sum_{j=1}^{7}(u_j-1)=S_R-7 .

For example, the answers $(2,1,2,1,1,2,1)$ to items 28, 29, 30, 34, 35, 36 and 37 give $u_j=1$ for every $j$ , an R score of 7 and a documented score of 0. On the 251 respondents of the 2010 file who have a score, the lines as deposited give a mean of $10.950642$ and a range of 8 to 14. With the subtraction the mean is $3.950642$ and the range is 1 to 7. The standard deviation is $1.4405$ and Cronbach’s alpha is $0.4394$ in both cases. These values round to the 3.95, 1.44 and 0.44 printed in Table A4.

What survives

The mean of 3.95, the standard deviation of 1.44 and the alpha of 0.44 in Table A4 stand as printed. A constant shift leaves standard deviations, alpha and correlations unchanged, including the correlation matrix that the R program computes from this score. The support index of the main experiment is built separately, with the subtraction, and the models of Table 4 are fitted before the 2010 file is read. This line therefore affects none of the experimental results.

Corrected

R program, line 848:

command <- paste("data.E$Ctolerance <-", paste("data.E$",vars,sep="",collapse="+"), "-7")

With this line the score runs from 0 to 7, and its mean on the 2010 file is $3.950642$ , which rounds to the 3.95 of Table A4. No entry of Table A4 changes.

Listing of lines 843 to 849 in monospaced type with line numbers. Lines 843 to 845 reverse three items by subtracting them from 3. A red box marks line 848, which assembles the command that assigns to data.E Ctolerance the sum of the seven items joined by plus signs, with no subtraction. — Replication files, R program `Green_et_al_JP_2011_Public_Use_EDITED.R`, lines 843–849, typeset from the file with line numbers added. The marked line builds the score as the sum of seven items coded from 1 to 2 and subtracts nothing.

Listing of lines 773 to 777 in monospaced type with line numbers. Line 777 generates Ctolerance as the sum of the same seven items. A red box marks the term 'minus 7' at the end of that line. — Replication files, Stata program `Green_et_al_JP_2011_Public_Use_EDITED.do`, lines 773–777, typeset from the file with line numbers added. The marked term is the subtraction that the R program omits.

Crop of the block 'Posttreatment Survey 2008 Indices' of Table A4, with columns for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index. A red box marks the entry 3.95 in the row 'Mean (2010 Survey)' of the support column. The same column shows 7 questions, standard deviation 1.44 and alpha 0.44 for the 2010 survey. — Online appendix, Table A4, p. 8, block for the 2008 indices. The marked mean of the seven-item support index in the 2010 survey is correct as printed; the R program as deposited gives 10.95 for it.

P043-E011 Minor

The pretest knowledge score that the replication files use as a moderator adds the raw True/False response codes of questions 18 to 24 and is not a count of correct answers; scored by correctness, the six interactions remain non-significant in a recalculation on the public file.

Replication files, Green_et_al_JP_2011_Public_Use_EDITED.do, lines 31–37 and 358–366, with the same construction in Green_et_al_JP_2011_Public_Use_EDITED.R; the paper’s counterparts are the sentence on heterogeneous treatment effects, p. 475 (PDF p. 13), left column, and footnote 14

As printed

Stata script, lines 31–37 and 358. forvalues i = 18/24 {

gen Aq`i'p = Aq`i'

replace Aq`i'p = 1 if Aq`i' == 9

replace Aq`i'p = . if Aq`i' == 10

}

gen ATOTknow = Aq18p + Aq19p + Aq20p + Aq21p + Aq22p + Aq23p + Aq24p

gen trpreknow = treatment*ATOTknow

p. 475. Our search for heterogeneous treatment effects came up with no convincing evidence of subgroup differences; partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother’s level of education, employment status and seven factual questions yielded no significant interactions in models predicting knowledge or support for civil liberties.

The problem

In the coding of the pretreatment survey, each of the questions 18 to 24 has the code 1 for True, 2 for False and 9 for no answer or both boxes checked. These are response labels and not marks of correctness. The correct answer is False for questions 18, 20, 21, 22 and 24 and True for questions 19 and 23. The script copies the raw code, sets the code 9 to 1, which is the code for True, and adds the seven values. For a fully answered survey the sum is 7 plus the number of False responses. A correct answer raises the sum on five questions and lowers it on the other two. The variable ATOTknow is therefore not a knowledge score, and the six regressions that interact it with the treatment do not test the moderator that the text names.

Evidence an exact counterexample

The evidence is a derivation, an exact counterexample and a recalculation on the public file. Let $r_i \in \{1,2\}$ be the response code of question $i$ , let $C_F$ be the number of correct answers among the five questions whose answer is False, and let $C_T$ be the number among the two whose answer is True. The script’s sum is

S = \sum_{i=18}^{24} r_i = 7 + \#\{i : r_i = 2\} = 9 + C_F - C_T ,

and the number of correct answers is $K = C_F + C_T$ . The response patterns $(2,1,2,2,2,1,2)$ and $(2,2,1,2,2,1,2)$ both have $S = 12$ ; the first has $K = 7$ and the second $K = 5$ . These two patterns imply that $K$ is not a function of $S$ , and no rescaling of $S$ recovers it. The pattern $(2,2,2,2,2,2,2)$ has $S = 14$ and $K = 5$ . Among the 1095 students of the public file with a pretest, the correlation of $S$ and $K$ is 0.669.

For the recalculation, I fit the interaction models of the R script with each score, and compute Wald $p$ -values for the interaction.

Interaction of treatment with the pretest score: $p$ -values with the script’s sum $S$ and with the number correct $K$ , and Holm-adjusted $p$ -values for $K$ within the six tests.
Outcome	$n$	$p$ with $S$	$p$ with $K$	Holm $p$ with $K$
Support for civil liberties, 2007	974	0.242	0.074	0.443
Support for civil liberties, 2008	553	0.767	0.126	0.505
Support for civil liberties, 2009	478	0.194	0.761	1.000
Civil liberties knowledge, 2007	974	0.026	0.095	0.475
Civil liberties knowledge, 2008	553	0.382	0.977	1.000
Civil liberties knowledge, 2009	478	0.244	0.569	1.000

With $K$ , the six $p$ -values exceed 0.05, and none is significant after the Holm–Bonferroni correction in any family that contains them. The models follow the R specification and not the Stata estimator. The original family of comparisons, which included the other moderators, was not rebuilt.

What survives

The adjusted models of Table 4 enter the seven questions as separate categorical controls and never use the sum, and the unadjusted models omit the questions. No entry of Table 4 changes. The reported absence of significant interactions holds for pretest knowledge under the corrected score in the recalculation. The moderator and the six auxiliary regressions change (lines 360–362 and 364–366 of the Stata script, and the matching loop of the R script), and their coefficients and $p$ -values must be recomputed.

Corrected

Score questions 18 to 24 by correctness, and do not add the raw True/False codes. With the key $(k_{18},\dots,k_{24}) = (2,1,2,2,2,1,2)$ and the raw code $r_i$ of question $i$ , set

K = \sum_{i=18}^{24} \colorbox{#ffe58a}{$\displaystyle \mathbf{1}\{r_i = k_i\}$} .

A response with the code 9 (no answer or both boxes checked) receives zero credit. For a student with no pretreatment survey, which the script codes as 10, $K$ is missing. Replace ATOTknow by $K$ in the interaction with the treatment, fit the six regressions again, and apply the Holm–Bonferroni correction of footnote 14. In the recalculation on the public file, the six $p$ -values with the corrected score exceed 0.05. The adjusted models of Table 4 use the questions as separate categorical controls and do not change.

Crop of seven lines of text. The passage reads: partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother's level of education, employment status and seven factual questions yielded no significant interactions in models predicting knowledge or support for civil liberties, followed by the footnote mark 14 and the first line of the next paragraph, Another interpretation is that knowledge and. A red outline surrounds the words seven factual questions. — p. 475 (PDF p. 13), left column. The outline marks “seven factual questions”, the moderator that the replication files build as a sum of raw response codes.

Crop of two survey questions with their codes. Question 19 reads: If Congress passes a law and the President vetoes it, Congress can enact it anyway with a two-thirds majority vote. Question 20 reads: The Constitution created a weaker central government than the Articles of Confederation. Under each question three codes are listed: 1 True, 2 False, 9 No answer or both checked. Red outlines surround the lines 1 True and 2 False under each of the two questions. — Coding document of the pretreatment survey in the replication files, p. 4, questions 19 and 20. The outlines mark the response codes, 1 for True and 2 for False, under both questions. The statement of question 19 is true and that of question 20 is false, so the code of the correct answer is 1 for the first and 2 for the second.

P043-E012 Minor

Two cleanup statements in the R script name columns that do not exist, and executing those statements stops with an error before the 2008 and 2009 support indices are built; deleting the two statements removes the error.

Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 144 and 169

As printed

Line 144. data <- subset(data, select= c(-Cq25no9,-Cq25avg,…,-Cq33no9,-Cq33avg))

Line 169. data <- subset(data, select= c(-Dq11no9,-Dq11avg,…,-Dq44no9,-Dq44avg))

Each statement lists fourteen names, abbreviated here: the suffixes no9 and avg attached to Cq25, Cq27, Cq28, Cq29, Cq31, Cq32, Cq33 in line 144, and to Dq11, Dq13, Dq15, Dq17, Dq18, Dq39b, Dq44 in line 169.

The problem

Each statement is meant to drop temporary columns after the loop that recodes the support items. The Stata script creates columns with these names and then drops them. The R loops assign the item means directly and never create them, and the data file does not contain them. In R, subset does not ignore a name that is absent, and the statement stops with the error “object 'Cq25no9' not found”. When execution reaches line 144, that statement stops it before the 2008 support index (line 150), the 2009 support index (line 173) and the models of Table 4 (from line 181). With line 144 removed, line 169 fails in the same way.

Evidence an exact counterexample

The evidence is an exact counterexample and a check on the replication data. Let the data frame have the seven columns Cq25, …, Cq33, each equal to $(1, 9)$ . After lines 135–142 are run unchanged, every recoded column equals $(1, 1)$ and no column with the suffix no9 or avg exists. Line 144 then raises the error. The same holds for lines 159–167 and line 169 with the seven Dq columns. I checked the data transformations using the replication data (1215 rows, 128 columns), omitting package loads and reporting commands. All fourteen names of each statement are absent, before and after the preceding transformations, and each statement raises the error. With both statements deleted, those transformations build the three support indices, with 1045, 591 and 508 nonmissing values. Removing only the columns that exist gives an identical data frame.

What survives

No printed number, index definition or conclusion is in question, and the Stata script does not have this defect. The correction removes these two errors only. It does not by itself make the R script reproduce the tables; other differences between the R script and the printed analysis are described in P043-E009, P043-N020 and P043-N021.

Corrected

Delete lines 144 and 169 of the R script. Alternatively, replace each by a statement that removes only the columns that exist, data <- data[, !(names(data) %in% cleanup_names), drop=FALSE], where cleanup_names is the character vector of the fourteen names of the original statement.

Listing of lines 140 to 146 in monospaced type with line numbers. Lines 140 to 142 end the loop over the 2008 support items: a command that assigns the item mean to responses coded 9, then eval(parse(text=command)) and a closing brace. A red outline surrounds line 144: data is assigned subset(data, select= c(-Cq25no9, -Cq25avg, -Cq27no9, -Cq27avg, -Cq28no9, -Cq28avg, -Cq29no9, -Cq29avg, -Cq31no9, -Cq31avg, -Cq32no9, -Cq32avg, -Cq33no9, -Cq33avg)). Line 146 assigns Cq28pW as 3 minus Cq28p. — Replication files, R script `Green_et_al_JP_2011_Public_Use_EDITED.R`, lines 140–146, typeset from the file with line numbers added and long lines wrapped. The outline marks line 144, which drops fourteen columns that the loop above does not create.

Listing of lines 165 to 171 in monospaced type with line numbers. Lines 165 to 167 end the loop over the 2009 support items. A red outline surrounds line 169: data is assigned subset(data, select= c(-Dq11no9, -Dq11avg, -Dq13no9, -Dq13avg, -Dq15no9, -Dq15avg, -Dq17no9, -Dq17avg, -Dq18no9, -Dq18avg, -Dq39bno9, -Dq39bavg, -Dq44no9, -Dq44avg)). Line 171 assigns Dq18pW as Dq18p minus 1, times one half, plus 1. — The same R script, lines 165–171, typeset in the same way. The outline marks line 169, the corresponding statement for the 2009 support items.

P043-E013 Minor

The README says that respondents under 18 were removed from the public data, but the privacy script removes no respondent: it sets every year of birth above 1989 other than the code 9999, and every missing year of birth, to 1989, and it deletes one variable from the 2010 survey file.

Replication files (public deposit of 2019): Green_et_al_JP_2011_EDITED_README.txt, lines 5 and 18; privacy script Green_et_al_JP_2011_piiremoval.do, lines 17 and 25

As printed

README, line 5. The pii variables of interest are Aq33year and seg3, which are found in the following do file, R file and datasets:

README, line 18. The dta files have been edited to remove the observations where the person’s age $<$ 18.

Privacy script, line 17. replace Aq33year = 1989 if Aq33year > 1989 & Aq33year != 9999

Privacy script, line 25. drop seq3

The problem

The README misdescribes the edit in two ways. It says that observations were removed, and no command of the privacy script removes a row. The script replaces values of the year of birth, Aq33year, in the main file, and drops the variable seq3 from the file of the 2010 survey. The main file keeps all 1215 students and the 2010 file all 254 respondents. The README also names the second variable seg3; its name is seq3.

The script itself does more than its comment describes. The comment says that the year of birth is to be top-coded at 1989, so that no age below 18 in 2007 can be inferred, and that the code 9999 is left alone. In Stata a missing numeric value is larger than every number. This implies that the condition Aq33year > 1989 & Aq33year != 9999 also holds for a missing year of birth, and the script sets such a year to 1989. In the public file all 120 students who did not take the pretreatment survey have year of birth 1989, and every other pretreatment variable is missing for them. The analysis script later recodes a missing pretreatment answer to its own category; a year already set to 1989 is not recoded, and it enters the age interactions as an observed year.

Evidence an exact counterexample

The evidence is an exact counterexample and a check of the replication files. The privacy script has eight commands, none of which deletes observations. When line 17 is applied to two students born in 1988 and 1991, aged 19 and 16 in the reference year 2007, the output has two rows, with years 1988 and 1989; removing respondents under 18 would leave one row. For a missing year, both comparisons hold, and the year becomes 1989. In the public main file the year of birth takes the values 1986 (1 student), 1988 (21), 1989 (1149) and 9999 (44). Since the original files are not available, I could not observe the original values for the 120 students who missed the pretreatment survey.

What survives

This finding concerns the deposit of 2019, and it shows no error in the article. The public file keeps the 627 treatment and 588 control students and the response counts 1045, 591 and 508 of the article. The privacy edit leaves the assignment, the outcomes and every analysis without year of birth unchanged, and the analyses of the 2010 survey do not use seq3. For the models of Table 4 with covariates, the space spanned by the covariates is unchanged when only the 120 missing years are recoded, because the same 120 students are already identified by the missing-answer category of another pretreatment covariate. The public file does not preserve the original year-of-birth categories needed to reconstruct the original covariate-adjusted models, randomization check and age interactions.

Corrected

README, in place of line 18, with seq3 for seg3 in line 5. The privacy edit retains all observations in both files. In the main file, the script replaces Aq33year with 1989 whenever Aq33year is greater than 1989 and is not 9999; in Stata this condition includes numeric missing values. The code 9999 is retained. In the Survey E file, the variable seq3 is removed. The edited files contain 1215 and 254 observations, respectively.

Privacy script, line 17, if missing years of birth are to stay missing: replace Aq33year = 1989 if Aq33year > 1989 & Aq33year != 9999 & Aq33year < . The public files would then have to be regenerated from the original data, and the README would describe the corrected operation.

Listing of lines 1 to 21 of the README in monospaced type with line numbers. Lines 1 to 3 say that the data files were revised on December 3, 2019 to minimize disclosure risk and refer to the program file Green\_et\_al\_JP\_2011\_piiremoval.do for the procedure. Line 5 reads: The pii variables of interest are Aq33year and seg3, which are found in the following do file, R file and datasets. A red outline surrounds seg3. Lines 7 to 15 list the do file, the two dta files and the R file. A second red outline surrounds line 18: The dta files have been edited to remove the observations where the person's age is less than 18. Lines 19 to 21 name the two edited files. — Replication files, `Green_et_al_JP_2011_EDITED_README.txt`, lines 1–21, typeset from the file with line numbers added and long lines wrapped. The outlines mark the misnamed variable in line 5 and the sentence in line 18 that says observations were removed.

Listing of lines 10 to 25 of the script in monospaced type with line numbers. Line 10 loads the main data file. Lines 11 to 16 are a comment: the variable to top code is Aq33year; 2007 is the reference year; the individual's age is to be bottom coded at 18, so the year of birth is top coded at 2007 minus 18, which is 1989; the 44 observations with a birth year of 9999 will not be top-coded. A red outline surrounds line 17: replace Aq33year = 1989 if Aq33year is greater than 1989 and Aq33year is not equal to 9999. Lines 18 and 19 save and export the file. Line 22 loads the Survey E file, and the comment in lines 23 and 24 says that the variable to drop is seq3. A second red outline surrounds line 25: drop seq3. — Replication files, privacy script `Green_et_al_JP_2011_piiremoval.do`, lines 10–25, typeset in the same way. The outlines mark the two commands that change the data: line 17 replaces years of birth, and line 25 drops a variable. Neither removes an observation.

P043-N001 Minor

Six schools, not seven, supplied completed surveys in 2008 and in all four waves; seven is the number of schools that remained in the study after 2007.

Research Sites, last paragraph, p. 466 (PDF p. 4), right column; Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence that runs from the foot of the left column to the top of the right column

As printed

p. 466. All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. Seven of the ten original schools participated in all four waves of surveys.

p. 469. The 2008 and 2009 surveys were thus completed by students in seven schools, taught by 19 of the original 25 teachers.

The problem

Three schools left the study after 2007, and seven remained. One of the seven has no completed survey in 2008. Strata C and D are the whole of this school, which has 86 assigned students. Table 2 shows that neither arm of either stratum has a 2008 respondent, and that the school has 11 respondents in 2009. Completed surveys come from six schools in 2008, from seven in 2009, and from six in every wave.

On p. 466, “participated” can be read loosely as “remained in the study”, and with that meaning seven is correct. The sentence on p. 469 says that the 2008 and 2009 surveys were completed by students in seven schools, and for 2008 this is false. The data show no completed 2008 survey for this school. They do not show whether the survey was attempted there, or why none was returned.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data. I counted the completed surveys by school and by wave. The same counts by stratum and arm agree with all 240 numeric cells of Table 2, the class counts and the total row included.

Completed surveys by school, from the replication data. The school codes are those of the data file.
School	Strata	Assigned	Pretreatment	2007	2008	2009	Remained after 2007
1	A, L	86	79	72	49	44	yes
2	C, D	86	80	66	0	11	yes
3	E, F	65	63	61	0	0	no
4	H, N	87	79	69	0	0	no
5	R, S	225	200	211	145	46	yes
6	G, P	100	81	76	0	0	no
7	I, J	146	144	129	112	112	yes
8	K, M	121	102	90	63	73	yes
9	B, T	120	102	105	88	89	yes
10	Q	179	165	166	134	133	yes

For school 2, the cells of Table 2 for strata C and D (control and treatment) give $20 + 24 + 27 + 15 = 86$ assigned, $0 + 0 + 0 + 0 = 0$ respondents in 2008 and $0 + 2 + 5 + 4 = 11$ in 2009.

What survives

The wording correction leaves Table 2, the sample sizes 1045, 591 and 508, and the reported estimates unchanged; it changes no survey responses or model specification. The 2009 response rates among the seven schools that remained also stand: $253/477 = 53.0\%$ and $255/486 = 52.5\%$ . Only the two sentences change.

Corrected

p. 466. Seven of the ten original schools remained in the study after 2007; six supplied completed surveys in all four waves.

p. 469. The 2008 and 2009 surveys were thus completed by students in six and seven schools, respectively, …

The number of teachers in the rest of the second sentence is corrected separately in P043-N004.

Crop of five lines of the right column. The text reads: who go on to college ranges widely, from 28\% to 88\%. All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. Seven of the ten original schools participated in all four waves of surveys. Attrition occurred in schools IV and VI. A red outline surrounds the word Seven. — Research Sites, last paragraph, p. 466 (PDF p. 4), right column. The outline marks the number of schools said to have participated in all four waves of surveys.

Crop of the first three lines of the right column. The text reads: completed by students in seven schools, taught by 19 of the original 25 teachers. Teachers whose students participated in the follow-up study had an average. A red outline surrounds the words seven schools. — Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), top of the right column. The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The outline marks the number of schools.

P043-N004 Minor

The seven continuing schools had 18 of the original 25 teachers, not 19.

Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence running from the foot of the left column to the top of the right column

As printed

“The 2008 and 2009 surveys were thus completed by students in seven schools, taught by 19 of the original 25 teachers.”

The problem

The replication data identify 25 teachers in 10 schools. The three schools that left the study after the 2007 survey had 3, 2 and 2 teachers, which leaves $25-7=18$ teachers in the seven continuing schools. Counting the teachers of the students who answered a follow-up survey does not give 19 either.

Evidence a recalculation from the source

This is a recalculation from the replication data. Each teacher identifier belongs to one school, and the seven continuing schools have $1+2+6+2+1+3+3=18$ teachers. The students who completed the 2008 survey had 16 of these teachers, those who completed the 2009 survey had all 18, and the two groups together had 18. The same data reproduce every entry of Table 2, including its totals.

What survives

The 25 original teachers, the 10 schools, the seven continuing schools, the 19 strata and the counts of Table 2 are unchanged. No model in the replication code uses the number of teachers, and no estimate or standard error changes. The ages and other attributes of the teachers are not in the replication data, and the description of the follow-up teachers in the next sentences of the paper was not checked.

Corrected

“The 2008 and 2009 surveys were thus completed by students in seven schools, taught by 18 of the original 25 teachers.”

Crop of the page number 469 and the first five lines of the right column. A red box marks the number 19 at the end of the first line, 'completed by students in seven schools, taught by 19', which continues 'of the original 25 teachers'. — Top of the right column, p. 469 (PDF p. 7). The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The marked number is the count of teachers in the continuing schools.

P043-N006 Minor

The example of footnote 8 describes Stratum I (and equally J or K), not Stratum H.

Footnote 8, p. 469 (PDF p. 7)

As printed

“Consider, for example, Stratum H, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.”

The problem

In the replication data, Stratum H has two teachers. One taught its single control course (10 students) and the other taught both of its treatment courses (31 students). The footnote describes one instructor with three courses, two of them treated. Strata I, J and K have this composition.

Evidence a recalculation from the source

This is a recalculation from the replication data, tabulated by stratum, teacher and assignment. The numbers of courses and students that it gives for strata H, I, J and K agree with Table 2 (p. 470).

What survives

The footnote’s account of assignment within a stratum is unchanged, and the definition in the text allows a stratum taught by a cluster of teachers. Nothing else depends on the label. The check does not establish how the courses of Stratum H were assigned.

Corrected

“Consider, for example, Stratum I, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.”

Crop of the last two lines of the left column and footnote 8 below them. A red box marks the words 'Stratum H' in the first line of the footnote, 'Consider, for example, Stratum H, consisting of three courses taught by an instructor'. — Footnote 8, p. 469 (PDF p. 7). The marked stratum is named as the example of one instructor with three courses.

P043-N010 Minor

The essay commands in the replication files are not restricted to the students who took each survey, and they do not return the estimates of footnote 13; the printed estimates are not shown to be wrong.

Footnote 13, p. 473 (PDF p. 11); the essay commands of the replication files: lines 483 and 485 of the Stata script Green_et_al_JP_2011_Public_Use_EDITED.do, and lines 613 and 615 of the R script Green_et_al_JP_2011_Public_Use_EDITED.R

As printed

Footnote 13, p. 473. On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128).

Stata script, line 483 (broken here after i.Aq21); line 485 is the same command for Copenended.

\begin{aligned} &\texttt{xi: xtreg Bopenended treat i.stratum i.Aq1 i.Aq3 i.Aq4 i.Aq18 i.Aq19 i.Aq20 i.Aq21}\\ &\qquad\texttt{i.Aq22 i.Aq23 i.Aq24 i.Aq28 i.Aq32 i.Aq33year, re i(classcode)} \end{aligned}

R script, lines 613 and 615.

\begin{aligned} &\texttt{reg.B.open <- lmer(Bopenended \textasciitilde{} treatment + stratum + (1|classcode), }\colorbox{#ffe58a}{$\displaystyle \texttt{data=data}$}\texttt{)}\\ &\texttt{reg.C.open <- lmer(Copenended \textasciitilde{} treatment + stratum + (1|classcode), }\colorbox{#ffe58a}{$\displaystyle \texttt{data=data}$}\texttt{)} \end{aligned}

The problem

Footnote 12 says that the analyses include all students who took each survey, and Table 3 reports 1045 such students in 2007 and 591 in 2008. Neither script restricts the essay regressions to them. On the public file, the 2007 command selects 1014 students. It includes the 170 students flagged as not having taken the 2007 survey, whose essay score is zero, and it excludes 201 students who took the survey and whose essay score is missing. The 2008 command selects all 1215 students, of whom 624 are flagged as not having taken the 2008 survey. The R formulas also omit the pretest covariates of the Stata commands (see P043-N021). As written, the commands do not return the two pairs printed in footnote 13.

Evidence a recalculation from the source

The evidence is a recalculation on the public file. The R formulas were run as written. Stata was not available, and the lines marked OLS are ordinary least squares with the fixed regressors of the Stata commands and conventional standard errors.

Effect of treatment on the essay score: footnote 13 and recalculations on the public file.
Wave	Calculation	$N$	Estimate	SE
2007	Footnote 13		.182	0.068
2007	R formula as written	1014	0.139609	0.076139
2007	OLS, sample of the Stata command	1014	0.123438	0.068956
2007	OLS, students who took the survey, missing scores set to zero	1045	0.181744	0.067518
2008	Footnote 13		$-.007$	0.128
2008	R formula as written	1215	$-0.002728$	0.100683
2008	OLS, sample of the Stata command	1215	$-0.007565$	0.070603
2008	OLS, students who took the survey	591	$-0.007703$	0.128322

The 2007 pair is recovered at printed precision on the 1045 students who took the survey, with their 201 missing scores set to zero. The 2008 standard error is recovered on the 591 students who took the survey; the estimate there rounds to $-.008$ and not to $-.007$ . Note that setting missing scores to zero is a diagnostic coding and not an established rule for the original analysis. The birth-year covariate of the public file was top-coded in 2019 (see P043-E013), which changes one set of regressors.

What survives

The estimates of footnote 13 are not shown to be wrong, and the 2007 pair is recovered at printed precision under one coding of the students who took the survey. The models of Table 4 use other outcome variables and other commands, and they do not depend on the essay scores. The essay commands of the replication files change.

Corrected

In the Stata script, each essay command takes the students who took that survey:

\begin{aligned} &\texttt{xi: xtreg Bopenended treat} \;\ldots\; \texttt{i.Aq33year }\colorbox{#ffe58a}{$\displaystyle \texttt{if Bcompleted == 1}$}\texttt{, re i(classcode)}\\ &\texttt{xi: xtreg Copenended treat} \;\ldots\; \texttt{i.Aq33year }\colorbox{#ffe58a}{$\displaystyle \texttt{if Ccompleted == 1}$}\texttt{, re i(classcode)} \end{aligned}

The R calls take the same two subsets. The files should also document how the essay score was built, how a missing essay is scored, the fixed covariates and the variance estimator. These filters alone do not reproduce both printed pairs, and the numbers of footnote 13 stay as printed.

Crop of the end of footnote 13. The text reads: of relevant constitutional principles identified. In the 2007 posttreatment survey, students in the treatment group performed significantly better than students in the control group, indicating that students were able to apply the constitutional knowledge they had gained. On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128). This finding comports with the general trend of declining differences between treatment and control groups over time. Red outlines surround .182, (SE=0.068) and (-.007 rights, SE = 0.128). — Footnote 13, p. 473 (PDF p. 11), last ten lines. The outlines mark the two essay estimates with their standard errors, which the essay commands of the replication files do not return.

P043-N015 Minor

The replication scripts interact the treatment with one summed pretest score where the text describes each of seven factual questions, and they contain no Holm–Bonferroni adjustment, no balance simulation and no code for Table A6; the scripts cannot reproduce these analyses, and no published result is shown to be wrong.

Footnote 14 and the sentence on heterogeneous treatment effects, p. 475 (PDF p. 13), left column; also the balance check on p. 468 (PDF p. 6), footnote 12 on p. 473 (PDF p. 11) and Table A6, p. 10 of the online appendix; read against the Stata and R scripts of the replication files

As printed

Footnote 14, p. 475. We tested for heterogeneous treatment effects by interacting the treatment term with each of the individual-level covariates. None of these effects were significant after taking into account multiple comparisons using the Holm-Bonferroni method.

The problem

Each script fits one interaction model for each of seven moderators and six outcomes, 42 models in all. The seven factual questions enter through a single moderator, their sum ATOTknow, and not through seven separate interactions; the scoring of this sum is the subject of P043-E011. Neither script computes a Holm–Bonferroni adjustment. The scripts also contain no code for the balance check with 20,000 simulated assignments (p. 468), and none for the analysis of Table A6, which is restricted to the 387 students present in all three posttreatment waves. These analyses cannot be reproduced from the scripts as the paper describes them.

Evidence a check of the printed page

The evidence is a check of the pages and of the two scripts. I read the heterogeneity blocks (lines 313–408 of the Stata script and lines 283–440 of the R script). I also searched both scripts for a multiplicity adjustment, for simulated assignments and for a restriction to the three-wave sample, and found none. The check does not show that the analyses were not performed outside these scripts. The top-coded year of birth in the public file also prevents an exact reconstruction of the original interactions with age.

What survives

No published result is refuted, and the code for the main treatment effects is separate from the missing blocks. The replication files change: the family of heterogeneity tests as described, its Holm–Bonferroni adjustment, the balance check and Table A6 are at present not reproducible from them.

Corrected

The replication files should supply the code and output for the interactions of the treatment with each of the seven factual questions, state the family of comparisons to which the Holm–Bonferroni adjustment was applied and the adjustment itself, and supply the code for the balance simulation and for Table A6. The description in the paper changes only if the original analysis is found to differ from it.

Crop of footnote 14, four lines. The footnote reads: We tested for heterogeneous treatment effects by interacting the treatment term with each of the individual-level covariates. None of these effects were significant after taking into account multiple comparisons using the Holm-Bonferroni method. Red outlines surround the words each of the individual-level covariates and the words Holm-Bonferroni method. — Footnote 14, p. 475 (PDF p. 13). The outlines mark the two parts of the description that the replication scripts do not implement: interactions with each individual-level covariate, and the Holm–Bonferroni adjustment.

P043-N016 Minor

The page range of Galston (2001) is 217–34, not 217–41.

References, entry Galston 2001, p. 475 (PDF p. 13), right column

As printed

Galston, William A. 2001. “Political Knowledge, Political Engagement, and Civic Education.” Annual Review of Political Science 4 (1): 217–41.

The problem

The cited article ends on page 234 and not on page 241.

Evidence a check of the printed page

The evidence is a check of the page and of the publisher’s page for the article (Annual Reviews, DOI 10.1146/annurev.polisci.4.1.217), which reads “Vol. 4:217-234”.

What survives

The author, title, year and volume identify the work, and nothing in the paper depends on the page range.

Corrected

Galston, William A. 2001. “Political Knowledge, Political Engagement, and Civic Education.” Annual Review of Political Science 4 (1): 217–34.

Crop of seven lines of the reference list. The last line of the preceding entry ends with Political Psychology 26 (3): 333-64. The next entry reads: Galston, William A. 2001. Political Knowledge, Political Engagement, and Civic Education. Annual Review of Political Science 4 (1): 217-41. It is followed by the entry Galston, William A. 2007. Civic Knowledge, Civic Education, and Civic Engagement: A Summary of Recent Research. International Journal of Public Administration 30: 623-42. A red outline surrounds the page range 217-41. — References, p. 475 (PDF p. 13), right column, the entries Galston 2001 and Galston 2007. The outline marks the page range “217–41”, which should read “217–34”.

P043-N017 Minor

Three standard deviations in Table A4 match truncation at the second decimal rather than rounding: 2.56, 1.31 and 2.81 should read 2.57, 1.32 and 2.82.

Online Appendix, p. 8, Table A4: Posttreatment Survey 2007 Indices, row “Standard Deviation (2007 Survey)”; Posttreatment Survey 2008 Indices, row “Standard Deviation (2010 Survey)”

As printed

Posttreatment Survey 2007 Indices. Standard Deviation (2007 Survey): 2.56 (Civil Liberties Knowledge Index*), 1.31 (General Political Knowledge Index).

Posttreatment Survey 2008 Indices. Standard Deviation (2010 Survey): 2.81 (Civil Liberties Knowledge Index*).

The problem

The three standard deviations are 2.568, 1.319 and 2.818 to three decimals. Rounded to two decimals they are 2.57, 1.32 and 2.82. The printed values are the same numbers with the third decimal dropped. Table 3 (p. 472) prints 1.32 for the same five-item 2007 General Political Knowledge Index.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data, with the item sets of the replication code and the sample standard deviation (denominator $N - 1$ ). It gives 2.568148678 with $N = 1045$ , 1.318642601 with $N = 1045$ , and 2.817537565 with $N = 251$ . The recalculation establishes the numerical pattern only; it does not show how the three cells came to be printed this way.

What survives

The definitions of the indices and every finding of the paper are unaffected. Each of the three cells changes by 0.01.

Corrected

Posttreatment Survey 2007 Indices. Standard Deviation (2007 Survey): 2.57 (Civil Liberties Knowledge Index*), 1.32 (General Political Knowledge Index).

Posttreatment Survey 2008 Indices. Standard Deviation (2010 Survey): 2.82 (Civil Liberties Knowledge Index*).

Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index, and Support for Civil Liberties Index. The rows are Number of Questions (13, 5, 3), Mean (2010 Survey) (7.59, 2.50, 2.35), Standard Deviation (2010 Survey) (2.21, 1.33, 0.81), Cronbach's Alpha (2010 Survey) (0.55, 0.54, 0.37), Mean (2007 Survey) (7.41, 1.93, 2.36), Standard Deviation (2007 Survey) (2.56, 1.31, 0.80) and Cronbach's Alpha (2007 Survey) (0.65, 0.48, 0.37). Red outlines surround 2.56 and 1.31. — Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2007 Indices”. The outlines mark the standard deviations 2.56 and 1.31 in the row for the 2007 survey.

P043-N018 Minor

The 2008 and 2010 versions of the General Political Knowledge Index, each with eight questions, have seven questions in common; the eighth asks for the name of the town’s head official in 2008 and for the name of the governor in 2010, and the column does not compare one fixed set of questions.

Online Appendix, p. 8, Table A4, Posttreatment Survey 2008 Indices, column “General Political Knowledge Index*”, and the note under the table; the note to Table A5, p. 9, describes the same indices

As printed

Table A4, Posttreatment Survey 2008 Indices, General Political Knowledge Index*. Number of Questions 8; Mean (2010 Survey) 4.67; Standard Deviation (2010 Survey) 1.81; Cronbach’s Alpha (2010 Survey) 0.57; Mean (2008 Survey) 3.88; Standard Deviation (2008 Survey) 1.68; Cronbach’s Alpha (2008 Survey) 0.49.

The problem

The note says that each index of 2007–2009 is restricted to the questions that were asked again in 2010. For this column that is not the case. The 2008 questionnaire asks for the title of the head official of the respondent’s town or city and then asks “What is that person’s name?” The 2010 questionnaire asks for the same title and then asks the respondent to name the governor of the state; it has no question on the name of the local official. The replication code builds the 2008 score with the question on the local official’s name and the 2010 score with the question on the governor’s name; the other seven questions are the same.

The printed statistics are correct for the two scores as built. They do not describe the same eight questions. The difference between the two means, $4.67 - 3.88 = 0.79$ , combines a difference between the two groups of respondents with a difference between two questions.

Evidence a recalculation from the source

The evidence is a check of the two questionnaires and of the replication code, and a recalculation from the replication data in exact arithmetic. The six statistics printed in the column reproduce at the printed precision for the scores as built ( $N = 591$ in 2008 and $N = 251$ in 2010). The question on the local official’s name was answered correctly by 216 of 591 respondents in 2008, and the question on the governor’s name by 186 of 251 in 2010. Each score on eight questions is the score on the seven shared questions plus the question that differs. For the 2010 mean minus the 2008 mean, I have

\frac{1172}{251} - \frac{764}{197} \;=\; \Bigl( \frac{986}{251} - \frac{692}{197} \Bigr) + \Bigl( \frac{186}{251} - \frac{72}{197} \Bigr), \qquad\text{that is,}\qquad \frac{39120}{49447} = \frac{20550}{49447} + \frac{18570}{49447} .

The three terms are approximately 0.791, 0.416 and 0.376. The two questions that differ account for $619/1304$ , about 47.5%, of the difference between the means on eight questions. This is arithmetic on two samples. It is not an estimate of what the substitution caused, since the 2010 respondents were not asked for the local official’s name.

The column as printed and the same statistics for the seven questions asked in both surveys.
Score	Questions	Mean	Standard deviation	Alpha
2008 survey, as printed	8	3.88	1.68	0.49
2010 survey, as printed	8	4.67	1.81	0.57
2008 survey, seven shared questions	7	3.51	1.47	0.43
2010 survey, seven shared questions	7	3.93	1.65	0.55

What survives

Every printed number in the column is accurate for the score it was computed from. The figure on p. 11 of the Online Appendix calibrates the Civil Liberties Knowledge indices, which contain neither of these two questions, and the treatment-effect regressions use the full 22-question 2008 index of Table 3; neither changes. What changes is the description of this column, and of the corresponding entries of Table A5, as a comparison of equivalent questions.

Corrected

Keep the entries of Tables A4 and A5 and add to both notes: The 2008 General Political Knowledge Index versions share seven questions. The eighth question asks for the town or city head official’s name in 2008 and for the governor’s name in 2010. The eight-question versions therefore contain different questions, and their mean difference does not compare one fixed set of questions.

Alternatively, compare the seven shared questions. The column of Table A4 then reads: Number of Questions 7; Mean (2010 Survey) 3.93; Standard Deviation (2010 Survey) 1.65; Cronbach’s Alpha (2010 Survey) 0.55; Mean (2008 Survey) 3.51; Standard Deviation (2008 Survey) 1.47; Cronbach’s Alpha (2008 Survey) 0.43. The sixteen correlations of Table A5 that involve this index change as follows, with the other eight indices as built in the replication code.

Correlations of the 2008 General Political Knowledge Index in Table A5, as printed and with the seven shared questions: the lower triangle (2010 survey) and the upper triangle (2007–2009 surveys).
Other index	Lower, printed	Lower, seven	Upper, printed	Upper, seven
Civil Liberties Knowledge 2007	0.440	0.436	0.411	0.414
Civil Liberties Knowledge 2008	0.559	0.532	0.468	0.444
Civil Liberties Knowledge 2009	0.616	0.610	0.420	0.426
General Political Knowledge 2007	0.615	0.509	0.487	0.441
General Political Knowledge 2009	0.486	0.481	0.408	0.389
Support for Civil Liberties 2007	0.247	0.267	0.106	0.111
Support for Civil Liberties 2008	0.215	0.256	0.133	0.129
Support for Civil Liberties 2009	0.205	0.233	0.164	0.157

Crop of four lines of notes. The text reads: asterisk, Index consists of a subset of questions originally included. Notes: All questions rescaled to 0 to 1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007 to 2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey. A red outline in two parts surrounds the last sentence, from For the 2007 to 2009 surveys to the 2010 survey. — Online Appendix, p. 8, the notes under Table A4. The outline marks the sentence that describes the 2007–2009 statistics as those of the equivalent indices made of the questions included on the 2010 survey; it runs over a line break.

P043-N019 Minor

Item 12 of the 2008 list asks which branch of the federal government can declare a law unconstitutional; it is not a question about the federal, state or local level.

Mini Appendix, p. 2 (pages unnumbered), “2008 Civil Liberties Knowledge Index Questions”, item 12

As printed

12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government]

The problem

The bracket states the type of question. In the 2008 questionnaire this item stands under the heading Separation of Powers, with the choices Executive, Legislative, Judicial, and Both Executive and Legislative, and the coding key scores Judicial as correct. The questions on the federal, state and local levels of government form a different group of items in that questionnaire. Table A2 of the Online Appendix (p. 4) labels the same item “[Which branch?]”.

Evidence a check of the printed page

The evidence is a check of the pages: the Mini Appendix, Table A2 of the Online Appendix, and the 2008 questionnaire with its coding key in the replication files.

What survives

The question, its key, the coding and the results are unaffected; only the label in the Mini Appendix changes. The numbering of the 2009 list on the same page, which passes from 1 to 3, is not an error, since the request to name two rights is scored as two answers, which Table A2 shows as (1) and (2).

Corrected

12. Declare a law unconstitutional [Identification: Which branch of the federal government?]

Crop of four lines of running text. The text reads: the prosecutor receives new evidence that he believes will result in a conviction. Does the Constitution allow the prosecutor to retry Mr. Jones in this case? (Yes/No) 12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government] 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case] 14. School. A red outline surrounds the bracket Identification: Federal, state, or local branch of government. — Mini Appendix, p. 2, list of the 2008 Civil Liberties Knowledge Index questions. The outline marks the bracket that describes item 12.

P043-N020 Minor

The R script of the replication files fits the unadjusted models of Table 4 by restricted maximum likelihood and not by the random-effects GLS with a moment estimate of the classroom variance, which reproduces the printed table; six of its nine estimates with standard errors differ from the print in the third decimal, and the printed table is supported.

Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 198–205 and 227–234; the paper’s counterparts are Table 4, the “Covariates: No” columns, p. 474 (PDF p. 12), and the description of the estimator on p. 472 (PDF p. 10)

As printed

R script, lines 198, 202 and 203 (line 202 is broken here after its third argument); lines 227–234 repeat the construction for the three support indices.

\begin{aligned} &\texttt{ivs <- c("treatment", "stratum")}\\ &\texttt{f <- as.formula(paste(name," \textasciitilde", paste(ivs, collapse="+"),}\\ &\qquad\texttt{"+ (1|classcode)", sep=""))}\\ &\texttt{command <- paste("GLS.",name," <- }\colorbox{#ffe58a}{$\displaystyle \texttt{lmer(formula=f, data=data)}$}\texttt{", sep="");} \end{aligned}

Statistical model, p. 472. We use Generalized Least Squares (GLS) regression with random effects at the classroom level to estimate average treatment effects (Green and Vavreck 2008).

The problem

The block of the R script is headed TABLE 4 and builds a table named TABLE4.nocov. It calls lmer without the argument REML, whose default is restricted maximum likelihood (REML). The Stata script of the same files fits these nine models with xtreg and the option re, which is a random-effects GLS that estimates the classroom variance by the method of moments; it is called moment-based GLS below. Both procedures are GLS with a classroom random effect in the sense of p. 472, since the REML coefficients are the GLS coefficients for the covariance that REML estimates. The sentence on p. 472 is not wrong. The two procedures estimate the classroom variance differently, and the output of the R script does not reproduce the printed table. Six of the nine estimates with standard errors differ from the print at three decimals, and the standard errors differ by up to 0.019.

Evidence a recalculation from the source

The evidence is a recalculation on the public file and an exact check. The nine models of the R script were fitted as written. The moment-based GLS of the corrected block below was computed on the same outcomes, strata, classrooms and samples.

Table 4, “Covariates: No”: treatment estimate and standard error as printed, by moment-based GLS, and from the R script. Shaded entries differ from the print at three decimals.
Index and wave	Printed	Moment-based GLS	R script (REML)
Civil liberties knowledge, 2007	1.248 (0.181)	1.248136 (0.180988)	$\colorbox{#ffe58a}{$1.250977\ (0.186482)$}$
Civil liberties knowledge, 2008	0.796 (0.228)	0.796109 (0.227714)	$\colorbox{#ffe58a}{$0.794538\ (0.246765)$}$
Civil liberties knowledge, 2009	0.492 (0.268)	0.492164 (0.267887)	0.492164 (0.267887)
General political knowledge, 2007	0.058 (0.113)	0.058406 (0.113359)	$\colorbox{#ffe58a}{$0.059166\ (0.114956)$}$
General political knowledge, 2008	0.055 (0.279)	0.054524 (0.278920)	$\colorbox{#ffe58a}{$0.059257\ (0.291241)$}$
General political knowledge, 2009	$-0.004$ (0.204)	$-0.004015$ (0.204283)	$\colorbox{#ffe58a}{$-0.003581\ (0.200669)$}$
Support for civil liberties, 2007	$-0.061$ (0.050)	$-0.060545$ (0.050440)	$-0.060545$ (0.050440)
Support for civil liberties, 2008	0.026 (0.150)	0.026044 (0.149720)	$\colorbox{#ffe58a}{$0.023627\ (0.159055)$}$
Support for civil liberties, 2009	0.022 (0.118)	0.022227 (0.117634)	0.022227 (0.117634)

The moment-based GLS agrees with all nine printed entries to rounding. For the exact check, I take the 2008 model for civil liberties knowledge, which has 591 students, 40 classrooms and a design of rank 12. Computing the moment estimates in exact rational arithmetic, I obtain the variance ratio $\lambda = \widehat\sigma_u^2/\widehat\sigma_e^2 \approx 0.000465 > 0$ . The derivative of the profiled REML criterion at this $\lambda$ is about $-80.48$ , and exact arithmetic confirms that it is negative. Since $\lambda$ is an interior point and the derivative is not zero, I conclude that the moment estimate is not a REML optimum. The two procedures differ on these data, and the differences in the table are not rounding in the optimizer. Stata was not available, and the Stata script was not run.

What survives

The nine unadjusted entries of Table 4 are supported as printed, and no entry changes. At the two-sided 5% level, the decisions under REML and under moment-based GLS agree in all nine models. The statement $p < 0.001$ for the 2008 knowledge effect (p. 473) holds for the unadjusted GLS entry in print, whose normal $p$ -value is 0.00047; under the REML fit of the R script it would be 0.0013. The adjusted columns are not touched by this finding. The R script changes.

Corrected

The Table 4 block of the R script should compute the moment-based GLS of the Stata commands in place of the call to lmer, or the replication should be run through the Stata commands. For one outcome, let classroom $i$ have $n_i$ students with outcomes $y_{ij}$ and mean $\bar y_i$ , let $G$ be the number of classrooms and $N = \sum_i n_i$ . Let $x_i$ be the column vector of regressors of classroom $i$ (the intercept, the treatment and the stratum indicators, all constant within a classroom), and let $k$ be the rank of the design. Let $b_B$ be the least-squares coefficient of $\bar y_i$ on $\colorbox{#ffe58a}{$x_i^{\top}$}$ over the $G$ classrooms. Define

\begin{aligned} \widehat\sigma_e^2 &= \frac{1}{N-G}\sum_i\sum_j (y_{ij}-\bar y_i)^2, \qquad \widehat\sigma_B^2 = \frac{1}{G-k}\sum_i (\bar y_i - x_i^{\top} b_B)^2,\\ \widehat\sigma_u^2 &= \max\Bigl\{0,\ \widehat\sigma_B^2 - \widehat\sigma_e^2\,\frac{1}{G}\sum_i \frac{1}{n_i}\Bigr\}, \qquad \theta_i = 1 - \sqrt{\frac{\widehat\sigma_e^2}{\widehat\sigma_e^2 + n_i\,\widehat\sigma_u^2}} . \end{aligned}

The estimate $b$ is the least-squares coefficient of $y^*_{ij} = y_{ij} - \theta_i \bar y_i$ on $x^*_{ij} = (1-\theta_i)\,x_i$ over the $N$ students. Its covariance matrix is $s^2 (X^{*\top}X^{*})^{-1}$ , where $X^*$ has rows $x^{*\top}_{ij}$ and $s^2 = \sum_{i}\sum_{j} (y^*_{ij} - x^{*\top}_{ij} b)^2/(N-k)$ . Setting REML=FALSE in lmer is not this repair, since it selects maximum likelihood. Table 4 stays as printed.

Crop of Table 4, Generalized Least Squares Estimates of Experimental Treatment Effects on Civil Liberties Knowledge, General Political Knowledge, and Support for Civil Liberties. Three panels, one for each index, each give six treatment estimates with standard errors in parentheses: without and with covariates for the 2007, 2008 and 2009 posttreatment surveys. Red outlines surround six entries in the columns without covariates: civil liberties knowledge 1.248 (0.181) for 2007 and 0.796 (0.228) for 2008; general political knowledge 0.058 (0.113) for 2007, 0.055 (0.279) for 2008 and -0.004 (0.204) for 2009; support for civil liberties 0.026 (0.150) for 2008. The notes below the table read: Standard errors in parentheses. Clusters for random effects model are at the classroom level. — Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the six unadjusted estimates, with their standard errors, that the R script does not return at three decimals.

P043-N021 Minor

The R script of the replication files does not fit the nine covariate-adjusted models of Table 4, and it fits the item models without the pretreatment covariates that the tables report and the essay models without those specified in the Stata script; no printed number is shown to be wrong.

Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 193 and 224, with the Table 4 block (lines 181–253), the item blocks (lines 442–583) and the essay models (lines 613 and 615); the paper’s counterparts are Table 4, the “Covariates: Yes” columns, p. 474 (PDF p. 12), Tables A1 to A3 of the online appendix, and footnote 13, p. 473 (PDF p. 11)

As printed

R script, lines 193 and 224 (line 224 is broken here after its first sentence).

\begin{aligned} &\colorbox{#ffe58a}{$\displaystyle \texttt{\#Cannot do random effects with covariates - matrices will not invert}$}\\ &\colorbox{#ffe58a}{$\displaystyle \texttt{\#Cannot perform random effects regression with covariates.}$}\\ &\qquad\colorbox{#ffe58a}{$\displaystyle \texttt{Too many dummy variables results in uninvertible matrix.}$} \end{aligned}

Table 4, notes, p. 474. $^{\dagger}$ Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother’s highest level of education, and seven factual questions measuring political knowledge.

Table A1, note, p. 2 of the online appendix. All estimates include covariates and fixed effects for strata.

The problem

The block of the R script headed TABLE 4 requests nine models, each with the treatment, the strata and a classroom random intercept. Two comments say that the models with covariates are left out because the matrices will not invert. The 17 item models for Table A1, the 49 and 37 item models for Tables A2 and A3 and the two essay models have the same right-hand side. The R script alone therefore cannot reproduce the “Covariates: Yes” columns of Table 4 or the item tables of the appendix, whose notes say that all estimates include covariates. The Stata script of the same files specifies each of these models with 13 pretreatment variables as categorical controls. The comments describe a difficulty of this R implementation. They do not show that the adjusted models cannot be fitted, and the paper does not say that the R script produced the printed adjusted entries.

Evidence a check of the printed page

The evidence is a check of the pages and of the two scripts. Evaluating the formula loops of the R script, with the fitting function replaced by one that records its formula, I obtain 114 formulas ( $9 + 17 + 49 + 37 + 2$ ), each with exactly the variables treatment, stratum and classcode. The matching Stata commands (lines 264–270 and 286–288 for Table 4, lines 419–476 for the items, lines 483 and 485 for the essays) each include the 13 variables Aq1, Aq3, Aq4, Aq18 to Aq24, Aq28, Aq32 and Aq33year. I did not fit a model or run the Stata script.

What survives

This omission does not refute the printed tables, the statistical model or the conclusions of the paper. The unadjusted specifications of the R script and the adjusted specifications of the Stata script stand. The R script changes, since at present it cannot serve to check the adjusted results.

Corrected

The R script should fit the nine covariate-adjusted models of Table 4 and include the pretreatment controls in the item and essay models. The controls are the categorical variables Aq1, Aq3, Aq4, Aq18 to Aq24, Aq28, Aq32 and Aq33year of the Stata commands, with the same coding of a missing pretest, the same handling of redundant indicator columns and the same estimator. Its output should then be compared with the Stata output on one dataset, since adding the terms to lmer does not by itself establish numerical agreement. The comments at lines 193 and 224 should be replaced by a statement of what the script does not compute. No entry of Table 4 or of Tables A1 to A3 changes.

Crop of Table 4 with its notes. Three panels, for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, each give six treatment estimates with standard errors, and rows for Covariates (No or Yes), Strata (Yes) and N. Red outlines surround the nine columns with Covariates equal to Yes: 1.220 (0.155), 0.762 (0.226) and 0.221 (0.269) for civil liberties knowledge; 0.023 (0.076), 0.051 (0.259) and -0.037 (0.202) for general political knowledge; -0.071 (0.051), -0.073 (0.131) and -0.080 (0.119) for support for civil liberties. The second note reads: Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother's highest level of education, and seven factual questions measuring political knowledge. — Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the nine covariate-adjusted models (estimate, standard error and the entry “Yes” in the Covariates row), which the R script does not fit.

Examined and not confirmed

A possible error that the checks did not establish. It is listed because it was discussed, not because it is an error.

P043-N014 Not confirmed

Whether the favorable-mediation claim omits the sign premise needed for its comparison.

p. 474 (PDF p. 12), left column below Table 4, sentence beginning “This pattern of results is striking”; the argument it refers to is on p. 473 (PDF p. 11), left column

Classification before my review (the classification when the audit was first completed, before I reviewed it): Error, Moderate (entry P043-N014).

I disputed this classification, but as a rule, deferred to the models’ final judgment: see my comment and the history of this entry.

Why it is not confirmed

The allegation was that p. 474 drops a necessary sign condition and claims that the reduced-form test always favors finding mediation. The relevant passages are:

p. 474. This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship.

p. 473. As Figure 1 suggests, estimating the reduced form provides a conservative test, because if the treatment leads to a direct increase in support for civil liberties (i.e., course content encourages support for norms of free expression and association) or indirectly through channels other than knowledge (e.g., course discussion promotes appreciation for dissenting viewpoints), we would attribute these effects to the mediating role of knowledge and understanding.

The phrase “as noted above” refers back to p. 473’s positive direct or other-channel effects. Read in context, the claim carries that hypothesized sign premise forward: the curriculum’s net effect through channels other than knowledge is nonnegative. On that premise, attributing the reduced form to knowledge weakly favors finding mediation. The published conditional is mathematically valid, so I did not establish a false statement under this natural reading.

The design does not verify the premise. Random assignment makes the treatment independent of the disturbances of Figure 1 but imposes no sign on the other-channel contribution. The null result on general political knowledge addresses one possible offset, and does not identify the net sign of every other channel. Its application to mediation consequently remains conditional; no actual negative channel or positive mediated effect in the experiment is established here.

The derivation and two exact examples explain the limitation of an unconditional reading. In a model without feedback from support to knowledge and with well-defined interventions on knowledge, let $K(t)$ be a student’s knowledge score under curriculum $t$ , with $t = 1$ for the experimental curriculum and $t = 0$ for the usual one, and let $S(t,k)$ be the support score under curriculum $t$ when knowledge is set to $k$ . Define

\begin{aligned} \tau &= \mathbb{E}\bigl[S(1,K(1)) - S(0,K(0))\bigr],\\ M &= \mathbb{E}\bigl[S(0,K(1)) - S(0,K(0))\bigr],\\ D &= \mathbb{E}\bigl[S(1,K(1)) - S(0,K(1))\bigr]. \end{aligned}

Here $\tau$ is the reduced form, $M$ is the effect through knowledge and $D$ is the net effect through the other channels. Adding and subtracting $S(0,K(1))$ gives $\tau = M + D$ . Random assignment identifies $\tau$ and puts no restriction on the sign of $D$ . The identity implies that $\tau \ge M$ if and only if $D \ge 0$ .

In the first example, $T$ takes the values $0$ and $1$ , $u_1$ the values $\pm 1/5$ and $u_2$ the values $\pm 1/10$ , each with probability $1/2$ , and $T$ , $u_1$ , $u_2$ are independent. Set $K(t) = 8 + (6/5)t + u_1$ and $S(t,k) = 23/10 + (k-8)/20 - (3/50)t + u_2$ . A direct computation gives $M = 3/50$ , $D = -3/50$ and $\tau = 0$ . The observed support score is $23/10 + u_1/20 + u_2$ in both arms.

The second example has integer scores inside the 2007 ranges of Table 3. Let $W$ be uniform on $\{1,\dots,5\}$ and $V$ uniform on $\{1,\dots,100\}$ , with $T$ , $W$ , $V$ independent. Set $K(t) = 8 + t\,(1 + \mathbf{1}\{W = 5\})$ and $S(t,k) = 2 + \mathbf{1}\{V \le 36 + 5(k-8) - 6t\}$ . These definitions give $\mathbb{E}[S(0,K(0))] = 59/25$ , $\mathbb{E}[S(0,K(1))] = 121/50$ and $\mathbb{E}[S(1,K(1))] = 59/25$ .

The two examples violate the contextual nonnegative-other-channel premise: positive mediation is canceled by a negative effect through other channels.
Quantity	First example	Second example
Mean knowledge change, $\mathbb{E}[K(1)-K(0)]$	$6/5$	$6/5$
Expected support gain, $8\to9$ or $9\to10$	$1/20$	$1/20$
Effect through knowledge, $M$	$3/50$	$3/50$
Net effect through the other channels, $D$	$-3/50$	$-3/50$
Reduced form, $\tau = M + D$	$0$	$0$

In both examples the treatment is independent of every disturbance, as Figure 1 assumes. In the second, a general-knowledge score can be held equal in the two arms, so the null result on general political knowledge does not exclude the example. Note that the examples are not fitted to the data.

Both examples have $D=-3/50<0$ . They satisfy random assignment and the unsigned diagram, but violate the hypothesized sign premise carried forward from p. 473. They therefore refute only an unconditional version of the favorable-mediation comparison, not the paper’s contextual conditional. Setting the second example’s threshold term $-6t$ to $0t$ or $+6t$ gives, respectively, $(M,D,\tau)=(3/50,0,3/50)$ and $(3/50,3/50,3/25)$ , which satisfy the weak comparison exactly.

Table 4 and the reported total-effect estimates are unchanged by this mediation argument. I separately recalculated the unadjusted estimates (P043-N020); exact reproduction of the adjusted estimates remains blocked (P043-E013). The reported gain in knowledge without a statistically significant increase in support remains a statement about curriculum total effects. The p. 472 conditional exclusion-restriction discussion, the p. 473 positive-channel argument and the Conclusion’s cautious multiple interpretations remain. I did not check the ample-power claim.

An optional clarification would repeat the premise: “under the positive other-channel effects described above, the reduced-form contrast weakly favors attributing an effect to knowledge.” More explicitly, $\tau\ge M$ if and only if $D\ge0$ , with strict inequality only for $D>0$ ; the condition concerns the net of the other channels, not each path separately. A negative net contribution can conceal positive mediation, as the examples show. This makes the existing contextual condition explicit without changing an estimator or numerical estimate.

Crop of eight lines of text. The passage reads: dissent, or due process. This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship. The theoretical implications of this finding are profound. Evidently, it is possible to increase awareness and understanding of civil liberties without producing an increase in; the crop ends in the middle of this sentence. Red outlines surround the words the statistical test is biased in favor of finding a mediating relationship, which run over two lines. — p. 474 (PDF p. 12), left column below Table 4. The outlines mark the clause that refers back to the preceding positive-other-channel explanation.

Line chart. The horizontal axis is the net effect of the curriculum on support through channels other than knowledge, D, from -3/25 to 3/25 points of the support index. The vertical axis is the reduced-form effect on support, M + D, from -3/50 to 9/50. A straight line of slope one passes through the point D = 0, reduced form 3/50. Its part with D at least 0 is a solid blue line labelled reduced form at least M, the case of p. 473. Its part with D below 0 is a dashed orange line labelled reduced form below M. A dotted horizontal line marks the effect through knowledge, M = 3/50. A dot on the dashed part at D = -3/50 and reduced form 0 is labelled exact example: D = -3/50, reduced form 0 although M = 3/50. — The identity $\tau = M + D$ with the effect through knowledge fixed at $M = 3/50$ , the value in the two examples. The reduced form is at least $M$ when $D \ge 0$ , the case described on p. 473, and smaller than $M$ when $D < 0$ . The dot marks the examples outside that sign premise, in which $D = -3/50$ and the reduced form is zero.

My comment and the history of this entry

I think the context implies the hypothesized sign of the effect that would make this claim true.

Suggested classification: clarity note.

My position My comment above; I suggested a clarity note.

The models’ position The contextual reading was accepted; under the classification rule the allegation is then not confirmed, so the entry is neither an error nor a clarity note.

Classification before my review (the classification when the audit was first completed, before I reviewed it): Moderate error.
I commented and suggested a clarity note.
Independent evaluation. Incorporated in part. The contextual reading was accepted, and the entry is now not confirmed, with no severity: under the classification rule, an allegation that is not confirmed is classed as not confirmed rather than as the clarity note I suggested.

Formal statements (Lean)

Formal results: 4 whole; 1 restricted; 0 refuted; 1 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

P043-T1

Two counterexamples to an unconditional extension of the mediation comparison; the paper’s contextual conditional is not refuted

p. 474 (PDF p. 12), left column below Table 4; the passage referred to is on p. 473 (PDF p. 11), left column

p. 474 (PDF p. 12), below Table 4, referring to p. 473 (PDF p. 11). The two specified validator constructions are exact randomized counterexamples to an unconditional mediation comparison. In the linear construction all eight states have mass 1/8, knowledge increases by 6/5, support increases by h/20 with knowledge, M=3/50 and D=-3/50, and observed support cancels assignment pointwise. Both arm laws, all covariances, population regression slopes, and structural coefficient identities are as specified. In the integer construction all 1000 states have mass 1/1000; K(0)=8 and K(1)=9 for W<=4 or 10 for W=5, with W stored as i.1.val+1. Knowledge, support and constant general knowledge respect the 2007 index ranges. All stated arm means, nested means and one-unit effects hold; both support arm means are 59/25 and both support laws put mass 9/25 at 3. In each construction D<0, M>0 and tau=0<M, so the design alone gives neither tau>=M nor the null inference M<=0. The contextual comparison assuming D>=0 is not refuted. The usual measure-theoretic conditions are assumed; the specified probability spaces are finite and all expectations exist.

Historical component credits

Counterexample proved AronowAudit.P043.mediation_counterexamples Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is a witness for the unconditional extension specified in the current target, not a refutation of the paper’s contextual D≥0 conditional: both witnesses have D<0. No prior-record fact changes the claim-stage verdict.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:d47cef1022f975a78d85677e5325b6c17e7927181277e624078da62ede8fe630

Standing assumptions: The paper is an empirical randomized study with no standalone theorem for N014. Its backward reference carries the positive-other-channel sign premise; N014 is now not_confirmed. Its finite mediation examples are retained as scope mathematics. The confirmed mathematical corrections E002 and the equation part of E008 retain their own targets. Original page transcriptions and definitions follow unchanged. (S1) Design. p. 467 (PDF p. 5): "In each high school, we randomly assigned students to exposure to a \emph{BRRL}-enhanced civics course or to a control group in which civics was taught in its usual manner." p. 468 (PDF p. 6): "The net result of this process over two semesters was 19 distinct clusters of classrooms, or strata, within which randomization occurred. Each stratum is composed of all classes in a single school that were in the same randomization pool. Within each stratum, every classroom had an equal probability of receiving the treatment. We have, in effect, nineteen distinct experiments. When estimating the treatment effect, we include fixed effects for strata to account for the fact that assignment occurred within strata but not across strata. Because classrooms, rather than specific individuals, were assigned to treatment and control groups, the statistical analysis that follows takes account of clustered random assignment for purposes of calculating standard errors." (S2) Statistical model, p. 471 (PDF p. 9), as printed: "The focus of our analysis is the treatment effect on the outcome measures listed in the previous section. The basic model is a straightforward generalization of a comparison of means across treatment and control groups. The sole complication is the need to control for strata, because assignment occurred within rather than across strata. These controls are easily implemented by adding dummy variables for all strata but one. Thus, our model has the form: $$Y_i = \beta_0 + \beta_1 X_i + \sum_{k=1}^{K-1} \gamma_k D_k + u_i, \qquad (1)$$ where $X_i$ is scored 0 for control and 1 for treatment, the $K-1$ dummy variables marking each of the $K$ strata are denoted $D_i$, and $u_i$ denotes the unobserved disturbance term. The key parameter of interest in equation (1) is $\beta_1$, which represents the average treatment effect on $Y_i$, which represents one of the three outcome measures: domain-specific knowledge, general knowledge, or support for civil liberties." [The subscripts of $D$ are as printed here; they are corrected in P043-T6.] (S3) Estimation, pp. 471-472 (PDF pp. 9-10): "In addition, because assignment was carried out at the level of the classroom rather than the level of the individual, our estimator must allow for the possibility that individuals within classrooms share unobserved attributes. We use Generalized Least Squares (GLS) regression with random effects at the classroom level to estimate average treatment effects (Green and Vavreck 2008). Because our treatment assignment is orthogonal to classroom-level effects, this method of estimation is efficient and consistent." (S4) Causal model, p. 472 (PDF p. 10): "The central theoretical question in this study is whether an exogenous increase in civil liberties knowledge and understanding produces a corresponding increase in support for civil liberties. This causal relationship is depicted in Figure 1. Unlike traditional path models, the model in Figure 1 does not stipulate a simple recursive relationship among the variables. Rather, the system of equations is agnostic about whether unobserved factors that affect support for civil liberties are correlated with unobserved factors that affect knowledge. The key feature of the diagram is the fact that the treatment, which is randomly assigned, is assumed to be statistically independent of these two disturbance terms." Figure 1 (p. 472), "Depiction of the Causal Relations among Assigned Treatment, Domain-Specific Knowledge and Support for Civil Liberties": a box 'Treatment' with one arrow to 'Domain-Specific Knowledge' and one arrow to 'Support for Civil Liberties'; a double-headed arrow between 'Domain-Specific Knowledge' and 'Support for Civil Liberties'; $u_1$ with an arrow to 'Domain-Specific Knowledge'; $u_2$ with an arrow to 'Support for Civil Liberties'; a curved double-headed arrow between $u_1$ and $u_2$. No arrow carries a sign, and the paper writes no equations for the figure. (S5) Outcome scales, Table 3, p. 472 (PDF p. 10), note "All questions rescaled to 0–1 for additive index." Ranges of the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index: Posttreatment Survey 2007: 0–15, 0–5, 0–3; 2008: 0–18, 0–22, 0–7; 2009: 0–16, 0–10, 0–7. (S6) Not in the paper. The paper uses no potential-outcome notation, states no monotonicity condition on the response of knowledge to assignment and no sign condition on the Treatment-to-Support arrow of Figure 1, and reports no instrumental-variable estimate (Table 4 reports coefficients of treatment only). The potential-outcome notation of the targets (assignment $T$ or $Z$; potential knowledge $K(t)$; potential support $S(t,k)$ or $Y(k)$) is the validators', introduced afresh in each target. Every target is an exact statement about a population (a finite probability space suffices); none is asymptotic and none concerns the sampling distribution of the GLS estimator or its standard errors. Accepted contextual scope: the sign scenario of p.473 is carried into p.474 by the backward reference; absence of a sign on the Figure 1 arrow does not turn that contextual comparison into an unconditional claim.

Effective claim (source TeX): Two exact witnesses (validator's constructions; not in the paper) in which treatment is randomized independently of all disturbances, treatment raises knowledge, knowledge raises support, and the reduced-form effect of treatment on support is nevertheless exactly zero, so that the reduced form lies strictly below the knowledge-mediated effect. Notation for both witnesses. $T \in \{0,1\}$ is assignment with $P(T=1) = 1/2$, independent of the exogenous variables. $K(t)$ is potential knowledge under assignment $t$ and $S(t,k)$ is potential support under assignment $t$ and knowledge $k$. The observed variables are $K = K(T)$ and $S = S(T, K(T))$. Define the reduced-form contrast $\tau = E[S \mid T=1] - E[S \mid T=0]$, the knowledge-mediated contrast $M = E[S(0,K(1)) - S(0,K(0))]$ and the net non-knowledge (direct) contrast $D = E[S(1,K(1)) - S(0,K(1))]$. (A) Linear witness. $u_1$ is uniform on $\{-1/5,\ 1/5\}$, $u_2$ is uniform on $\{-1/10,\ 1/10\}$, and $T$, $u_1$, $u_2$ are mutually independent, so each of the eight states $(T,u_1,u_2)$ has probability $1/8$. $$K(t) = 8 + \tfrac{6}{5}\,t + u_1, \qquad S(t,k) = \tfrac{23}{10} + \tfrac{1}{20}(k-8) - \tfrac{3}{50}\,t + u_2.$$ Claims. (A1) $K(1) - K(0) = 6/5 > 0$ in every state, and $E[K \mid T=1] - E[K \mid T=0] = 6/5$. (A2) $S(t,k+h) - S(t,k) = h/20 > 0$ for each $t$, every $k$, every $h > 0$ and every state. (A3) $M = 3/50$ and $D = -3/50$. (A4) Observed $S = \tfrac{23}{10} + \tfrac{1}{20}u_1 + u_2$ in every state; hence the conditional law of $S$ given $T=1$ equals its conditional law given $T=0$ (the values $219/100$, $221/100$, $239/100$, $241/100$, each with probability $1/4$), and $\tau = 0$. (A5) $\mathrm{Var}(T) = 1/4$, $\mathrm{Cov}(T,S) = 0$ and $\mathrm{Cov}(T,K) = 3/10$, so the population least-squares coefficient on $T$ (regression with an intercept) is $0$ for $S$ and $6/5$ for $K$. In the notation $K = \alpha + aT + u_1$, $S = \sigma + bK + cT + u_2$ this is $a = 6/5$, $b = 1/20$, $c = -3/50$, $ab = 3/50$, $ab + c = 0$. (B) Integer witness inside the 2007 index ranges. $W$ is uniform on $\{1,\dots,5\}$, $V$ is uniform on $\{1,\dots,100\}$, and $T$, $W$, $V$ are mutually independent, so each of the 1000 states $(T,W,V)$ has probability $1/1000$. $$K(t) = 8 + t\,\bigl(1 + \mathbf{1}\{W=5\}\bigr), \qquad S(t,k) = 2 + \mathbf{1}\{V \le 36 + 5(k-8) - 6t\}.$$ A general political knowledge index is identically 2 under either assignment. Claims. (B1) $K(0) = 8$; $K(1) = 9$ if $W \le 4$ and $K(1) = 10$ if $W = 5$; observed $K \in \{8,9,10\} \subseteq \{0,\dots,15\}$ and observed $S \in \{2,3\} \subseteq \{0,\dots,3\}$; $E[K \mid T=0] = 8$, $E[K \mid T=1] = 46/5$, $E[K(1) - K(0)] = 6/5$. (B2) For $t \in \{0,1\}$ and $k \in \{8,9\}$: $S(t,k+1) - S(t,k) \in \{0,1\}$ in every state, and $E[S(t,k+1) - S(t,k)] = 1/20$. (B3) $E[S(0,K(0))] = 59/25$, $E[S(0,K(1))] = 121/50$, $E[S(1,K(1))] = 59/25$; hence $M = 3/50$, $D = -3/50$ and $\tau = E[S \mid T=1] - E[S \mid T=0] = 59/25 - 59/25 = 0$. (B4) $P(S=3 \mid T=0) = P(S=3 \mid T=1) = 9/25$, so the conditional law of observed $S$ is the same in the two arms. Consequence. In each witness assignment is randomized independently of the disturbances, but the contextual sign premise D≥0 fails, knowledge mediates a strictly positive effect of treatment on support ($M = 3/50 > 0$), and $\tau = 0 < M$. So the inequality $\tau \ge M$ (the reduced form errs, if at all, in favor of finding a knowledge-mediated effect) is not a consequence of the design, and a zero reduced form does not imply $M \le 0$.

: SAP.P043.T1WholeStatement

Theorem: SAP.P043.T1

Statement hash: 664dde790d9c684237d220385d6f5d394668ce6840e9de3c97566f6dd777e224

Statement definition: SAP.P043.T1WholeStatement

Certificate. 20261104T162856Z

Declaration review: faithful; 2026-11-04T16:48:10Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P043-T2

The mediation comparison with its contextual sign condition made explicit

p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column

p. 474 (PDF p. 12), below Table 4, and p. 473 (PDF p. 11). For arbitrary probability laws with the five specified finite expectations, independent nondegenerate assignment identifies the mediation decomposition, its weak and strict sign comparisons and null inferences, the linear and positive-first-stage ratio formulas, and the sufficient but unnecessary channelwise sign condition; the fixed witness records failure without the net sign premise.

Historical component credits

Proved AronowAudit.P043.Wave1.mediation_repair_integrable Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; MediationLawClaims and LinearMediationLawClaims only package conclusions, not assumptions. The probability hypothesis and 0<p<1 avoid infinite/zero ENNReal conversion or arm division; integrability prevents the undefined-integral-equals-zero convention from supplying the conclusions. No claim-stage verdict change.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:e05581f82242343979d09079f3050571e9e37bc1f60d74f3240650ac811b86e5

Effective claim (source TeX): The contextual sign condition is made explicit: τ=M+D, τ≥M iff D≥0 and τ>M iff D>0, with the original population/assignment/consistency and linear-special-case conditions. This is clarification of an already conditional argument, not a newly required assumption or evidence about D in the experiment. Mathematical content, with every hypothesis (validator's derivation.md, section 2). Setting: a population of units with a probability law; assignment $T \in \{0,1\}$ with $0 < P(T=1) < 1$; potential knowledge $K(t)$ for $t \in \{0,1\}$ and potential support $S(t,k)$ for $t \in \{0,1\}$ and every knowledge value $k$ (an acyclic potential-outcome model with well-defined interventions on knowledge); the variables $S(0,K(0))$, $S(0,K(1))$, $S(1,K(1))$, $K(0)$ and $K(1)$ have finite expectations. (H1) Consistency: the observed variables are $K = K(T)$ and $S = S(T, K(T))$. (H2) Random assignment: $T$ is independent of all the potential outcomes $(K(0), K(1), S(\cdot,\cdot))$. Define $$\tau = E[S \mid T=1] - E[S \mid T=0], \qquad M = E[S(0,K(1)) - S(0,K(0))], \qquad D = E[S(1,K(1)) - S(0,K(1))].$$ Then: (i) $\tau = E[S(1,K(1))] - E[S(0,K(0))]$. (ii) For every unit, $S(1,K(1)) - S(0,K(0)) = [S(1,K(1)) - S(0,K(1))] + [S(0,K(1)) - S(0,K(0))]$; hence $\tau = M + D$. (iii) Corrected direction claim. If $D \ge 0$ then $\tau \ge M$. More precisely, $\tau \ge M \iff D \ge 0$, and $\tau > M \iff D > 0$. (iv) Corrected inference from a null. If $D \ge 0$ and $\tau \le 0$, then $M \le 0$; in particular $D \ge 0$ and $\tau = 0$ imply $M \le 0$. Without the hypothesis $D \ge 0$ this fails (P043-T1: $\tau = 0$, $M = 3/50$). (v) Linear special case. If $K(t) = \alpha + a\,t + u_1$ and $S(t,k) = \sigma + b\,k + c\,t + u_2$ with $u_1$, $u_2$ of finite expectation and $T$ independent of $(u_1,u_2)$, then $E[K \mid T=1] - E[K \mid T=0] = a$, $M = ab$, $D = c$ and $\tau = ab + c$; so $\tau \ge M \iff c \ge 0$. If moreover $a > 0$, the instrumental-variable ratio is $\tau/a = b + c/a$, and $\tau/a \ge b \iff c \ge 0$. If the non-knowledge effect is a sum of several channels, $c = \sum_{m} c_m$, the condition is $\sum_m c_m \ge 0$: nonnegativity of every $c_m$ is sufficient but not necessary.

.{u} : T2Statement.{u}

Theorem: SAP.P043.T2

Statement hash: e8a259b4f3611b0e92c4a4f6215e6543ea168a4b4605d642a391769d35ec0fef

Statement definition: SAP.P043.T2Statement

Certificate. 20261104T162856Z

Declaration review: faithful; 2026-11-04T16:48:10Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P043-T3

Results, sentence beginning 'This pattern of results is striking', with the reduced-form passage of the Statistical Model section that it cites; as printed

p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column

No current theorem is supplied for this target.

Historical component credits

Not applicable Excluded association; baseline fidelity: None; manifest: not applicable; Solution use unverified; Historical excluded association.

Formal result: omitted

Scope: No credited Lean declarations and no standalone formal mathematical assertion for this contextual empirical and interpretive prose. Neither a reformulation of the contextual sign comparison nor the cancellation witnesses establishes its empirical or interpretive clauses.

Omission: not formalized

Reason: No credited Lean declarations and no standalone formal mathematical assertion for this contextual empirical and interpretive prose. Neither a reformulation of the contextual sign comparison nor the cancellation witnesses establishes its empirical or interpretive clauses.

Omission explanation

Assumptions and verification

Claim revision: catalog:652b07c7c7e6bda12d42c1bbbd1fc7f975f94370ede7f20991fa2872b8e4d044

Effective claim (source TeX): As printed. Results, pp. 473-474 (PDF pp. 11-12): "The findings imply that students can become more knowledgeable about the Constitution and workings of government without experiencing a concomitant shift in their support for free speech, dissent, or due process. This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship. The theoretical implications of this finding are profound. Evidently, it is possible to increase awareness and understanding of civil liberties without producing an increase in support for those civil liberties. This finding therefore calls into question the longstanding argument that beliefs and attitudes are causally linked in this domain." The passage referred to by 'as noted above', Statistical Model, p. 473 (PDF p. 11): "A more conservative estimation approach is to examine the reduced form relationship between random assignment and support for civil liberties. This regression model takes the same form as equation (1), but now the dependent variable is the Support for Civil Liberties Index. If knowledge and understanding mediate the effects of the treatment, we should obtain a positive estimate of $\beta_1$. As Figure 1 suggests, estimating the reduced form provides a \emph{conservative} test, because if the treatment leads to a direct increase in support for civil liberties (i.e., course content encourages support for norms of free expression and association) or indirectly through channels other than knowledge (e.g., course discussion promotes appreciation for dissenting viewpoints), we would attribute these effects to the mediating role of knowledge and understanding." A qualification the paper attaches, Results, p. 473 (PDF p. 11): "The treatment had no effect, positive or negative, on general political knowledge. The estimates hover near zero in each wave, regardless of whether the model controls for covariates. This null finding is important, because if the treatment had caused a decrease in general civic knowledge, the net effect on support for civil liberties would be unclear: the effects of augmenting specific knowledge might have been offset by a deficit of general knowledge. The fact that general knowledge is unaffected by the treatment allays this concern."

Unit certificate (omission has no certified theorem). 20261104T162856Z

Declaration review: omission_ok; 2026-11-04T16:36:29Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Not supplied by this export. Manifest uses are reported separately; actual use is unverified.

Source and evidence metadata · sap-2026-11-08-assured-release

P043-T4

Statistical Model, 'then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable', and Conclusion, 'the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum'; the witnesses are the validator's constructions and are not in the paper

p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)

p. 472 (PDF p. 10) and pp. 474-475 (PDF pp. 12-13). The specified two-type and two five-type validator constructions satisfy fair independent assignment, exclusion, consistency and a strictly positive first stage. The two-type witness has the stated arm means and covariances, zero IV and covariance ratios, increaser mass 2/3 and increaser effect one. Every increaser crosses only margin 9 with increment one; every normalized nonnegative average of these increments is one, different from the zero IV ratio. This includes arbitrary probability averages, whose integrability is certified. Cancellation explains the zero reduced form, and responses are nondecreasing with the stated 2008 index bounds. The five-type models have the specified complete conditional and joint observable laws, common positive first stage 1/5 and zero reduced form and IV ratio, but increaser probabilities 2/5 and 3/5 and effects one and 4/3 respectively. Every increase is one unit, so these are also the average per-unit effects. No functional of the observable law recovers the increaser effect in both admissible models, hence none works on the entire claimed class. These are validator witnesses, not empirical claims about the curriculum. The usual measure-theoretic conditions are assumed; all finite expectations exist and every conditioning or ratio denominator used here is positive.

Historical component credits

Counterexample proved AronowAudit.P043.iv_counterexamples Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The fixed models satisfy exclusion, consistency, full assignment independence, positive first stage and the specified ranges; knowledge monotonicity is not a premise of the assertion being tested. IncreaserNonidentificationClaim is a real negated-existence conclusion for two indistinguishable laws with unequal effects, which suffices to refute identification on the larger admissible class. No claim-stage verdict change.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:f0e8ab5d0250cd80197b392f1ddcac1d24a485b944a23bb935c52b348e95bd2e

Effective claim (source TeX): Exact witnesses (validator's constructions; not in the paper) showing that random assignment, exclusion, consistency and a strictly positive first stage (a) do not make the instrumental-variable ratio an effect confined to the students whose knowledge the curriculum raises, and (b) do not identify such an effect from the observable law. Notation. $Z \in \{0,1\}$ is assignment with $P(Z=1) = 1/2$, independent of a latent type. Each type has potential knowledge scores $K(0)$, $K(1)$ and a support response $Y(k)$ that depends on the knowledge value $k$ and on the type only (exclusion: $Y(z,k) = Y(k)$ for both $z$ and every $k$). The observed variables are $K = K(Z)$ and $Y = Y(K(Z))$. $H(k) = \mathbf{1}\{k \ge 9\}$. $\Delta_K = E[K \mid Z=1] - E[K \mid Z=0]$ (first stage), $\Delta_Y = E[Y \mid Z=1] - E[Y \mid Z=0]$ (reduced form), $\beta_{IV} = \Delta_Y / \Delta_K$. The increasers are the types with $K(1) > K(0)$. (a) Zero ratio with a positive effect for every increaser. Two types: 'up', probability $2/3$, $K(0) = 8$, $K(1) = 9$, $Y(k) = 3 + H(k)$; 'down', probability $1/3$, $K(0) = 9$, $K(1) = 8$, $Y(k) = 2 + 2H(k)$. Claims. (a1) $E[K \mid Z=0] = 25/3$, $E[K \mid Z=1] = 26/3$, $\Delta_K = 1/3 > 0$. (a2) $E[Y \mid Z=0] = E[Y \mid Z=1] = 10/3$, $\Delta_Y = 0$, $\beta_{IV} = 0$; also $\mathrm{Cov}(Z,K) = 1/12$ and $\mathrm{Cov}(Z,Y) = 0$, so the covariance ratio is the same $0$. (a3) The increasers are exactly the type 'up' (probability $2/3$); each crosses the single margin $j = 9$ and has $Y(K(1)) - Y(K(0)) = Y(9) - Y(8) = 1$. Hence the average effect among increasers is $1$, and every average with nonnegative weights summing to one of the increasers' one-unit increments at the margins they cross equals $1 \ne 0 = \beta_{IV}$. (a4) The reduced form is zero by cancellation: $\Delta_Y = \tfrac{2}{3}(1) + \tfrac{1}{3}(-2) = 0$. (a5) $Y(\cdot)$ is nondecreasing in $k$ for both types; all knowledge values are in $\{8,9\} \subseteq \{0,\dots,18\}$ and all support values in $\{2,3,4\} \subseteq [0,7]$ (the 2008 ranges of Table 3). (b) Two models with the same observable law and different effects among increasers. Each listed type has probability $1/5$; a type is written $(K(0), K(1), Y(\cdot))$. Model A: $(8,9,\,3+H)$, $(8,9,\,3+H)$, $(9,8,\,2+2H)$, $(8,8,\,Y \equiv 2)$, $(9,9,\,Y \equiv 4)$. Model B: $(8,9,\,3+H)$, $(8,9,\,3+H)$, $(9,8,\,2+2H)$, $(8,9,\,2+2H)$, $(9,8,\,2+2H)$. Claims. (b1) In both models $P((K,Y) = (8,2) \mid Z=0) = 1/5$, $P((8,3) \mid Z=0) = 2/5$, $P((9,4) \mid Z=0) = 2/5$, and $P((8,2) \mid Z=1) = 2/5$, $P((9,4) \mid Z=1) = 3/5$, all other conditional probabilities being zero; with the same law of $Z$ the joint law of $(Z,K,Y)$ is therefore identical in A and B. (b2) In both models $E[K \mid Z=0] = 42/5$, $E[K \mid Z=1] = 43/5$, $\Delta_K = 1/5 > 0$, $E[Y \mid Z=0] = E[Y \mid Z=1] = 16/5$, $\Delta_Y = 0$, $\beta_{IV} = 0$. (b3) In A, $P(K(1) > K(0)) = 2/5$ and $E[Y(K(1)) - Y(K(0)) \mid K(1) > K(0)] = 1$; in B, $P(K(1) > K(0)) = 3/5$ and $E[Y(K(1)) - Y(K(0)) \mid K(1) > K(0)] = 4/3$. Every increase is of one unit, so the same two values are the average per-unit effects among increasers. (b4) Consequence: no function of the law of $(Z,K,Y)$ equals the average effect of knowledge among increasers in every model that satisfies assignment independence, exclusion, consistency and $\Delta_K > 0$.

: SAP.P043.T4WholeStatement

Theorem: SAP.P043.T4

Statement hash: 66bba5a134ec5f432603ad7cdb79bc52adc7491217b6550a7e0611c0cf3c9008

Statement definition: SAP.P043.T4WholeStatement

Certificate. 20261104T162856Z

Declaration review: faithful; 2026-11-04T16:48:10Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P043-T5

The local-effect statements of p. 472 and pp. 474-475, corrected: the instrumental-variable ratio under knowledge monotonicity

p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)

p. 472 (PDF p. 10) and pp. 474–475 (PDF pp. 12–13). Individual knowledge monotonicity and a positive first stage yield the normalized crossed-margin IV formula and its finite-population identities.

Historical component credits

Proved AronowAudit.P043.iv_repair Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P043/P043_Formalization.lean:1293 requires [Fintype Ω], and Population and mean at P043/P043_Formalization.lean:20 and :27 encode only finite weighted population laws. P043-T5 instead specifies a population "with a probability law" and finite-expectation real potential responses, without a finite-support hypothesis. For example, independent fair assignment, U uniform on [0,1], J=1, K(0)=0, K(1)=1 and Y(k)=kU satisfy H1–H5 and give β_IV=1/2, but their continuous response law cannot be represented by this finite population type. The statement that a finite probability space "suffices" supplies a possible realization, not a finiteness assumption on every population in the claim. The reviewer’s secondary signed-membership observation is only a packaging issue: the unrestricted membership statement is separately present at P043/P043_Formalization.lean:1107, and the signed telescoping and downward-crossing conclusions are exported by iv_repair.', 'suggested_fix': 'Generalize Population and the expectation, event-mean, arm-mean and covariance definitions to arbitrary probability spaces, and restate iv_repair with the measurability and finite-expectation conditions of T5. Keep the bounded integer doses, individual monotonicity, positive first stage and existing conclusions; a description-only change would not establish the current target.', 'second_review': "P043/P043_Formalization.lean:1293 requires [Fintype Ω], and Population at P043/P043_Formalization.lean:20 and mean at P043/P043_Formalization.lean:27 restrict population laws and expectations to finite weighted sums, whereas P043-T5 specifies a population 'with a probability law' and finite-expectation real potential responses without a finite-support hypothesis. Neither the standing observation that 'a finite probability space suffices' nor the description's disclosed finite-population choice imposes finiteness on the target's quantifiers. For example, independent fair assignment, U uniform on [0,1], J=1, K(0)=0, K(1)=1 and Y(k)=kU satisfy H1–H5 and give β_IV=1/2, but the continuous treated-response law cannot be represented on a finite Ω, so the excluded cases are substantive. The declaration establishes the intended IV component in its finite domain, including the normalized margin formula at P043/P043_Formalization.lean:1232, so retain that credit through a statement repair generalizing the probability and expectation framework with appropriate measurability and integrability hypotheses.", 'severity': 'major'}

Formal result: restricted

Scope: Restricts the population to finitely many types with arbitrary nonnegative weights summing to one; arbitrary probability laws, including nonatomic laws, are not covered. Identification and covariance conclusions retain the aggregate's monotonicity and positive-first-stage hypotheses. Omits signed-crossing value-set membership for arbitrary nonmonotone models, the particular T4 down-type value R(9) = -1, and the separate nonidentification assertion without monotonicity. Retains general bounded signed telescoping and the downward-crossing implication.

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:46a690134d6cb46cbb05d882cf5b09aa77f5851d21fbaa9da9d692c62b782cfc

Effective claim (source TeX): Sol's corrected text (validated.json, P043-E002, prior_correction.corrected_text; the same repair as smallest_repair), verbatim: "Let K_i(z) be student i's potential knowledge score under curriculum assignment z. In addition to assignment independence, exclusion, and consistency, assume K_i(1) >= K_i(0) for every student and E[K_i(1)-K_i(0)] > 0. For integer knowledge scores, the IV ratio is a weighted average of E[Y_i(j)-Y_i(j-1) | K_i(0)<j<=K_i(1)], with weight P(K_i(0)<j<=K_i(1))/E[K_i(1)-K_i(0)] for each crossed margin j. These weights are nonnegative and sum to one. Without monotonicity or another sufficient response restriction, random assignment and exclusion alone do not identify an effect confined to students whose knowledge increases. No numerical IV estimate is corrected; the reported curriculum reduced-form findings are unchanged by this defect." Mathematical content, with every hypothesis (validator's derivation.md, sections 2 and 5). Setting: a population of students $i$ with a probability law; assignment $Z \in \{0,1\}$ with $0 < P(Z=1) < 1$; potential knowledge scores $K_i(0), K_i(1) \in \{0,1,\dots,J\}$ (integers; $J = 18$ for the 2008 index); potential support $Y_i(z,k)$, real-valued with finite expectation, for $z \in \{0,1\}$ and $k \in \{0,\dots,J\}$. Hypotheses: (H1) exclusion: $Y_i(z,k) = Y_i(k)$ for both $z$ and every $k$; (H2) consistency: the observed variables are $K_i = K_i(Z)$ and $Y_i = Y_i(K_i(Z))$; (H3) assignment independence: $Z$ is independent of $(K_i(0), K_i(1), Y_i(0), \dots, Y_i(J))$; (H4) knowledge monotonicity: $K_i(1) \ge K_i(0)$ for every student; (H5) positive first stage: $E[K_i(1) - K_i(0)] > 0$. Define, for $j = 1,\dots,J$, $$\delta_i(j) = Y_i(j) - Y_i(j-1), \qquad C_i(j) = \mathbf{1}\{K_i(0) < j \le K_i(1)\},$$ and $\Delta_K = E[K \mid Z=1] - E[K \mid Z=0]$, $\Delta_Y = E[Y \mid Z=1] - E[Y \mid Z=0]$, $\beta_{IV} = \Delta_Y/\Delta_K$. Then: (i) [from (H1)-(H3)] $\Delta_K = E[K_i(1) - K_i(0)]$ and $\Delta_Y = E[Y_i(K_i(1)) - Y_i(K_i(0))]$. Moreover $\mathrm{Cov}(Z,K) = P(Z=1)P(Z=0)\,\Delta_K$ and $\mathrm{Cov}(Z,Y) = P(Z=1)P(Z=0)\,\Delta_Y$, so $\mathrm{Cov}(Z,Y)/\mathrm{Cov}(Z,K) = \beta_{IV}$ whenever $\Delta_K \ne 0$. (ii) [from (H4)] For every student, $$Y_i(K_i(1)) - Y_i(K_i(0)) = \sum_{j=1}^{J} C_i(j)\,\delta_i(j), \qquad K_i(1) - K_i(0) = \sum_{j=1}^{J} C_i(j).$$ If $K_i(1) = K_i(0)$ then $C_i(j) = 0$ for every $j$, so only students whose knowledge increases contribute. (iii) Hence $\Delta_Y = \sum_{j=1}^{J} E[C_i(j)\,\delta_i(j)]$ and $\Delta_K = \sum_{j=1}^{J} P(C_i(j) = 1)$. (iv) [with (H5)] Put $w_j = P(C_i(j) = 1)/\Delta_K$. Then $w_j \ge 0$, $\sum_{j=1}^{J} w_j = 1$, and $$\beta_{IV} = \sum_{j:\ P(C_i(j)=1) > 0} w_j\, E\bigl[\,Y_i(j) - Y_i(j-1) \,\bigm|\, K_i(0) < j \le K_i(1)\,\bigr].$$ (v) Without (H4): with the signed crossing $R_i(j) = \mathbf{1}\{K_i(1) \ge j\} - \mathbf{1}\{K_i(0) \ge j\} \in \{-1,0,1\}$, the two identities of (ii) hold for all $K_i(0), K_i(1) \in \{0,\dots,J\}$ with $R_i(j)$ in place of $C_i(j)$; under (H4), $R_i(j) = C_i(j)$. When some students have $K_i(1) < K_i(0)$, $R_i(j)$ takes the value $-1$ for them, so the resulting weights need not be nonnegative and do not involve increasers only (P043-T4 (a): the type 'down' has $R(9) = -1$).

.{u} : T5Statement.{u}

Theorem: SAP.P043.T5

Statement hash: 0a36cd9dfcd8f56c8a2da8d410ef37faa9b61d264fa194baf41691961eb0ccff

Statement definition: SAP.P043.T5Statement

Certificate. 20261104T162856Z

Declaration review: faithful; 2026-11-04T16:48:10Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P043-T6

Equation (1) and the sentence explaining it, with the stratum indicators indexed consistently ( $D_{ik}$ ; printed $D_k$ in the display and $D_i$ in the sentence)

p. 471 (PDF p. 9), right column

equation (1), p. 471 (PDF p. 9), right column. Correctly indexed stratum indicators select the nonreference offset, encode the omitted reference indicator, and give the equivalent stratum-intercept equation.

Historical component credits

Proved AronowAudit.P043.corrected_strata Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The statement concerns corrected notation and equivalent equations, matching this target’s express scope; it does not assert causal interpretation of β₁ or a GLS property. No claim-stage verdict change.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:71884ae97732185165ccd394f29b45fadd8da6c845e90f42ba54ba2fdc424192

Effective claim (source TeX): Corrected statement (Sol's smallest repair, validated.json, P043-E008, applied to the printed sentence). "Thus, our model has the form: $$Y_i = \beta_0 + \beta_1 X_i + \sum_{k=1}^{K-1} \gamma_k D_{ik} + u_i, \qquad (1)$$ where $X_i$ is scored 0 for control and 1 for treatment," the $K-1$ dummy variables for all strata except the reference stratum are denoted $D_{ik}$, where $D_{ik} = 1$ if observation $i$ belongs to stratum $k$ and 0 otherwise, "and $u_i$ denotes the unobserved disturbance term." Formal content of the corrected notation (this audit's unfolding of the definition; the paper states the model and proves nothing about it). Let the observations be $i = 1,\dots,n$, let the strata be labelled $1,\dots,K$ with $K \ge 1$ and stratum $K$ the reference stratum, let $s(i) \in \{1,\dots,K\}$ be the stratum of observation $i$, and let $D_{ik} = \mathbf{1}\{s(i) = k\}$ for $k = 1,\dots,K-1$. Then for all real $\gamma_1,\dots,\gamma_{K-1}$ and every $i$: (i) $\sum_{k=1}^{K-1} \gamma_k D_{ik} = \gamma_{s(i)}$ if $s(i) \le K-1$, and $= 0$ if $s(i) = K$; so, with the convention $\gamma_K := 0$, equation (1) is $Y_i = (\beta_0 + \gamma_{s(i)}) + \beta_1 X_i + u_i$, an intercept for each stratum and one coefficient $\beta_1$ on treatment. (ii) $\sum_{k=1}^{K-1} D_{ik} = 1 - \mathbf{1}\{s(i) = K\}$, so the indicator of the reference stratum equals $1 - \sum_{k=1}^{K-1} D_{ik}$: the $K$ stratum indicators sum to the constant regressor, which is why one stratum is left out ('adding dummy variables for all strata but one', p. 471).

: T6Statement

Theorem: SAP.P043.T6

Statement hash: e74ce9ec1140a54778c88863bd65fb6f32086495b9c9fbf7330c99a150134667

Statement definition: SAP.P043.T6Statement

Certificate. 20261104T162856Z

Declaration review: faithful; 2026-11-04T16:48:10Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status: Partial reproduction
Agreement: Agrees in part

Reported results that were targeted
Target	Where	Method	Status	Agreement
Table 4: civil liberties knowledge	p. 474 (PDF p. 12), first panel; discussed p. 473 (PDF p. 11)	I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.	Partial reproduction	Agrees in part
Table 4: general political knowledge	p. 474 (PDF p. 12), second panel	I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.	Partial reproduction	Agrees in part
Table 4: support for civil liberties	p. 474 (PDF p. 12), third panel	I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.	Partial reproduction	Agrees in part
In-text standardized 2007 knowledge effect	p. 473 (PDF p. 11), Results, left column	I divided the unadjusted 2007 treatment–control difference by the pooled within-group standard deviation on the public file. The result, about 0.485, agrees with the printed Cohen’s d = 0.49 to rounding.	Recalculated	To rounding
Footnote 13: open-ended constitutional-rights essays	p. 473 (PDF p. 11), footnote 13	I compared OLS calculations with the printed essay estimates on the public file (P043-N010). For 2007, restricting to the 1,045 survey respondents and setting their 201 missing scores to zero gives 0.181744 (SE 0.067518), matching 0.182 (0.068) to rounding; this coding is diagnostic. For the 591 respondents in 2008, -0.007703 (SE 0.128322) rounds to -0.008 (0.128), rather than -0.007 (0.128). Original inputs, score coding and the variance estimator remain unresolved, so exact original reproduction is blocked.	Partial reproduction	Agrees in part
Appendix Table A1: individual support items	Official online appendix p. 2 (PDF p. 2); cited p. 473 (main PDF p. 11)	I checked the printed significance stars against the printed coefficients and standard errors. I did not independently reproduce the original covariate-adjusted item models: the public file lacks the original birth-year categories, and the R script omits the stated pretreatment covariates (P043-N021).	Blocked	Not compared
Appendix figure: rescaled knowledge trend	Official online appendix p. 11 (PDF p. 11); described p. 473 (main PDF p. 11), right column	I recalculated the six normalized treatment and control means from the public data using the paper’s rescaling. They agree with the plotted means; P043-E007 corrects the interpretation of the control-group trend.	Reproduced	To rounding