{ "claims": [ { "claim": "More than 1,000 students in 59 classrooms at 10 high schools were randomly assigned, within 19 strata, to a civics course with or without the Bill of Rights curriculum; attrition was similar in the two arms.", "claim_id": "P043-H1", "related_corrections": [ "P043-N001", "P043-N004", "P043-N006" ], "tied_to": "Tables 1 and 2; Experimental Design, pp. 467-469" }, { "claim": "The paper reports increases in civil-liberties knowledge of $1.220$ correct answers out of 15 in 2007 (standard error $0.155$, Cohen’s $d=0.49$) and $0.762$ out of 18 in 2008 (standard error $0.226$), both with $p<0.001$.", "claim_id": "P043-H2", "related_corrections": [ "P043-N010" ], "tied_to": "Table 4, top panel; Table A2; footnote 13; abstract, pp. 465 and 473" }, { "claim": "The paper reports that the knowledge gain faded to $0.221$ in 2009 (standard error $0.269$), no longer statistically significant. After calibrating item difficulty with a 2010 survey, it describes control-group means as essentially unchanged across waves and the treatment group as declining to the control mean.", "claim_id": "P043-H3", "related_corrections": [ "P043-E007", "P043-E003" ], "tied_to": "Table 4; Table A4; unnumbered figure on Online Appendix p. 11; pp. 465 and 473" }, { "claim": "The treatment had no effect, positive or negative, on general political knowledge in any wave, so learning in the targeted domain did not displace other knowledge.", "claim_id": "P043-H4", "related_corrections": [ "P043-E005" ], "tied_to": "Table 4, middle panel; Table A3; Table A6; pp. 465 and 473" }, { "claim": "The treatment had no detectable effect on support for civil liberties in 2007, 2008 or 2009, whether measured by the additive index or item by item, and the same holds among students present in all three waves.", "claim_id": "P043-H5", "related_corrections": [ "P043-E005" ], "tied_to": "Table 4, bottom panel; Table A1; Table A6; abstract, pp. 465, 473-474" }, { "claim": "The paper argues that the rise in knowledge without a detected rise in support calls into question a causal connection between knowledge and attitudes. It claims ample power to detect small effects and, under the hypothesized positive effects through other channels described on p. 473, a reduced-form test that favors finding mediation.", "claim_id": "P043-H6", "related_corrections": [ "P043-E002" ], "tied_to": "Figure 1; Table 4; abstract, pp. 473-475" }, { "claim": "The paper reports significant correlations among the knowledge and support indices in 2007: $r=0.42$ between the knowledge indices and $r=0.30$ and $r=0.16$ with support. It describes this as reproducing the pattern of observational studies and reports greater reliability for the knowledge scales than for the support scales.", "claim_id": "P043-H7", "related_corrections": [ "P043-E004" ], "tied_to": "Table 3; text p. 471; conclusion p. 474" }, { "claim": "Results are robust: covariates make little difference, the same pattern holds among students present in all three posttreatment waves, and there are no significant heterogeneous treatment effects after Holm-Bonferroni correction.", "claim_id": "P043-H8", "related_corrections": [ "P043-E005", "P043-E011", "P043-N015" ], "tied_to": "Table 4; Table A6; footnotes 12 and 14; pp. 468, 471, 473, 475" }, { "claim": "This is the first large-scale randomized experiment testing the effect of civics instruction on knowledge of and support for civil liberties.", "claim_id": "P043-H9", "related_corrections": [], "tied_to": "Abstract; p. 465; conclusion p. 474; literature review pp. 463-464" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the last seven lines of the left column of page 472. A red box marks the last line, 'then we can estimate the local average treatment'. The two lines above it read 'assume that the treatment only influences support for civil liberties indirectly by influencing knowledge,'.", "caption": "Statistical Model, p. 472 (PDF p. 10), foot of the left column. The marked line begins the conclusion drawn from random assignment and the exclusion restriction; it continues at the top of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002.png" }, { "alt": "Crop of the first four lines of the right column below Table 3. Red boxes mark the words 'effect of knowledge by using randomly assigned curriculum as an instrumental variable', which are followed by the citation '(Angrist, Imbens, and Rubin 1996)'.", "caption": "p. 472 (PDF p. 10), top of the right column, below Table 3. The marked words complete the sentence: the effect of knowledge is to be estimated with the curriculum as an instrument.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-b.png" }, { "alt": "Crop of the last seven lines of the right column of page 474. A red box marks the last line, 'effects of increases in knowledge are identified only'. The lines above it attribute the interpretation to the work of Angrist, Imbens, and Rubin (1996).", "caption": "Conclusion, p. 474 (PDF p. 12), foot of the right column. The marked line begins the statement that effects of knowledge are identified in a subgroup; it continues on p. 475.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-c.png" }, { "alt": "Crop of the running head and the first seven lines of the left column of page 475. Red boxes mark the words 'among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.' The next sentence begins 'It may be that support for civil liberties among students who are affected on the margin'.", "caption": "Conclusion, p. 475 (PDF p. 13), top of the left column. The marked words name the subgroup: the students whose knowledge the curriculum raises.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-d.png" } ], "formal_release": null, "formal_statements": [ { "id": "P043-T4", "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T4" }, { "id": "P043-T5", "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T5" } ], "headline": "Random assignment and the exclusion restriction do not by themselves give a local effect of knowledge among the students whose knowledge the curriculum raises; monotonicity of knowledge in the assignment and a positive first stage suffice to justify this interpretation.", "id": "P043-E002", "location": "Statistical Model, p. 472 (PDF p. 10), foot of the left column to the top of the right column; Conclusion, pp. 474–475 (PDF pp. 12–13)", "pdf_page": 10, "printed_page": "472", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P043-T4", "P043-T5" ], "text": { "as_printed": "p.~472: ``If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, \\hl{then we can estimate the local average treatment effect of knowledge} by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).''\n\npp.~474--475: ``This line of argument takes notice of the fact that the statistical effects of increases in knowledge \\hl{are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum}.''", "corrected": "p.~472: ``If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, \\hl{and that the treatment lowers no student's knowledge and raises knowledge on average,} then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).''\n\npp.~474--475: ``This line of argument takes notice of the fact that\\hl{, if the treatment curriculum lowers no student's knowledge and raises knowledge on average,} the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.''\n\nIn full, let $Z_i\\in\\{0,1\\}$ be the assignment of student $i$, let $K_i(z)$ be the student's knowledge score under assignment $z$, and let $Y_i(k)$ be the student's support at knowledge score $k$. Assume that $Z_i$ is independent of $(K_i(0),K_i(1),Y_i(\\cdot))$, that \\hl{$0<\\Pr(Z_i=1)<1$}, that assignment affects support only through knowledge, and that the observed scores are $K_i=K_i(Z_i)$ and $Y_i=Y_i(K_i)$. Assume \\hl{in addition that $K_i(1)\\ge K_i(0)$ for every student and that $\\mathbb{E}[K_i(1)-K_i(0)]>0$}. Then, for an integer knowledge score,\n\\[\n\\begin{aligned}\n\\frac{\\mathbb{E}[Y_i\\mid Z_i=1]-\\mathbb{E}[Y_i\\mid Z_i=0]}{\\mathbb{E}[K_i\\mid Z_i=1]-\\mathbb{E}[K_i\\mid Z_i=0]}\n&=\\sum_{j}w_j\\,\\mathbb{E}\\bigl[Y_i(j)-Y_i(j-1)\\mid K_i(0)0$. The weights $w_j$ are nonnegative and sum to one. Only students whose knowledge the curriculum raises enter the conditional means, and a student who crosses several levels contributes at each of them. Another sufficient restriction on the response of knowledge to the curriculum may replace monotonicity. Without such a restriction, random assignment and exclusion do not identify an effect confined to the students whose knowledge the curriculum raises. No numerical estimate is corrected.", "evidence": "The evidence is an exact counterexample and a derivation. Consider a population with two types of student, in which every knowledge score lies inside the 2008 range of 0 to 18 and every support score inside the range of 0 to 7. The curriculum is assigned with probability $1/2$, independently of type. I write $Z$ for the assignment and $K$ and $Y$ for the observed knowledge and support.\n\n\\begin{audittable}{lccccc}{A population in which the instrumental-variable ratio is zero although knowledge raises support.}\nType & Share & $K_i(0)$ & $K_i(1)$ & $Y_i(8)$ & $Y_i(9)$ \\\\\nKnowledge rises & $2/3$ & 8 & 9 & 3 & 4 \\\\\nKnowledge falls & $1/3$ & 9 & 8 & 2 & 4\n\\end{audittable}\n\nSupport depends on the assignment only through knowledge, as the exclusion restriction requires. By the table, I have $\\mathbb{E}[K\\mid Z=0]=25/3$ and $\\mathbb{E}[K\\mid Z=1]=26/3$, and the first stage is $1/3>0$. I also have $\\mathbb{E}[Y\\mid Z=0]=\\mathbb{E}[Y\\mid Z=1]=10/3$. Then the reduced form is $0$ and $\\beta_{\\mathrm{IV}}=0$. However, every student whose knowledge rises has a causal effect of $4-3=1$ on support. The numerator vanishes by cancellation: $(2/3)(1)+(1/3)(-2)=0$.\n\nOther features of the observed data do not identify this effect in general either. Two further populations, each with five equally likely types, have the same joint distribution of assignment, knowledge and support, with first stage $1/5$ and reduced form $0$. The average effect among the students whose knowledge rises is $1$ in the first population and $4/3$ in the second.\n\nUnder monotonicity, the corrected statement follows by telescoping. For an integer score,\n\\[\n\\begin{aligned}\nY_i(K_i(1))-Y_i(K_i(0))&=\\sum_{j}\\mathbf{1}\\{K_i(0) 1989 \\& Aq33year != 9999}\n\n\\emph{Privacy script, line 25.} \\texttt{drop seq3}", "corrected": "\\emph{README, in place of line 18, with} seq3 \\emph{for} seg3 \\emph{in line 5.} \\hl{The privacy edit retains all observations in both files. In the main file, the script replaces Aq33year with 1989 whenever Aq33year is greater than 1989 and is not 9999; in Stata this condition includes numeric missing values. The code 9999 is retained. In the Survey E file, the variable seq3 is removed. The edited files contain 1215 and 254 observations, respectively.}\n\n\\emph{Privacy script, line 17,} if missing years of birth are to stay missing: \\texttt{replace Aq33year = 1989 if Aq33year > 1989 \\& Aq33year != 9999 \\hl{\\& Aq33year < .}} The public files would then have to be regenerated from the original data, and the README would describe the corrected operation.", "evidence": "The evidence is an exact counterexample and a check of the replication files. The privacy script has eight commands, none of which deletes observations. When line 17 is applied to two students born in 1988 and 1991, aged 19 and 16 in the reference year 2007, the output has two rows, with years 1988 and 1989; removing respondents under 18 would leave one row. For a missing year, both comparisons hold, and the year becomes 1989. In the public main file the year of birth takes the values 1986 (1 student), 1988 (21), 1989 (1149) and 9999 (44). Since the original files are not available, I could not observe the original values for the 120 students who missed the pretreatment survey.", "problem": "The README misdescribes the edit in two ways. It says that observations were removed, and no command of the privacy script removes a row. The script replaces values of the year of birth, \\texttt{Aq33year}, in the main file, and drops the variable \\texttt{seq3} from the file of the 2010 survey. The main file keeps all 1215 students and the 2010 file all 254 respondents. The README also names the second variable \\texttt{seg3}; its name is \\texttt{seq3}.\n\nThe script itself does more than its comment describes. The comment says that the year of birth is to be top-coded at 1989, so that no age below 18 in 2007 can be inferred, and that the code 9999 is left alone. In Stata a missing numeric value is larger than every number. This implies that the condition \\texttt{Aq33year > 1989 \\& Aq33year != 9999} also holds for a missing year of birth, and the script sets such a year to 1989. In the public file all 120 students who did not take the pretreatment survey have year of birth 1989, and every other pretreatment variable is missing for them. The analysis script later recodes a missing pretreatment answer to its own category; a year already set to 1989 is not recoded, and it enters the age interactions as an observed year.", "what_survives": "This finding concerns the deposit of 2019, and it shows no error in the article. The public file keeps the 627 treatment and 588 control students and the response counts 1045, 591 and 508 of the article. The privacy edit leaves the assignment, the outcomes and every analysis without year of birth unchanged, and the analyses of the 2010 survey do not use \\texttt{seq3}. For the models of Table 4 with covariates, the space spanned by the covariates is unchanged when only the 120 missing years are recoded, because the same 120 students are already identified by the missing-answer category of another pretreatment covariate. The public file does not preserve the original year-of-birth categories needed to reconstruct the original covariate-adjusted models, randomization check and age interactions." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E013", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of five lines of the right column. The text reads: who go on to college ranges widely, from 28\\% to 88\\%. All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. Seven of the ten original schools participated in all four waves of surveys. Attrition occurred in schools IV and VI. A red outline surrounds the word Seven.", "caption": "Research Sites, last paragraph, p. 466 (PDF p. 4), right column. The outline marks the number of schools said to have participated in all four waves of surveys.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N001.png" }, { "alt": "Crop of the first three lines of the right column. The text reads: completed by students in seven schools, taught by 19 of the original 25 teachers. Teachers whose students participated in the follow-up study had an average. A red outline surrounds the words seven schools.", "caption": "Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), top of the right column. The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The outline marks the number of schools.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N001-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Six schools, not seven, supplied completed surveys in 2008 and in all four waves; seven is the number of schools that remained in the study after 2007.", "id": "P043-N001", "location": "Research Sites, last paragraph, p. 466 (PDF p. 4), right column; Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence that runs from the foot of the left column to the top of the right column", "pdf_page": 4, "printed_page": "466", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{p.~466.} All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. \\hl{Seven} of the ten original schools participated in all four waves of surveys.\n\n\\emph{p.~469.} The 2008 and 2009 surveys were thus completed by students in \\hl{seven schools}, taught by 19 of the original 25 teachers.", "corrected": "\\emph{p.~466.} Seven of the ten original schools \\hl{remained in the study after 2007; six supplied completed surveys in all four waves}.\n\n\\emph{p.~469.} The 2008 and 2009 surveys were thus completed by students in \\hl{six and seven schools, respectively}, \\ldots\n\nThe number of teachers in the rest of the second sentence is corrected separately in \\fid{P043-N004}.", "evidence": "The evidence is a recalculation from the replication data. I counted the completed surveys by school and by wave. The same counts by stratum and arm agree with all 240 numeric cells of Table 2, the class counts and the total row included.\n\\begin{audittable}{llrrrrrl}{Completed surveys by school, from the replication data. The school codes are those of the data file.}\nSchool & Strata & Assigned & Pretreatment & 2007 & 2008 & 2009 & Remained after 2007 \\\\\n1 & A, L & 86 & 79 & 72 & 49 & 44 & yes \\\\\n2 & C, D & 86 & 80 & 66 & 0 & 11 & yes \\\\\n3 & E, F & 65 & 63 & 61 & 0 & 0 & no \\\\\n4 & H, N & 87 & 79 & 69 & 0 & 0 & no \\\\\n5 & R, S & 225 & 200 & 211 & 145 & 46 & yes \\\\\n6 & G, P & 100 & 81 & 76 & 0 & 0 & no \\\\\n7 & I, J & 146 & 144 & 129 & 112 & 112 & yes \\\\\n8 & K, M & 121 & 102 & 90 & 63 & 73 & yes \\\\\n9 & B, T & 120 & 102 & 105 & 88 & 89 & yes \\\\\n10 & Q & 179 & 165 & 166 & 134 & 133 & yes \\\\\n\\end{audittable}\nFor school 2, the cells of Table 2 for strata C and D (control and treatment) give $20 + 24 + 27 + 15 = 86$ assigned, $0 + 0 + 0 + 0 = 0$ respondents in 2008 and $0 + 2 + 5 + 4 = 11$ in 2009.", "problem": "Three schools left the study after 2007, and seven remained. One of the seven has no completed survey in 2008. Strata C and D are the whole of this school, which has 86 assigned students. Table 2 shows that neither arm of either stratum has a 2008 respondent, and that the school has 11 respondents in 2009. Completed surveys come from six schools in 2008, from seven in 2009, and from six in every wave.\n\nOn p.~466, ``participated'' can be read loosely as ``remained in the study'', and with that meaning seven is correct. The sentence on p.~469 says that the 2008 and 2009 surveys were completed by students in seven schools, and for 2008 this is false. The data show no completed 2008 survey for this school. They do not show whether the survey was attempted there, or why none was returned.", "what_survives": "The wording correction leaves Table 2, the sample sizes 1045, 591 and 508, and the reported estimates unchanged; it changes no survey responses or model specification. The 2009 response rates among the seven schools that remained also stand: $253/477 = 53.0\\%$ and $255/486 = 52.5\\%$. Only the two sentences change." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N001", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the page number 469 and the first five lines of the right column. A red box marks the number 19 at the end of the first line, 'completed by students in seven schools, taught by 19', which continues 'of the original 25 teachers'.", "caption": "Top of the right column, p. 469 (PDF p. 7). The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The marked number is the count of teachers in the continuing schools.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N004.png" } ], "formal_release": null, "formal_statements": [], "headline": "The seven continuing schools had 18 of the original 25 teachers, not 19.", "id": "P043-N004", "location": "Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence running from the foot of the left column to the top of the right column", "pdf_page": 7, "printed_page": "469", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "``The 2008 and 2009 surveys were thus completed by students in seven schools, taught by \\hl{19} of the original 25 teachers.''", "corrected": "``The 2008 and 2009 surveys were thus completed by students in seven schools, taught by \\hl{18} of the original 25 teachers.''", "evidence": "This is a recalculation from the replication data. Each teacher identifier belongs to one school, and the seven continuing schools have $1+2+6+2+1+3+3=18$ teachers. The students who completed the 2008 survey had 16 of these teachers, those who completed the 2009 survey had all 18, and the two groups together had 18. The same data reproduce every entry of Table 2, including its totals.", "problem": "The replication data identify 25 teachers in 10 schools. The three schools that left the study after the 2007 survey had 3, 2 and 2 teachers, which leaves $25-7=18$ teachers in the seven continuing schools. Counting the teachers of the students who answered a follow-up survey does not give 19 either.", "what_survives": "The 25 original teachers, the 10 schools, the seven continuing schools, the 19 strata and the counts of Table 2 are unchanged. No model in the replication code uses the number of teachers, and no estimate or standard error changes. The ages and other attributes of the teachers are not in the replication data, and the description of the follow-up teachers in the next sentences of the paper was not checked." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N004", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the last two lines of the left column and footnote 8 below them. A red box marks the words 'Stratum H' in the first line of the footnote, 'Consider, for example, Stratum H, consisting of three courses taught by an instructor'.", "caption": "Footnote 8, p. 469 (PDF p. 7). The marked stratum is named as the example of one instructor with three courses.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N006.png" } ], "formal_release": null, "formal_statements": [], "headline": "The example of footnote 8 describes Stratum I (and equally J or K), not Stratum H.", "id": "P043-N006", "location": "Footnote 8, p. 469 (PDF p. 7)", "pdf_page": 7, "printed_page": "469", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "``Consider, for example, \\hl{Stratum H}, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.''", "corrected": "``Consider, for example, \\hl{Stratum I}, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.''", "evidence": "This is a recalculation from the replication data, tabulated by stratum, teacher and assignment. The numbers of courses and students that it gives for strata H, I, J and K agree with Table 2 (p.~470).", "problem": "In the replication data, Stratum H has two teachers. One taught its single control course (10 students) and the other taught both of its treatment courses (31 students). The footnote describes one instructor with three courses, two of them treated. Strata I, J and K have this composition.", "what_survives": "The footnote's account of assignment within a stratum is unchanged, and the definition in the text allows a stratum taught by a cluster of teachers. Nothing else depends on the label. The check does not establish how the courses of Stratum H were assigned." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N006", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the end of footnote 13. The text reads: of relevant constitutional principles identified. In the 2007 posttreatment survey, students in the treatment group performed significantly better than students in the control group, indicating that students were able to apply the constitutional knowledge they had gained. On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128). This finding comports with the general trend of declining differences between treatment and control groups over time. Red outlines surround .182, (SE=0.068) and (-.007 rights, SE = 0.128).", "caption": "Footnote 13, p. 473 (PDF p. 11), last ten lines. The outlines mark the two essay estimates with their standard errors, which the essay commands of the replication files do not return.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N010.png" } ], "formal_release": null, "formal_statements": [], "headline": "The essay commands in the replication files are not restricted to the students who took each survey, and they do not return the estimates of footnote 13; the printed estimates are not shown to be wrong.", "id": "P043-N010", "location": "Footnote 13, p. 473 (PDF p. 11); the essay commands of the replication files: lines 483 and 485 of the Stata script Green_et_al_JP_2011_Public_Use_EDITED.do, and lines 613 and 615 of the R script Green_et_al_JP_2011_Public_Use_EDITED.R", "pdf_page": 11, "printed_page": "473", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Footnote~13, p.~473.} On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128).\n\n\\emph{Stata script, line 483 (broken here after \\texttt{i.Aq21}); line 485 is the same command for \\texttt{Copenended}.}\n\\[\n\\begin{aligned}\n&\\texttt{xi: xtreg Bopenended treat i.stratum i.Aq1 i.Aq3 i.Aq4 i.Aq18 i.Aq19 i.Aq20 i.Aq21}\\\\\n&\\qquad\\texttt{i.Aq22 i.Aq23 i.Aq24 i.Aq28 i.Aq32 i.Aq33year, re i(classcode)}\n\\end{aligned}\n\\]\n\n\\emph{R script, lines 613 and 615.}\n\\[\n\\begin{aligned}\n&\\texttt{reg.B.open <- lmer(Bopenended \\textasciitilde{} treatment + stratum + (1|classcode), }\\hl{\\texttt{data=data}}\\texttt{)}\\\\\n&\\texttt{reg.C.open <- lmer(Copenended \\textasciitilde{} treatment + stratum + (1|classcode), }\\hl{\\texttt{data=data}}\\texttt{)}\n\\end{aligned}\n\\]", "corrected": "In the Stata script, each essay command takes the students who took that survey:\n\\[\n\\begin{aligned}\n&\\texttt{xi: xtreg Bopenended treat} \\;\\ldots\\; \\texttt{i.Aq33year }\\hl{\\texttt{if Bcompleted == 1}}\\texttt{, re i(classcode)}\\\\\n&\\texttt{xi: xtreg Copenended treat} \\;\\ldots\\; \\texttt{i.Aq33year }\\hl{\\texttt{if Ccompleted == 1}}\\texttt{, re i(classcode)}\n\\end{aligned}\n\\]\nThe R calls take the same two subsets. The files should also document how the essay score was built, how a missing essay is scored, the fixed covariates and the variance estimator. These filters alone do not reproduce both printed pairs, and the numbers of footnote~13 stay as printed.", "evidence": "The evidence is a recalculation on the public file. The R formulas were run as written. Stata was not available, and the lines marked OLS are ordinary least squares with the fixed regressors of the Stata commands and conventional standard errors.\n\\begin{audittable}{llrrr}{Effect of treatment on the essay score: footnote~13 and recalculations on the public file.}\nWave & Calculation & $N$ & Estimate & SE \\\\\n2007 & Footnote~13 & & .182 & 0.068 \\\\\n2007 & R formula as written & 1014 & 0.139609 & 0.076139 \\\\\n2007 & OLS, sample of the Stata command & 1014 & 0.123438 & 0.068956 \\\\\n2007 & OLS, students who took the survey, missing scores set to zero & 1045 & 0.181744 & 0.067518 \\\\\n2008 & Footnote~13 & & $-.007$ & 0.128 \\\\\n2008 & R formula as written & 1215 & $-0.002728$ & 0.100683 \\\\\n2008 & OLS, sample of the Stata command & 1215 & $-0.007565$ & 0.070603 \\\\\n2008 & OLS, students who took the survey & 591 & $-0.007703$ & 0.128322 \\\\\n\\end{audittable}\nThe 2007 pair is recovered at printed precision on the 1045 students who took the survey, with their 201 missing scores set to zero. The 2008 standard error is recovered on the 591 students who took the survey; the estimate there rounds to $-.008$ and not to $-.007$. Note that setting missing scores to zero is a diagnostic coding and not an established rule for the original analysis. The birth-year covariate of the public file was top-coded in 2019 (see \\fid{P043-E013}), which changes one set of regressors.", "problem": "Footnote~12 says that the analyses include all students who took each survey, and Table~3 reports 1045 such students in 2007 and 591 in 2008. Neither script restricts the essay regressions to them. On the public file, the 2007 command selects 1014 students. It includes the 170 students flagged as not having taken the 2007 survey, whose essay score is zero, and it excludes 201 students who took the survey and whose essay score is missing. The 2008 command selects all 1215 students, of whom 624 are flagged as not having taken the 2008 survey. The R formulas also omit the pretest covariates of the Stata commands (see \\fid{P043-N021}). As written, the commands do not return the two pairs printed in footnote~13.", "what_survives": "The estimates of footnote~13 are not shown to be wrong, and the 2007 pair is recovered at printed precision under one coding of the students who took the survey. The models of Table~4 use other outcome variables and other commands, and they do not depend on the essay scores. The essay commands of the replication files change." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N010", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": { "class": "error", "severity": "Moderate" }, "class": "not_confirmed", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of eight lines of text. The passage reads: dissent, or due process. This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship. The theoretical implications of this finding are profound. Evidently, it is possible to increase awareness and understanding of civil liberties without producing an increase in; the crop ends in the middle of this sentence. Red outlines surround the words the statistical test is biased in favor of finding a mediating relationship, which run over two lines.", "caption": "p. 474 (PDF p. 12), left column below Table 4. The outlines mark the clause that refers back to the preceding positive-other-channel explanation.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N014.png" }, { "alt": "Line chart. The horizontal axis is the net effect of the curriculum on support through channels other than knowledge, D, from -3/25 to 3/25 points of the support index. The vertical axis is the reduced-form effect on support, M + D, from -3/50 to 9/50. A straight line of slope one passes through the point D = 0, reduced form 3/50. Its part with D at least 0 is a solid blue line labelled reduced form at least M, the case of p. 473. Its part with D below 0 is a dashed orange line labelled reduced form below M. A dotted horizontal line marks the effect through knowledge, M = 3/50. A dot on the dashed part at D = -3/50 and reduced form 0 is labelled exact example: D = -3/50, reduced form 0 although M = 3/50.", "caption": "The identity $\\tau = M + D$ with the effect through knowledge fixed at $M = 3/50$, the value in the two examples. The reduced form is at least $M$ when $D \\ge 0$, the case described on p. 473, and smaller than $M$ when $D < 0$. The dot marks the examples outside that sign premise, in which $D = -3/50$ and the reduced form is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N014-sign-condition.png" } ], "formal_release": null, "formal_statements": [], "headline": "Whether the favorable-mediation claim omits the sign premise needed for its comparison.", "id": "P043-N014", "location": "p. 474 (PDF p. 12), left column below Table 4, sentence beginning “This pattern of results is striking”; the argument it refers to is on p. 473 (PDF p. 11), left column", "pdf_page": 12, "printed_page": "474", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "reason": "The allegation was that p.~474 drops a necessary sign condition and claims that the reduced-form test always favors finding mediation. The relevant passages are:\n\n\\emph{p.~474.} This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship.\n\n\\emph{p.~473.} As Figure 1 suggests, estimating the reduced form provides a \\emph{conservative} test, because if the treatment leads to a direct increase in support for civil liberties (i.e., course content encourages support for norms of free expression and association) or indirectly through channels other than knowledge (e.g., course discussion promotes appreciation for dissenting viewpoints), we would attribute these effects to the mediating role of knowledge and understanding.\n\nThe phrase ``as noted above'' refers back to p.~473's positive direct or other-channel effects. Read in context, the claim carries that hypothesized sign premise forward: the curriculum's net effect through channels other than knowledge is nonnegative. On that premise, attributing the reduced form to knowledge weakly favors finding mediation. The published conditional is mathematically valid, so I did not establish a false statement under this natural reading.\n\nThe design does not verify the premise. Random assignment makes the treatment independent of the disturbances of Figure~1 but imposes no sign on the other-channel contribution. The null result on general political knowledge addresses one possible offset, and does not identify the net sign of every other channel. Its application to mediation consequently remains conditional; no actual negative channel or positive mediated effect in the experiment is established here.\n\nThe derivation and two exact examples explain the limitation of an unconditional reading. In a model without feedback from support to knowledge and with well-defined interventions on knowledge, let $K(t)$ be a student's knowledge score under curriculum $t$, with $t = 1$ for the experimental curriculum and $t = 0$ for the usual one, and let $S(t,k)$ be the support score under curriculum $t$ when knowledge is set to $k$. Define\n\\[\n\\begin{aligned}\n\\tau &= \\mathbb{E}\\bigl[S(1,K(1)) - S(0,K(0))\\bigr],\\\\\nM &= \\mathbb{E}\\bigl[S(0,K(1)) - S(0,K(0))\\bigr],\\\\\nD &= \\mathbb{E}\\bigl[S(1,K(1)) - S(0,K(1))\\bigr].\n\\end{aligned}\n\\]\nHere $\\tau$ is the reduced form, $M$ is the effect through knowledge and $D$ is the net effect through the other channels. Adding and subtracting $S(0,K(1))$ gives $\\tau = M + D$. Random assignment identifies $\\tau$ and puts no restriction on the sign of $D$. The identity implies that $\\tau \\ge M$ if and only if $D \\ge 0$.\n\nIn the first example, $T$ takes the values $0$ and $1$, $u_1$ the values $\\pm 1/5$ and $u_2$ the values $\\pm 1/10$, each with probability $1/2$, and $T$, $u_1$, $u_2$ are independent. Set $K(t) = 8 + (6/5)t + u_1$ and $S(t,k) = 23/10 + (k-8)/20 - (3/50)t + u_2$. A direct computation gives $M = 3/50$, $D = -3/50$ and $\\tau = 0$. The observed support score is $23/10 + u_1/20 + u_2$ in both arms.\n\nThe second example has integer scores inside the 2007 ranges of Table~3. Let $W$ be uniform on $\\{1,\\dots,5\\}$ and $V$ uniform on $\\{1,\\dots,100\\}$, with $T$, $W$, $V$ independent. Set $K(t) = 8 + t\\,(1 + \\mathbf{1}\\{W = 5\\})$ and $S(t,k) = 2 + \\mathbf{1}\\{V \\le 36 + 5(k-8) - 6t\\}$. These definitions give $\\mathbb{E}[S(0,K(0))] = 59/25$, $\\mathbb{E}[S(0,K(1))] = 121/50$ and $\\mathbb{E}[S(1,K(1))] = 59/25$.\n\\begin{audittable}{lrr}{The two examples violate the contextual nonnegative-other-channel premise: positive mediation is canceled by a negative effect through other channels.}\nQuantity & First example & Second example \\\\\nMean knowledge change, $\\mathbb{E}[K(1)-K(0)]$ & $6/5$ & $6/5$ \\\\\nExpected support gain, $8\\to9$ or $9\\to10$ & $1/20$ & $1/20$ \\\\\nEffect through knowledge, $M$ & $3/50$ & $3/50$ \\\\\nNet effect through the other channels, $D$ & $-3/50$ & $-3/50$ \\\\\nReduced form, $\\tau = M + D$ & $0$ & $0$ \\\\\n\\end{audittable}\nIn both examples the treatment is independent of every disturbance, as Figure~1 assumes. In the second, a general-knowledge score can be held equal in the two arms, so the null result on general political knowledge does not exclude the example. Note that the examples are not fitted to the data.\n\nBoth examples have $D=-3/50<0$. They satisfy random assignment and the unsigned diagram, but violate the hypothesized sign premise carried forward from p.~473. They therefore refute only an unconditional version of the favorable-mediation comparison, not the paper's contextual conditional. Setting the second example's threshold term $-6t$ to $0t$ or $+6t$ gives, respectively, $(M,D,\\tau)=(3/50,0,3/50)$ and $(3/50,3/50,3/25)$, which satisfy the weak comparison exactly.\n\nTable~4 and the reported total-effect estimates are unchanged by this mediation argument. I separately recalculated the unadjusted estimates (\\fid{P043-N020}); exact reproduction of the adjusted estimates remains blocked (\\fid{P043-E013}). The reported gain in knowledge without a statistically significant increase in support remains a statement about curriculum total effects. The p.~472 conditional exclusion-restriction discussion, the p.~473 positive-channel argument and the Conclusion's cautious multiple interpretations remain. I did not check the ample-power claim.\n\nAn optional clarification would repeat the premise: ``under the positive other-channel effects described above, the reduced-form contrast weakly favors attributing an effect to knowledge.'' More explicitly, $\\tau\\ge M$ if and only if $D\\ge0$, with strict inequality only for $D>0$; the condition concerns the net of the other channels, not each path separately. A negative net contribution can conceal positive mediation, as the examples show. This makes the existing contextual condition explicit without changing an estimator or numerical estimate." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N014", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of footnote 14, four lines. The footnote reads: We tested for heterogeneous treatment effects by interacting the treatment term with each of the individual-level covariates. None of these effects were significant after taking into account multiple comparisons using the Holm-Bonferroni method. Red outlines surround the words each of the individual-level covariates and the words Holm-Bonferroni method.", "caption": "Footnote 14, p. 475 (PDF p. 13). The outlines mark the two parts of the description that the replication scripts do not implement: interactions with each individual-level covariate, and the Holm–Bonferroni adjustment.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N015.png" } ], "formal_release": null, "formal_statements": [], "headline": "The replication scripts interact the treatment with one summed pretest score where the text describes each of seven factual questions, and they contain no Holm–Bonferroni adjustment, no balance simulation and no code for Table A6; the scripts cannot reproduce these analyses, and no published result is shown to be wrong.", "id": "P043-N015", "location": "Footnote 14 and the sentence on heterogeneous treatment effects, p. 475 (PDF p. 13), left column; also the balance check on p. 468 (PDF p. 6), footnote 12 on p. 473 (PDF p. 11) and Table A6, p. 10 of the online appendix; read against the Stata and R scripts of the replication files", "pdf_page": 13, "printed_page": "475", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{p.~475.} Our search for heterogeneous treatment effects came up with no convincing evidence of subgroup differences; partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother's level of education, employment status and \\hl{seven factual questions} yielded no significant interactions in models predicting knowledge or support for civil liberties.\n\n\\emph{Footnote~14, p.~475.} We tested for heterogeneous treatment effects by interacting the treatment term with \\hl{each of the individual-level covariates}. None of these effects were significant after taking into account multiple comparisons using the \\hl{Holm-Bonferroni method}.", "corrected": "The replication files should \\hl{supply the code and output} for the interactions of the treatment with each of the seven factual questions, state the family of comparisons to which the Holm--Bonferroni adjustment was applied and the adjustment itself, and supply the code for the balance simulation and for Table~A6. The description in the paper changes only if the original analysis is found to differ from it.", "evidence": "The evidence is a check of the pages and of the two scripts. I read the heterogeneity blocks (lines 313--408 of the Stata script and lines 283--440 of the R script). I also searched both scripts for a multiplicity adjustment, for simulated assignments and for a restriction to the three-wave sample, and found none. The check does not show that the analyses were not performed outside these scripts. The top-coded year of birth in the public file also prevents an exact reconstruction of the original interactions with age.", "problem": "Each script fits one interaction model for each of seven moderators and six outcomes, 42 models in all. The seven factual questions enter through a single moderator, their sum \\texttt{ATOTknow}, and not through seven separate interactions; the scoring of this sum is the subject of \\fid{P043-E011}. Neither script computes a Holm--Bonferroni adjustment. The scripts also contain no code for the balance check with 20,000 simulated assignments (p.~468), and none for the analysis of Table~A6, which is restricted to the 387 students present in all three posttreatment waves. These analyses cannot be reproduced from the scripts as the paper describes them.", "what_survives": "No published result is refuted, and the code for the main treatment effects is separate from the missing blocks. The replication files change: the family of heterogeneity tests as described, its Holm--Bonferroni adjustment, the balance check and Table~A6 are at present not reproducible from them." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N015", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of seven lines of the reference list. The last line of the preceding entry ends with Political Psychology 26 (3): 333-64. The next entry reads: Galston, William A. 2001. Political Knowledge, Political Engagement, and Civic Education. Annual Review of Political Science 4 (1): 217-41. It is followed by the entry Galston, William A. 2007. Civic Knowledge, Civic Education, and Civic Engagement: A Summary of Recent Research. International Journal of Public Administration 30: 623-42. A red outline surrounds the page range 217-41.", "caption": "References, p. 475 (PDF p. 13), right column, the entries Galston 2001 and Galston 2007. The outline marks the page range “217–41”, which should read “217–34”.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N016.png" } ], "formal_release": null, "formal_statements": [], "headline": "The page range of Galston (2001) is 217–34, not 217–41.", "id": "P043-N016", "location": "References, entry Galston 2001, p. 475 (PDF p. 13), right column", "pdf_page": 13, "printed_page": "475", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Galston, William A. 2001. ``Political Knowledge, Political Engagement, and Civic Education.'' \\emph{Annual Review of Political Science} 4 (1): \\hl{217--41}.", "corrected": "Galston, William A. 2001. ``Political Knowledge, Political Engagement, and Civic Education.'' \\emph{Annual Review of Political Science} 4 (1): \\hl{217--34}.", "evidence": "The evidence is a check of the page and of the publisher's page for the article (Annual Reviews, DOI 10.1146/annurev.polisci.4.1.217), which reads ``Vol.~4:217-234''.", "problem": "The cited article ends on page 234 and not on page 241.", "what_survives": "The author, title, year and volume identify the work, and nothing in the paper depends on the page range." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N016", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index, and Support for Civil Liberties Index. The rows are Number of Questions (13, 5, 3), Mean (2010 Survey) (7.59, 2.50, 2.35), Standard Deviation (2010 Survey) (2.21, 1.33, 0.81), Cronbach's Alpha (2010 Survey) (0.55, 0.54, 0.37), Mean (2007 Survey) (7.41, 1.93, 2.36), Standard Deviation (2007 Survey) (2.56, 1.31, 0.80) and Cronbach's Alpha (2007 Survey) (0.65, 0.48, 0.37). Red outlines surround 2.56 and 1.31.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2007 Indices”. The outlines mark the standard deviations 2.56 and 1.31 in the row for the 2007 survey.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N017.png" }, { "alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index. The rows are Number of Questions (17, 8, 7), Mean (2010 Survey) (8.48, 4.67, 3.95), Standard Deviation (2010 Survey) (2.81, 1.81, 1.44), Cronbach's Alpha (2010 Survey) (0.56, 0.57, 0.44), Mean (2008 Survey) (7.87, 3.88, 4.05), Standard Deviation (2008 Survey) (2.79, 1.68, 1.50) and Cronbach's Alpha (2008 Survey) (0.57, 0.49, 0.45). A red outline surrounds 2.81.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2008 Indices”. The outline marks the standard deviation 2.81 in the row for the 2010 survey.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N017-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Three standard deviations in Table A4 match truncation at the second decimal rather than rounding: 2.56, 1.31 and 2.81 should read 2.57, 1.32 and 2.82.", "id": "P043-N017", "location": "Online Appendix, p. 8, Table A4: Posttreatment Survey 2007 Indices, row “Standard Deviation (2007 Survey)”; Posttreatment Survey 2008 Indices, row “Standard Deviation (2010 Survey)”", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Posttreatment Survey 2007 Indices.} Standard Deviation (2007 Survey): \\hl{2.56} (Civil Liberties Knowledge Index*), \\hl{1.31} (General Political Knowledge Index).\n\n\\emph{Posttreatment Survey 2008 Indices.} Standard Deviation (2010 Survey): \\hl{2.81} (Civil Liberties Knowledge Index*).", "corrected": "\\emph{Posttreatment Survey 2007 Indices.} Standard Deviation (2007 Survey): \\hl{2.57} (Civil Liberties Knowledge Index*), \\hl{1.32} (General Political Knowledge Index).\n\n\\emph{Posttreatment Survey 2008 Indices.} Standard Deviation (2010 Survey): \\hl{2.82} (Civil Liberties Knowledge Index*).", "evidence": "The evidence is a recalculation from the replication data, with the item sets of the replication code and the sample standard deviation (denominator $N - 1$). It gives 2.568148678 with $N = 1045$, 1.318642601 with $N = 1045$, and 2.817537565 with $N = 251$. The recalculation establishes the numerical pattern only; it does not show how the three cells came to be printed this way.", "problem": "The three standard deviations are 2.568, 1.319 and 2.818 to three decimals. Rounded to two decimals they are 2.57, 1.32 and 2.82. The printed values are the same numbers with the third decimal dropped. Table 3 (p.~472) prints 1.32 for the same five-item 2007 General Political Knowledge Index.", "what_survives": "The definitions of the indices and every finding of the paper are unaffected. Each of the three cells changes by 0.01." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N017", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index. The rows are Number of Questions (17, 8, 7), Mean (2010 Survey) (8.48, 4.67, 3.95), Standard Deviation (2010 Survey) (2.81, 1.81, 1.44), Cronbach's Alpha (2010 Survey) (0.56, 0.57, 0.44), Mean (2008 Survey) (7.87, 3.88, 4.05), Standard Deviation (2008 Survey) (2.79, 1.68, 1.50) and Cronbach's Alpha (2008 Survey) (0.57, 0.49, 0.45). A red outline surrounds the heading General Political Knowledge Index with its asterisk and the entry 8 below it.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2008 Indices”. The outline marks the heading of the General Political Knowledge Index column, whose asterisk refers to the note on subsets of questions, and its number of questions, 8.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N018.png" }, { "alt": "Crop of four lines of notes. The text reads: asterisk, Index consists of a subset of questions originally included. Notes: All questions rescaled to 0 to 1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007 to 2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey. A red outline in two parts surrounds the last sentence, from For the 2007 to 2009 surveys to the 2010 survey.", "caption": "Online Appendix, p. 8, the notes under Table A4. The outline marks the sentence that describes the 2007–2009 statistics as those of the equivalent indices made of the questions included on the 2010 survey; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N018-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The 2008 and 2010 versions of the General Political Knowledge Index, each with eight questions, have seven questions in common; the eighth asks for the name of the town’s head official in 2008 and for the name of the governor in 2010, and the column does not compare one fixed set of questions.", "id": "P043-N018", "location": "Online Appendix, p. 8, Table A4, Posttreatment Survey 2008 Indices, column “General Political Knowledge Index*”, and the note under the table; the note to Table A5, p. 9, describes the same indices", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Table A4, Posttreatment Survey 2008 Indices, General Political Knowledge Index*.} Number of Questions 8; Mean (2010 Survey) 4.67; Standard Deviation (2010 Survey) 1.81; Cronbach's Alpha (2010 Survey) 0.57; Mean (2008 Survey) 3.88; Standard Deviation (2008 Survey) 1.68; Cronbach's Alpha (2008 Survey) 0.49.\n\n* Index consists of a subset of questions originally included. Notes: All questions rescaled to 0--1 for additive index. N is 251 for all indices in 2010 Survey. \\hl{For the 2007--2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey.}", "corrected": "Keep the entries of Tables A4 and A5 and add to both notes: \\hl{The 2008 General Political Knowledge Index versions share seven questions. The eighth question asks for the town or city head official's name in 2008 and for the governor's name in 2010. The eight-question versions therefore contain different questions, and their mean difference does not compare one fixed set of questions.}\n\nAlternatively, compare the seven shared questions. The column of Table A4 then reads: Number of Questions \\hl{7}; Mean (2010 Survey) \\hl{3.93}; Standard Deviation (2010 Survey) \\hl{1.65}; Cronbach's Alpha (2010 Survey) \\hl{0.55}; Mean (2008 Survey) \\hl{3.51}; Standard Deviation (2008 Survey) \\hl{1.47}; Cronbach's Alpha (2008 Survey) \\hl{0.43}. The sixteen correlations of Table A5 that involve this index change as follows, with the other eight indices as built in the replication code.\n\\begin{audittable}{lrrrr}{Correlations of the 2008 General Political Knowledge Index in Table A5, as printed and with the seven shared questions: the lower triangle (2010 survey) and the upper triangle (2007--2009 surveys).}\nOther index & Lower, printed & Lower, seven & Upper, printed & Upper, seven \\\\\nCivil Liberties Knowledge 2007 & 0.440 & 0.436 & 0.411 & 0.414 \\\\\nCivil Liberties Knowledge 2008 & 0.559 & 0.532 & 0.468 & 0.444 \\\\\nCivil Liberties Knowledge 2009 & 0.616 & 0.610 & 0.420 & 0.426 \\\\\nGeneral Political Knowledge 2007 & 0.615 & 0.509 & 0.487 & 0.441 \\\\\nGeneral Political Knowledge 2009 & 0.486 & 0.481 & 0.408 & 0.389 \\\\\nSupport for Civil Liberties 2007 & 0.247 & 0.267 & 0.106 & 0.111 \\\\\nSupport for Civil Liberties 2008 & 0.215 & 0.256 & 0.133 & 0.129 \\\\\nSupport for Civil Liberties 2009 & 0.205 & 0.233 & 0.164 & 0.157 \\\\\n\\end{audittable}", "evidence": "The evidence is a check of the two questionnaires and of the replication code, and a recalculation from the replication data in exact arithmetic. The six statistics printed in the column reproduce at the printed precision for the scores as built ($N = 591$ in 2008 and $N = 251$ in 2010). The question on the local official's name was answered correctly by 216 of 591 respondents in 2008, and the question on the governor's name by 186 of 251 in 2010. Each score on eight questions is the score on the seven shared questions plus the question that differs. For the 2010 mean minus the 2008 mean, I have\n\\[\n\\frac{1172}{251} - \\frac{764}{197} \\;=\\; \\Bigl( \\frac{986}{251} - \\frac{692}{197} \\Bigr) + \\Bigl( \\frac{186}{251} - \\frac{72}{197} \\Bigr),\n\\qquad\\text{that is,}\\qquad\n\\frac{39120}{49447} = \\frac{20550}{49447} + \\frac{18570}{49447} .\n\\]\nThe three terms are approximately 0.791, 0.416 and 0.376. The two questions that differ account for $619/1304$, about 47.5\\%, of the difference between the means on eight questions. This is arithmetic on two samples. It is not an estimate of what the substitution caused, since the 2010 respondents were not asked for the local official's name.\n\\begin{audittable}{lrrrr}{The column as printed and the same statistics for the seven questions asked in both surveys.}\nScore & Questions & Mean & Standard deviation & Alpha \\\\\n2008 survey, as printed & 8 & 3.88 & 1.68 & 0.49 \\\\\n2010 survey, as printed & 8 & 4.67 & 1.81 & 0.57 \\\\\n2008 survey, seven shared questions & 7 & 3.51 & 1.47 & 0.43 \\\\\n2010 survey, seven shared questions & 7 & 3.93 & 1.65 & 0.55 \\\\\n\\end{audittable}", "problem": "The note says that each index of 2007--2009 is restricted to the questions that were asked again in 2010. For this column that is not the case. The 2008 questionnaire asks for the title of the head official of the respondent's town or city and then asks ``What is that person's name?'' The 2010 questionnaire asks for the same title and then asks the respondent to name the governor of the state; it has no question on the name of the local official. The replication code builds the 2008 score with the question on the local official's name and the 2010 score with the question on the governor's name; the other seven questions are the same.\n\nThe printed statistics are correct for the two scores as built. They do not describe the same eight questions. The difference between the two means, $4.67 - 3.88 = 0.79$, combines a difference between the two groups of respondents with a difference between two questions.", "what_survives": "Every printed number in the column is accurate for the score it was computed from. The figure on p.~11 of the Online Appendix calibrates the Civil Liberties Knowledge indices, which contain neither of these two questions, and the treatment-effect regressions use the full 22-question 2008 index of Table 3; neither changes. What changes is the description of this column, and of the corresponding entries of Table A5, as a comparison of equivalent questions." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N018", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of four lines of running text. The text reads: the prosecutor receives new evidence that he believes will result in a conviction. Does the Constitution allow the prosecutor to retry Mr. Jones in this case? (Yes/No) 12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government] 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case] 14. School. A red outline surrounds the bracket Identification: Federal, state, or local branch of government.", "caption": "Mini Appendix, p. 2, list of the 2008 Civil Liberties Knowledge Index questions. The outline marks the bracket that describes item 12.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N019.png" } ], "formal_release": null, "formal_statements": [], "headline": "Item 12 of the 2008 list asks which branch of the federal government can declare a law unconstitutional; it is not a question about the federal, state or local level.", "id": "P043-N019", "location": "Mini Appendix, p. 2 (pages unnumbered), “2008 Civil Liberties Knowledge Index Questions”, item 12", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "12. Declare a law unconstitutional \\hl{[Identification: Federal, state, or local branch of government]}", "corrected": "12. Declare a law unconstitutional \\hl{[Identification: Which branch of the federal government?]}", "evidence": "The evidence is a check of the pages: the Mini Appendix, Table A2 of the Online Appendix, and the 2008 questionnaire with its coding key in the replication files.", "problem": "The bracket states the type of question. In the 2008 questionnaire this item stands under the heading Separation of Powers, with the choices Executive, Legislative, Judicial, and Both Executive and Legislative, and the coding key scores Judicial as correct. The questions on the federal, state and local levels of government form a different group of items in that questionnaire. Table A2 of the Online Appendix (p.~4) labels the same item ``[Which branch?]''.", "what_survives": "The question, its key, the coding and the results are unaffected; only the label in the Mini Appendix changes. The numbering of the 2009 list on the same page, which passes from 1 to 3, is not an error, since the request to name two rights is scored as two answers, which Table A2 shows as (1) and (2)." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N019", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Table 4, Generalized Least Squares Estimates of Experimental Treatment Effects on Civil Liberties Knowledge, General Political Knowledge, and Support for Civil Liberties. Three panels, one for each index, each give six treatment estimates with standard errors in parentheses: without and with covariates for the 2007, 2008 and 2009 posttreatment surveys. Red outlines surround six entries in the columns without covariates: civil liberties knowledge 1.248 (0.181) for 2007 and 0.796 (0.228) for 2008; general political knowledge 0.058 (0.113) for 2007, 0.055 (0.279) for 2008 and -0.004 (0.204) for 2009; support for civil liberties 0.026 (0.150) for 2008. The notes below the table read: Standard errors in parentheses. Clusters for random effects model are at the classroom level.", "caption": "Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the six unadjusted estimates, with their standard errors, that the R script does not return at three decimals.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N020.png" } ], "formal_release": null, "formal_statements": [], "headline": "The R script of the replication files fits the unadjusted models of Table 4 by restricted maximum likelihood and not by the random-effects GLS with a moment estimate of the classroom variance, which reproduces the printed table; six of its nine estimates with standard errors differ from the print in the third decimal, and the printed table is supported.", "id": "P043-N020", "location": "Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 198–205 and 227–234; the paper’s counterparts are Table 4, the “Covariates: No” columns, p. 474 (PDF p. 12), and the description of the estimator on p. 472 (PDF p. 10)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{R script, lines 198, 202 and 203 (line 202 is broken here after its third argument); lines 227--234 repeat the construction for the three support indices.}\n\\[\n\\begin{aligned}\n&\\texttt{ivs <- c(\"treatment\", \"stratum\")}\\\\\n&\\texttt{f <- as.formula(paste(name,\" \\textasciitilde\", paste(ivs, collapse=\"+\"),}\\\\\n&\\qquad\\texttt{\"+ (1|classcode)\", sep=\"\"))}\\\\\n&\\texttt{command <- paste(\"GLS.\",name,\" <- }\\hl{\\texttt{lmer(formula=f, data=data)}}\\texttt{\", sep=\"\");}\n\\end{aligned}\n\\]\n\n\\emph{Statistical model, p.~472.} We use Generalized Least Squares (GLS) regression with random effects at the classroom level to estimate average treatment effects (Green and Vavreck 2008).", "corrected": "The Table~4 block of the R script should compute the moment-based GLS of the Stata commands in place of the call to \\hl{\\texttt{lmer}}, or the replication should be run through the Stata commands. For one outcome, let classroom $i$ have $n_i$ students with outcomes $y_{ij}$ and mean $\\bar y_i$, let $G$ be the number of classrooms and $N = \\sum_i n_i$. Let $x_i$ be the \\hl{column vector} of regressors of classroom $i$ (the intercept, the treatment and the stratum indicators, all constant within a classroom), and let $k$ be the rank of the design. Let $b_B$ be the least-squares coefficient of $\\bar y_i$ on $\\hl{x_i^{\\top}}$ over the $G$ classrooms. Define\n\\[\n\\begin{aligned}\n\\widehat\\sigma_e^2 &= \\frac{1}{N-G}\\sum_i\\sum_j (y_{ij}-\\bar y_i)^2, \\qquad\n\\widehat\\sigma_B^2 = \\frac{1}{G-k}\\sum_i (\\bar y_i - x_i^{\\top} b_B)^2,\\\\\n\\widehat\\sigma_u^2 &= \\max\\Bigl\\{0,\\ \\widehat\\sigma_B^2 - \\widehat\\sigma_e^2\\,\\frac{1}{G}\\sum_i \\frac{1}{n_i}\\Bigr\\}, \\qquad\n\\theta_i = 1 - \\sqrt{\\frac{\\widehat\\sigma_e^2}{\\widehat\\sigma_e^2 + n_i\\,\\widehat\\sigma_u^2}} .\n\\end{aligned}\n\\]\nThe estimate $b$ is the least-squares coefficient of $y^*_{ij} = y_{ij} - \\theta_i \\bar y_i$ on $x^*_{ij} = (1-\\theta_i)\\,x_i$ over the $N$ students. Its covariance matrix is $s^2 (X^{*\\top}X^{*})^{-1}$, where $X^*$ has rows $x^{*\\top}_{ij}$ and $s^2 = \\sum_{i}\\sum_{j} (y^*_{ij} - x^{*\\top}_{ij} b)^2/(N-k)$. Setting \\texttt{REML=FALSE} in \\texttt{lmer} is not this repair, since it selects maximum likelihood. Table~4 stays as printed.", "evidence": "The evidence is a recalculation on the public file and an exact check. The nine models of the R script were fitted as written. The moment-based GLS of the corrected block below was computed on the same outcomes, strata, classrooms and samples.\n\\begin{audittable}{lrrr}{Table~4, ``Covariates: No'': treatment estimate and standard error as printed, by moment-based GLS, and from the R script. Shaded entries differ from the print at three decimals.}\nIndex and wave & Printed & Moment-based GLS & R script (REML) \\\\\nCivil liberties knowledge, 2007 & 1.248 (0.181) & 1.248136 (0.180988) & $\\hl{1.250977\\ (0.186482)}$ \\\\\nCivil liberties knowledge, 2008 & 0.796 (0.228) & 0.796109 (0.227714) & $\\hl{0.794538\\ (0.246765)}$ \\\\\nCivil liberties knowledge, 2009 & 0.492 (0.268) & 0.492164 (0.267887) & 0.492164 (0.267887) \\\\\nGeneral political knowledge, 2007 & 0.058 (0.113) & 0.058406 (0.113359) & $\\hl{0.059166\\ (0.114956)}$ \\\\\nGeneral political knowledge, 2008 & 0.055 (0.279) & 0.054524 (0.278920) & $\\hl{0.059257\\ (0.291241)}$ \\\\\nGeneral political knowledge, 2009 & $-0.004$ (0.204) & $-0.004015$ (0.204283) & $\\hl{-0.003581\\ (0.200669)}$ \\\\\nSupport for civil liberties, 2007 & $-0.061$ (0.050) & $-0.060545$ (0.050440) & $-0.060545$ (0.050440) \\\\\nSupport for civil liberties, 2008 & 0.026 (0.150) & 0.026044 (0.149720) & $\\hl{0.023627\\ (0.159055)}$ \\\\\nSupport for civil liberties, 2009 & 0.022 (0.118) & 0.022227 (0.117634) & 0.022227 (0.117634) \\\\\n\\end{audittable}\nThe moment-based GLS agrees with all nine printed entries to rounding. For the exact check, I take the 2008 model for civil liberties knowledge, which has 591 students, 40 classrooms and a design of rank 12. Computing the moment estimates in exact rational arithmetic, I obtain the variance ratio $\\lambda = \\widehat\\sigma_u^2/\\widehat\\sigma_e^2 \\approx 0.000465 > 0$. The derivative of the profiled REML criterion at this $\\lambda$ is about $-80.48$, and exact arithmetic confirms that it is negative. Since $\\lambda$ is an interior point and the derivative is not zero, I conclude that the moment estimate is not a REML optimum. The two procedures differ on these data, and the differences in the table are not rounding in the optimizer. Stata was not available, and the Stata script was not run.", "problem": "The block of the R script is headed TABLE 4 and builds a table named \\texttt{TABLE4.nocov}. It calls \\texttt{lmer} without the argument \\texttt{REML}, whose default is restricted maximum likelihood (REML). The Stata script of the same files fits these nine models with \\texttt{xtreg} and the option \\texttt{re}, which is a random-effects GLS that estimates the classroom variance by the method of moments; it is called moment-based GLS below. Both procedures are GLS with a classroom random effect in the sense of p.~472, since the REML coefficients are the GLS coefficients for the covariance that REML estimates. The sentence on p.~472 is not wrong. The two procedures estimate the classroom variance differently, and the output of the R script does not reproduce the printed table. Six of the nine estimates with standard errors differ from the print at three decimals, and the standard errors differ by up to 0.019.", "what_survives": "The nine unadjusted entries of Table~4 are supported as printed, and no entry changes. At the two-sided 5\\% level, the decisions under REML and under moment-based GLS agree in all nine models. The statement $p < 0.001$ for the 2008 knowledge effect (p.~473) holds for the unadjusted GLS entry in print, whose normal $p$-value is 0.00047; under the REML fit of the R script it would be 0.0013. The adjusted columns are not touched by this finding. The R script changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N020", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of Table 4 with its notes. Three panels, for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, each give six treatment estimates with standard errors, and rows for Covariates (No or Yes), Strata (Yes) and N. Red outlines surround the nine columns with Covariates equal to Yes: 1.220 (0.155), 0.762 (0.226) and 0.221 (0.269) for civil liberties knowledge; 0.023 (0.076), 0.051 (0.259) and -0.037 (0.202) for general political knowledge; -0.071 (0.051), -0.073 (0.131) and -0.080 (0.119) for support for civil liberties. The second note reads: Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother's highest level of education, and seven factual questions measuring political knowledge.", "caption": "Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the nine covariate-adjusted models (estimate, standard error and the entry “Yes” in the Covariates row), which the R script does not fit.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N021.png" } ], "formal_release": null, "formal_statements": [], "headline": "The R script of the replication files does not fit the nine covariate-adjusted models of Table 4, and it fits the item models without the pretreatment covariates that the tables report and the essay models without those specified in the Stata script; no printed number is shown to be wrong.", "id": "P043-N021", "location": "Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 193 and 224, with the Table 4 block (lines 181–253), the item blocks (lines 442–583) and the essay models (lines 613 and 615); the paper’s counterparts are Table 4, the “Covariates: Yes” columns, p. 474 (PDF p. 12), Tables A1 to A3 of the online appendix, and footnote 13, p. 473 (PDF p. 11)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{R script, lines 193 and 224 (line 224 is broken here after its first sentence).}\n\\[\n\\begin{aligned}\n&\\hl{\\texttt{\\#Cannot do random effects with covariates - matrices will not invert}}\\\\\n&\\hl{\\texttt{\\#Cannot perform random effects regression with covariates.}}\\\\\n&\\qquad\\hl{\\texttt{Too many dummy variables results in uninvertible matrix.}}\n\\end{aligned}\n\\]\n\n\\emph{Table~4, notes, p.~474.} $^{\\dagger}$Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother's highest level of education, and seven factual questions measuring political knowledge.\n\n\\emph{Table~A1, note, p.~2 of the online appendix.} All estimates include covariates and fixed effects for strata.", "corrected": "The R script should \\hl{fit the nine covariate-adjusted models of Table~4} and \\hl{include the pretreatment controls in the item and essay models}. The controls are the categorical variables \\texttt{Aq1}, \\texttt{Aq3}, \\texttt{Aq4}, \\texttt{Aq18} to \\texttt{Aq24}, \\texttt{Aq28}, \\texttt{Aq32} and \\texttt{Aq33year} of the Stata commands, with the same coding of a missing pretest, the same handling of redundant indicator columns and the same estimator. Its output should then be compared with the Stata output on one dataset, since adding the terms to \\texttt{lmer} does not by itself establish numerical agreement. The comments at lines 193 and 224 should be replaced by a statement of what the script does not compute. No entry of Table~4 or of Tables~A1 to~A3 changes.", "evidence": "The evidence is a check of the pages and of the two scripts. Evaluating the formula loops of the R script, with the fitting function replaced by one that records its formula, I obtain 114 formulas ($9 + 17 + 49 + 37 + 2$), each with exactly the variables \\texttt{treatment}, \\texttt{stratum} and \\texttt{classcode}. The matching Stata commands (lines 264--270 and 286--288 for Table~4, lines 419--476 for the items, lines 483 and 485 for the essays) each include the 13 variables \\texttt{Aq1}, \\texttt{Aq3}, \\texttt{Aq4}, \\texttt{Aq18} to \\texttt{Aq24}, \\texttt{Aq28}, \\texttt{Aq32} and \\texttt{Aq33year}. I did not fit a model or run the Stata script.", "problem": "The block of the R script headed TABLE 4 requests nine models, each with the treatment, the strata and a classroom random intercept. Two comments say that the models with covariates are left out because the matrices will not invert. The 17 item models for Table~A1, the 49 and 37 item models for Tables~A2 and~A3 and the two essay models have the same right-hand side. The R script alone therefore cannot reproduce the ``Covariates: Yes'' columns of Table~4 or the item tables of the appendix, whose notes say that all estimates include covariates. The Stata script of the same files specifies each of these models with 13 pretreatment variables as categorical controls. The comments describe a difficulty of this R implementation. They do not show that the adjusted models cannot be fitted, and the paper does not say that the R script produced the printed adjusted entries.", "what_survives": "This omission does not refute the printed tables, the statistical model or the conclusions of the paper. The unadjusted specifications of the R script and the adjusted specifications of the Stata script stand. The R script changes, since at present it cannot serve to check the adjusted results." }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N021", "work": { "arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/" }, "work_id": "P043" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P043.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:d47cef1022f975a78d85677e5325b6c17e7927181277e624078da62ede8fe630", "conditional_on": [], "credit_reason": "This is a witness for the unconditional extension specified in the current target, not a refutation of the paper’s contextual D≥0 conditional: both witnesses have D<0. No prior-record fact changes the claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P043.mediation_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P043/P043_Formalization.lean#L605", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:d47cef1022f975a78d85677e5325b6c17e7927181277e624078da62ede8fe630", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P043-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P043-T1" }, "html": { "body": "

p. 474 (PDF p. 12), below Table 4, referring to p. 473 (PDF p. 11). The two specified validator constructions are exact randomized counterexamples to an unconditional mediation comparison. In the linear construction all eight states have mass 1/8, knowledge increases by 6/5, support increases by h/20 with knowledge, M=3/50 and D=-3/50, and observed support cancels assignment pointwise. Both arm laws, all covariances, population regression slopes, and structural coefficient identities are as specified. In the integer construction all 1000 states have mass 1/1000; K(0)=8 and K(1)=9 for W<=4 or 10 for W=5, with W stored as i.1.val+1. Knowledge, support and constant general knowledge respect the 2007 index ranges. All stated arm means, nested means and one-unit effects hold; both support arm means are 59/25 and both support laws put mass 9/25 at 3. In each construction D<0, M>0 and tau=0<M, so the design alone gives neither tau>=M nor the null inference M<=0. The contextual comparison assuming D>=0 is not refuted. The usual measure-theoretic conditions are assumed; the specified probability spaces are finite and all expectations exist.

", "headline": "Two counterexamples to an unconditional extension of the mediation comparison; the paper’s contextual conditional is not refuted", "location": "p. 474 (PDF p. 12), left column below Table 4; the passage referred to is on p. 473 (PDF p. 11), left column" }, "id": "P043-T1", "text": { "body": "p. 474 (PDF p. 12), below Table 4, referring to p. 473 (PDF p. 11). The two specified validator constructions are exact randomized counterexamples to an unconditional mediation comparison. In the linear construction all eight states have mass 1/8, knowledge increases by 6/5, support increases by h/20 with knowledge, M=3/50 and D=-3/50, and observed support cancels assignment pointwise. Both arm laws, all covariances, population regression slopes, and structural coefficient identities are as specified. In the integer construction all 1000 states have mass 1/1000; K(0)=8 and K(1)=9 for W<=4 or 10 for W=5, with W stored as i.1.val+1. Knowledge, support and constant general knowledge respect the 2007 index ranges. All stated arm means, nested means and one-unit effects hold; both support arm means are 59/25 and both support laws put mass 9/25 at 3. In each construction D<0, M>0 and tau=0=M nor the null inference M<=0. The contextual comparison assuming D>=0 is not refuted. The usual measure-theoretic conditions are assumed; the specified probability spaces are finite and all expectations exist.", "headline": "Two counterexamples to an unconditional extension of the mediation comparison; the paper’s contextual conditional is not refuted", "location": "p. 474 (PDF p. 12), left column below Table 4; the passage referred to is on p. 473 (PDF p. 11), left column" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T1" }, { "declarations": [ { "claim_revision_id": "catalog:e05581f82242343979d09079f3050571e9e37bc1f60d74f3240650ac811b86e5", "conditional_on": [], "credit_reason": "MediationLawClaims and LinearMediationLawClaims only package conclusions, not assumptions. The probability hypothesis and 0p. 474 (PDF p. 12), below Table 4, and p. 473 (PDF p. 11). For arbitrary probability laws with the five specified finite expectations, independent nondegenerate assignment identifies the mediation decomposition, its weak and strict sign comparisons and null inferences, the linear and positive-first-stage ratio formulas, and the sufficient but unnecessary channelwise sign condition; the fixed witness records failure without the net sign premise.

", "headline": "The mediation comparison with its contextual sign condition made explicit", "location": "p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column" }, "id": "P043-T2", "text": { "body": "p. 474 (PDF p. 12), below Table 4, and p. 473 (PDF p. 11). For arbitrary probability laws with the five specified finite expectations, independent nondegenerate assignment identifies the mediation decomposition, its weak and strict sign comparisons and null inferences, the linear and positive-first-stage ratio formulas, and the sufficient but unnecessary channelwise sign condition; the fixed witness records failure without the net sign premise.", "headline": "The mediation comparison with its contextual sign condition made explicit", "location": "p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T2" }, { "declarations": [ { "claim_revision_id": "catalog:652b07c7c7e6bda12d42c1bbbd1fc7f975f94370ede7f20991fa2872b8e4d044", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:652b07c7c7e6bda12d42c1bbbd1fc7f975f94370ede7f20991fa2872b8e4d044", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P043-T3", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P043-T3" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Results, sentence beginning 'This pattern of results is striking', with the reduced-form passage of the Statistical Model section that it cites; as printed", "location": "p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column" }, "id": "P043-T3", "text": { "body": "No current theorem is supplied for this target.", "headline": "Results, sentence beginning 'This pattern of results is striking', with the reduced-form passage of the Statistical Model section that it cites; as printed", "location": "p. 474 (PDF p. 12), left column below Table 4; p. 473 (PDF p. 11), left column" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T3" }, { "declarations": [ { "claim_revision_id": "catalog:f0e8ab5d0250cd80197b392f1ddcac1d24a485b944a23bb935c52b348e95bd2e", "conditional_on": [], "credit_reason": "The fixed models satisfy exclusion, consistency, full assignment independence, positive first stage and the specified ranges; knowledge monotonicity is not a premise of the assertion being tested. IncreaserNonidentificationClaim is a real negated-existence conclusion for two indistinguishable laws with unequal effects, which suffices to refute identification on the larger admissible class. No claim-stage verdict change.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P043.iv_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P043/P043_Formalization.lean#L870", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f0e8ab5d0250cd80197b392f1ddcac1d24a485b944a23bb935c52b348e95bd2e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P043-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P043-T4" }, "html": { "body": "

p. 472 (PDF p. 10) and pp. 474-475 (PDF pp. 12-13). The specified two-type and two five-type validator constructions satisfy fair independent assignment, exclusion, consistency and a strictly positive first stage. The two-type witness has the stated arm means and covariances, zero IV and covariance ratios, increaser mass 2/3 and increaser effect one. Every increaser crosses only margin 9 with increment one; every normalized nonnegative average of these increments is one, different from the zero IV ratio. This includes arbitrary probability averages, whose integrability is certified. Cancellation explains the zero reduced form, and responses are nondecreasing with the stated 2008 index bounds. The five-type models have the specified complete conditional and joint observable laws, common positive first stage 1/5 and zero reduced form and IV ratio, but increaser probabilities 2/5 and 3/5 and effects one and 4/3 respectively. Every increase is one unit, so these are also the average per-unit effects. No functional of the observable law recovers the increaser effect in both admissible models, hence none works on the entire claimed class. These are validator witnesses, not empirical claims about the curriculum. The usual measure-theoretic conditions are assumed; all finite expectations exist and every conditioning or ratio denominator used here is positive.

", "headline": "Statistical Model, 'then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable', and Conclusion, 'the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum'; the witnesses are the validator's constructions and are not in the paper", "location": "p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)" }, "id": "P043-T4", "text": { "body": "p. 472 (PDF p. 10) and pp. 474-475 (PDF pp. 12-13). The specified two-type and two five-type validator constructions satisfy fair independent assignment, exclusion, consistency and a strictly positive first stage. The two-type witness has the stated arm means and covariances, zero IV and covariance ratios, increaser mass 2/3 and increaser effect one. Every increaser crosses only margin 9 with increment one; every normalized nonnegative average of these increments is one, different from the zero IV ratio. This includes arbitrary probability averages, whose integrability is certified. Cancellation explains the zero reduced form, and responses are nondecreasing with the stated 2008 index bounds. The five-type models have the specified complete conditional and joint observable laws, common positive first stage 1/5 and zero reduced form and IV ratio, but increaser probabilities 2/5 and 3/5 and effects one and 4/3 respectively. Every increase is one unit, so these are also the average per-unit effects. No functional of the observable law recovers the increaser effect in both admissible models, hence none works on the entire claimed class. These are validator witnesses, not empirical claims about the curriculum. The usual measure-theoretic conditions are assumed; all finite expectations exist and every conditioning or ratio denominator used here is positive.", "headline": "Statistical Model, 'then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable', and Conclusion, 'the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum'; the witnesses are the validator's constructions and are not in the paper", "location": "p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T4" }, { "declarations": [ { "claim_revision_id": "catalog:46a690134d6cb46cbb05d882cf5b09aa77f5851d21fbaa9da9d692c62b782cfc", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P043/P043_Formalization.lean:1293 requires [Fintype Ω], and Population and mean at P043/P043_Formalization.lean:20 and :27 encode only finite weighted population laws. P043-T5 instead specifies a population \"with a probability law\" and finite-expectation real potential responses, without a finite-support hypothesis. For example, independent fair assignment, U uniform on [0,1], J=1, K(0)=0, K(1)=1 and Y(k)=kU satisfy H1–H5 and give β_IV=1/2, but their continuous response law cannot be represented by this finite population type. The statement that a finite probability space \"suffices\" supplies a possible realization, not a finiteness assumption on every population in the claim. The reviewer’s secondary signed-membership observation is only a packaging issue: the unrestricted membership statement is separately present at P043/P043_Formalization.lean:1107, and the signed telescoping and downward-crossing conclusions are exported by iv_repair.', 'suggested_fix': 'Generalize Population and the expectation, event-mean, arm-mean and covariance definitions to arbitrary probability spaces, and restate iv_repair with the measurability and finite-expectation conditions of T5. Keep the bounded integer doses, individual monotonicity, positive first stage and existing conclusions; a description-only change would not establish the current target.', 'second_review': \"P043/P043_Formalization.lean:1293 requires [Fintype Ω], and Population at P043/P043_Formalization.lean:20 and mean at P043/P043_Formalization.lean:27 restrict population laws and expectations to finite weighted sums, whereas P043-T5 specifies a population 'with a probability law' and finite-expectation real potential responses without a finite-support hypothesis. Neither the standing observation that 'a finite probability space suffices' nor the description's disclosed finite-population choice imposes finiteness on the target's quantifiers. For example, independent fair assignment, U uniform on [0,1], J=1, K(0)=0, K(1)=1 and Y(k)=kU satisfy H1–H5 and give β_IV=1/2, but the continuous treated-response law cannot be represented on a finite Ω, so the excluded cases are substantive. The declaration establishes the intended IV component in its finite domain, including the normalized margin formula at P043/P043_Formalization.lean:1232, so retain that credit through a statement repair generalizing the probability and expectation framework with appropriate measurability and integrability hypotheses.\", 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P043.iv_repair", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P043/P043_Formalization.lean#L1152", "status": "proved", "support_scope": "limited", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:46a690134d6cb46cbb05d882cf5b09aa77f5851d21fbaa9da9d692c62b782cfc", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P043-T5", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P043-T5" }, "html": { "body": "

p. 472 (PDF p. 10) and pp. 474–475 (PDF pp. 12–13). Individual knowledge monotonicity and a positive first stage yield the normalized crossed-margin IV formula and its finite-population identities.

", "headline": "The local-effect statements of p. 472 and pp. 474-475, corrected: the instrumental-variable ratio under knowledge monotonicity", "location": "p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)" }, "id": "P043-T5", "text": { "body": "p. 472 (PDF p. 10) and pp. 474–475 (PDF pp. 12–13). Individual knowledge monotonicity and a positive first stage yield the normalized crossed-margin IV formula and its finite-population identities.", "headline": "The local-effect statements of p. 472 and pp. 474-475, corrected: the instrumental-variable ratio under knowledge monotonicity", "location": "p. 472 (PDF p. 10), left column, last lines, to right column, first lines; pp. 474-475 (PDF pp. 12-13)" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T5" }, { "declarations": [ { "claim_revision_id": "catalog:71884ae97732185165ccd394f29b45fadd8da6c845e90f42ba54ba2fdc424192", "conditional_on": [], "credit_reason": "The statement concerns corrected notation and equivalent equations, matching this target’s express scope; it does not assert causal interpretation of β₁ or a GLS property. No claim-stage verdict change.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P043.corrected_strata", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P043/P043_Strata.lean#L68", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:71884ae97732185165ccd394f29b45fadd8da6c845e90f42ba54ba2fdc424192", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P043-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P043-T6" }, "html": { "body": "

equation (1), p. 471 (PDF p. 9), right column. Correctly indexed stratum indicators select the nonreference offset, encode the omitted reference indicator, and give the equivalent stratum-intercept equation.

", "headline": "Equation (1) and the sentence explaining it, with the stratum indicators indexed consistently (DikD_{ik}Dik​; printed DkD_kDk​ in the display and DiD_iDi​ in the sentence)", "location": "p. 471 (PDF p. 9), right column" }, "id": "P043-T6", "text": { "body": "equation (1), p. 471 (PDF p. 9), right column. Correctly indexed stratum indicators select the nonreference offset, encode the omitted reference indicator, and give the equivalent stratum-intercept equation.", "headline": "Equation (1) and the sentence explaining it, with the stratum indicators indexed consistently ($D_{ik}$; printed $D_k$ in the display and $D_i$ in the sentence)", "location": "p. 471 (PDF p. 9), right column" }, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T6" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "partial", "targets": [ { "agreement": "agrees_in_part", "location": "p. 474 (PDF p. 12), first panel; discussed p. 473 (PDF p. 11)", "method": "I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.", "status": "partial", "target": "Table 4: civil liberties knowledge" }, { "agreement": "agrees_in_part", "location": "p. 474 (PDF p. 12), second panel", "method": "I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.", "status": "partial", "target": "Table 4: general political knowledge" }, { "agreement": "agrees_in_part", "location": "p. 474 (PDF p. 12), third panel", "method": "I translated the Stata random-effects GLS specification. Without covariates: all three estimates and standard errors reproduce to three decimals, and sample sizes agree exactly (P043-N020). With covariates: original reproduction remains blocked because the original birth-year categories are unavailable; fits on the top-coded public file are diagnostics.", "status": "partial", "target": "Table 4: support for civil liberties" }, { "agreement": "to_rounding", "location": "p. 473 (PDF p. 11), Results, left column", "method": "I divided the unadjusted 2007 treatment–control difference by the pooled within-group standard deviation on the public file. The result, about 0.485, agrees with the printed Cohen’s d = 0.49 to rounding.", "status": "recalculated", "target": "In-text standardized 2007 knowledge effect" }, { "agreement": "agrees_in_part", "location": "p. 473 (PDF p. 11), footnote 13", "method": "I compared OLS calculations with the printed essay estimates on the public file (P043-N010). For 2007, restricting to the 1,045 survey respondents and setting their 201 missing scores to zero gives 0.181744 (SE 0.067518), matching 0.182 (0.068) to rounding; this coding is diagnostic. For the 591 respondents in 2008, -0.007703 (SE 0.128322) rounds to -0.008 (0.128), rather than -0.007 (0.128). Original inputs, score coding and the variance estimator remain unresolved, so exact original reproduction is blocked.", "status": "partial", "target": "Footnote 13: open-ended constitutional-rights essays" }, { "agreement": "not_compared", "location": "Official online appendix p. 2 (PDF p. 2); cited p. 473 (main PDF p. 11)", "method": "I checked the printed significance stars against the printed coefficients and standard errors. I did not independently reproduce the original covariate-adjusted item models: the public file lacks the original birth-year categories, and the R script omits the stated pretreatment covariates (P043-N021).", "status": "blocked", "target": "Appendix Table A1: individual support items" }, { "agreement": "to_rounding", "location": "Official online appendix p. 11 (PDF p. 11); described p. 473 (main PDF p. 11), right column", "method": "I recalculated the six normalized treatment and control means from the public data using the paper’s rescaling. They agree with the plotted means; P043-E007 corrects the interpretation of the control-group trend.", "status": "reproduced", "target": "Appendix figure: rescaled knowledge trend" } ] }, "summary": "More than 1000 students in 59 classrooms were randomly assigned to a civics course with or without a curriculum on constitutional rights. The curriculum raised knowledge of civil liberties, with no detected change in support for them. 23 entries are classified as errors (1 Moderate, 22 Minor). The corrected description reports the decline in the observed normalized control means by the third wave (\\fid{P043-E007}). The mediation argument is valid under its stated premise of positive effects through other channels; the experiment does not verify that premise.", "what_was_checked": "I recalculated the following from the replication data or checked them against the pages; they agree with the paper, subject to the exceptions noted.\n\\begin{itemize}\n\\item Table 2: every cell reproduces from the data, and the column totals add up; $588 + 627 = 1215$ is the total of the students assigned in Table 1.\n\\item The response rates 87.4\\% and 84.7\\% in 2007, and 53.0\\% and 52.5\\% in 2009 among the schools that remained.\n\\item The text that describes Table 1, and the 21 lessons of the curriculum listed in Appendix A.\n\\item Table 3: all sample sizes, means and standard deviations, and six of the nine reliability coefficients; the other three are corrected in \\fid{P043-E004}.\n\\item Table 4, columns without covariates: all nine coefficients and standard errors, to three decimals, with a random-effects fit on classrooms. Pooled ordinary least squares on the public data gives an approximate comparison for the columns with covariates (coefficients within about 0.03, standard errors within 0.001); top-coded birth years prevent exact reproduction of the original adjusted models (\\fid{P043-E013}).\n\\item The direction of scoring of all 17 support items, against the coding keys.\n\\item Tables A1--A3: every significance star agrees with the printed coefficient and standard error, and no support item is significant.\n\\item Table A6: eight of the nine coefficient and standard-error pairs without covariates reproduce to three decimals; the exception is \\fid{P043-E005}. The adjusted columns agree only approximately on the public data.\n\\item Table A4, apart from \\fid{P043-E003}, \\fid{P043-N017} and \\fid{P043-N018}. In Table A5 the sample sizes 251 and 387 reproduce; in its upper triangle 32 of the 36 correlations agree to three decimals and four differ by 0.001, before the correction of \\fid{P043-E003}.\n\\item The six rescaled means in the figure on p.~11 of the Online Appendix.\n\\item The 2007 correlations on p.~471: 0.423, 0.297 and 0.157, printed as .42, .30 and .16.\n\\item Cohen's $d = 0.49$, as the unadjusted difference in means divided by the pooled standard deviation within groups (0.485).\n\\item Footnote 14: of the 36 interaction tests that can be run on the public data, three have $p < 0.05$ and none remains significant after the Holm--Bonferroni correction. The coding of one moderator and the coverage of these tests are the subjects of \\fid{P043-E011} and \\fid{P043-N015}.\n\\item The argument of footnote 11, that outcome error with zero treatment--control mean difference does not bias the estimated treatment effect.\n\\item The reference list: every citation in the text has an entry, and conversely, with the one misspelling of \\fid{P043-E008}.\n\\end{itemize}\nThe two appendix files in the replication deposit are identical to the supplements used here.\n\nI did not check the claim that this is the first large-scale experimental test of the proposition, which is a claim about the literature. The cited research was not read; of the court decisions, only \\emph{Miranda v.\\ Arizona} was read, for \\fid{P043-E006}. I checked the random-effects estimates with an independent implementation of the moment estimator and with restricted maximum likelihood, since Stata was not available. The R script was not run unmodified from start to finish (\\fid{P043-E012}).\n\nThe public deposit lacks the original unedited data files, the CSV files that the R script reads, the original randomization records, the classroom observations and the teacher questionnaires. Since the public data top-code the year of birth, the covariate-adjusted columns of Tables 4, A1 and A6, the essay results of footnote 13, the age interactions and the randomization check (6.3\\%) cannot be reconstructed exactly, and the estimator behind the standard errors of the models with covariates could not be settled. These checks do not show that the paper is correct in the respects that were not examined.\n\nThe p.~474 mediation sentence expressly refers back to the positive-other-channel scenario on p.~473. Its conditional comparison is valid; random assignment does not verify its sign premise. The two constructed, exact cancellation examples violate that premise and limit only an unconditional reading. The total-effect estimates are unaffected; I did not assess the ample-power claim.", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "Donald P. Green", "P. M. Aronow", "Daniel E. Bergan", "Pamela Greene", "Celia Paris", "Beth I. Weinberger" ], "category": "Articles - Empirics", "citation": "Donald P. Green, P. M. Aronow, Daniel E. Bergan, Pamela Greene, Celia Paris, and Beth I. Weinberger (2011). Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment. Journal of Politics 73(2):463–476.", "doi": "10.1017/s0022381611000107", "edition": { "pages_in_audited_file": 14, "sha256": "30d962cb6871dd7bb5811af49a11c4704b1e5e306ae7fb5a5c3a46f98b5053bd", "source_url": "https://doi.org/10.1017/s0022381611000107", "venue": "Journal of Politics", "year": "2011" }, "id": "P043", "later_materials": null, "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P043/", "venue": "Journal of Politics", "year": "2011" } }