{ "claims": [ { "claim": "The article reports that the intervention reduced school-level disciplinary reports of peer conflict over one school year: $0.20$ reports per student in control schools, $0.06$ fewer in treatment schools, a $30\\%$ reduction and statistical significance at $P<0.05$. The 2018 correction changes the reduction to $25\\%$ and states that the effect is not statistically significant.", "claim_id": "P037-H1", "related_corrections": [ "P037-N001", "P037-N008", "P037-N010", "P037-N011", "P037-N006", "P037-N009", "P037-N019", "P037-N007", "P037-N026", "P037-N018" ], "tied_to": "SI Table S6 (peer conflict $-0.06$, SE $0.03$); Figs. 1 and 2D; 2018 correction" }, { "claim": "The article reports a larger school-level reduction in disciplinary reports where a larger share of the seed group were social referents (interaction $P=0.07$). At $20\\%$ referents it reports $0.08$ reports per student, $60\\%$ below control and twice the average effect. Under the supplement's qualified linear causal-response approximation, the composition coefficient is interpreted as the causal effect of seed-group composition.", "claim_id": "P037-H2", "related_corrections": [ "P037-N010", "P037-N011", "P037-N005", "P037-N020", "P037-N006", "P037-N007", "P037-N026", "P037-N027" ], "tied_to": "Fig. 2D; SI Table S7 (treatment effect at zero referents $\\hat\\tau_0=0.04$, SE $0.07$; composition slope $\\hat\\beta=-0.85$, SE $0.46$); SI equations (2), (4), (6)" }, { "claim": "Within treatment schools, the article reports stronger anticonflict (prescriptive) norms among students with a social-referent seed among their peers than among students exposed only to non-referent seeds or to no seed ($P<0.01$ in both tests). It likens the difference to moving from a few to about $75\\%$ of students disapproving of racial jokes.", "claim_id": "P037-H3", "related_corrections": [ "P037-E009", "P037-E006", "P037-N013", "P037-N023", "P037-E002", "P037-E003", "P037-E004", "P037-N021", "P037-N025", "P037-N031", "P037-N032", "P037-N033", "P037-N034" ], "tied_to": "Fig. 3B; SI Table S5 (prescriptive norms: social-referent seed $0.007$ (SE $0.113$), non-referent seed $-0.468$ (SE $0.163$), no seed-peer exposure in a treatment school $-0.825$ (SE $0.254$), control school $0.141$ (SE $0.075$)); SI Tables S3–S4; Figs. S3, S5, S6" }, { "claim": "The article reports that students exposed to social-referent seeds were more likely to wear an orange wristband than every other exposure group ($P<0.00001$ in each of three tests) and reported more talk with friends about conflict than control-school students ($P<0.05$).", "claim_id": "P037-H4", "related_corrections": [ "P037-E009", "P037-E006", "P037-N025", "P037-N031", "P037-N033", "P037-N034" ], "tied_to": "Figs. 3A, 3C; SI Table S5 (wristband: social-referent seed $0.232$ (SE $0.012$), non-referent seed $0.124$ (SE $0.016$), no seed-peer exposure in a treatment school $0.095$ (SE $0.025$), control school $0.002$ (SE $0.008$); talk: social-referent seed $0.747$ (SE $0.032$), control school $0.624$ (SE $0.020$))" }, { "claim": "There was no statistically significant peer-to-peer influence on disciplinary reports, and no detectable school-level difference in perceived norms between treatment and control schools.", "claim_id": "P037-H5", "related_corrections": [ "P037-E009" ], "tied_to": "Figs. 2B, 3D; SI Table S5 (peer conflict: social-referent seed $0.170$, non-referent seed $0.190$, no seed-peer exposure in a treatment school $0.152$, control school $0.200$); SI Table S6 (prescriptive norms $-0.01$, SE $0.08$); Table S7" }, { "claim": "Students in treatment schools reported more talk with friends about reducing conflict and more wristband wearing than students in control schools.", "claim_id": "P037-H6", "related_corrections": [ "P037-E006", "P037-N006", "P037-E005" ], "tied_to": "Figs. 2A, 2C; SI Table S6 (talk $0.09$, SE $0.02$; wristband $0.16$, SE $0.01$); Table S7" }, { "claim": "The article describes social referents as having greater influence per student than other seeds and suggests that future intervention groups should include as many social referents as possible.", "claim_id": "P037-H7", "related_corrections": [ "P037-N027", "P037-N007", "P037-N032" ], "tied_to": "Significance statement; Discussion p. 571; Figs. 2D and 3B-C; SI Tables S5 and S7" }, { "claim": "The article describes a design with $56$ schools and $24{,}191$ students, with $28$ schools assigned to treatment. It says networks were measured by a survey taken by the entire student body; seed groups of $20$ to $32$ students ($728$ seeds, $26$ per school on average) were drawn at random from seed-eligible students; $24\\%$ of seeds declined; attendance averaged over $55\\%$; and administrative data were available for $49$ schools.", "claim_id": "P037-H8", "related_corrections": [ "P037-N003", "P037-E008", "P037-N007", "P037-N025", "P037-N030" ], "tied_to": "SI 'Study design' (pp. 5-6); SI Table S1; SI attendance table (p. 4); Fig. 1 legend; ICPSR documentation" }, { "claim": "The article presents the network results as robust: four placebo outcomes show no social-influence effects, and an alternative randomization-inference analysis following Bond et al. gives parallel results.", "claim_id": "P037-H9", "related_corrections": [ "P037-N013", "P037-N036", "P037-N031", "P037-N034", "P037-N026" ], "tied_to": "SI Fig. S4; SI Fig. S5 and text pp. 29, 32" }, { "claim": "Social referents differ from other students on observable traits (more likely to date, to be complimented on their house, to have older siblings).", "claim_id": "P037-H10", "related_corrections": [ "P037-N037" ], "tied_to": "SI Table S2" }, { "claim": "The article reports that its estimates imply a reduction in disciplinary events from $2{,}695$ to $2{,}012$ across the $11{,}938$ students in treatment schools.", "claim_id": "P037-H11", "related_corrections": [ "P037-N010", "P037-N009" ], "tied_to": "Discussion p. 571; SI Table S6; 2018 correction" } ], "date_published": "2026-10-25", "findings": [ { "before_review": { "class": "error", "severity": "Major" }, "class": "not_confirmed", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of nine lines of text. Red boxes mark the last three lines: 'any differences between the proportion of treated seeds who were referents and the proportion of seed-eligibles who were referents are attributable to randomization. As a result, our regression'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), foot of the left column. The marked lines attribute the differences in composition to randomization and begin the conclusion with “As a result”. The sentence continues at the head of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001.png" }, { "alt": "Crop of seven lines of text. Red boxes mark the first two lines: 'strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group on school-wide conflict and other outcomes.'", "caption": "Methods and Materials, p. 568 (PDF p. 3), head of the right column. The marked lines complete the sentence, which claims an estimate of the causal effect of the composition of the seed group.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-b.png" }, { "alt": "Crop of equation (2) and the six lines after it. The equation sets the outcome of school i in block j equal to a block effect, plus tau-zero times the treatment D, plus beta times the composition C of the seed group times D, plus gamma times the composition C-superscript-E of the seed-eligible group, plus an error term. Red boxes mark the end of the passage: 'As the expectation of C is proportional to C-superscript-E, controlling for C-superscript-E ensures that any random deviations in C from its expectation may be'.", "caption": "SI Appendix, p. 13, equation (2) and its interpretation. The passage expressly qualifies the interaction model’s causal interpretation by validity of the linear approximation, then gives an abbreviated randomization argument involving the eligible-pool control. The marked sentence ends on p. 14 with “interpreted causally”.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-c.png" }, { "alt": "Plot of the outcome of a school against the composition C of its seed group. A dashed grey line at 0.3 stands for every control school. A blue horizontal line at 0.2 carries five circles at C = 0, 0.05, 0.1, 0.15 and 0.2, the largest at 0.1: treated schools with q = 1/10. An orange horizontal line at 0 carries squares at C from 0 to 0.6, the largest at 0.3: treated schools with q = 3/10. Black triangles mark the mean composition, q. Text gives the coefficients from the expected normal equations: minus 16/31 on treatment times C in the printed regression and 0 on treatment times (C minus q) put in its place; the effect of C in every school is 0.", "caption": "The constructed example in P037-E001; it contains no data of the study. Treated schools whose seed-eligible group has composition $q=1/10$ have outcome $1/5$ at every composition $C$ of the seed group, treated schools with $q=3/10$ have outcome $0$, and every control school has outcome $3/10$. The area of a marker is the probability of that value of $C$. The composition of the seed group has no effect on any school, but treatment effects differ with $q$, violating the sufficient common-response restriction. The expected normal equations give $-16/31$ on $DC$, and $0$ when $D(C-q)$ replaces $DC$. This illustrates insufficiency of the design alone, not a refutation of the qualified causal-response model.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-counterexample.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T1", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T1" }, { "id": "P037-T2", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T2" }, { "id": "P037-T14", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T14" } ], "headline": "The composition coefficient is identified under the supplement’s qualified causal-response model; randomization and the eligible-pool control alone do not identify it.", "id": "P037-E001", "location": "Methods and Materials, p. 568 (PDF p. 3), paragraph beginning “The randomization of schools and seeds”, last two sentences; SI Appendix, p. 13, equation (2) and the sentences after it, repeated for equations (4) and (6) on pp. 15–16 and in the note to Table S7 (p. 42)", "pdf_page": 3, "printed_page": "568", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P037-T1", "P037-T14", "P037-T2" ], "text": { "reason": "I did not establish an identification error under the supplement's qualified causal-response model. The counterexample establishes that randomization and an eligible-pool main effect alone are insufficient. It does not establish failure within the qualified model or show that the identifying restriction fails in the actual schools. The exact conditioning and residual restriction remain insufficiently explicit.\n\n\\emph{The printed qualification.} The main-text randomization argument is read together with the supplement's interaction model and the validity qualification attached to it:\n\nMain text, p.~568: Although there was heterogeneity across schools in the proportion of seed-eligible students who were social referents, any differences between the proportion of treated seeds who were referents and the proportion of seed-eligibles who were referents are attributable to randomization. As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group on school-wide conflict and other outcomes.\n\nSI p.~13:\n\\[\nY_{ij}=\\alpha_j+\\tau_0D_{ij}+\\beta C_{ij}D_{ij}+\\gamma C^E_{ij}+u_{ij}. \\qquad (2)\n\\]\nHere, assuming that the linear approximation for causal effect heterogeneity is valid, $\\tau_0$ is the average causal effect of treatment when the composition of the seed group is 0\\% referents and $\\beta$ represents the slope of the average causal effect with respect to group composition in percentage points (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\beta c$.) As $\\mathrm{E}\\,[C_{ij}]\\propto C^E_{ij}$, controlling for $C^E_{ij}$ ensures that any random deviations in $C_{ij}$ from $\\mathrm{E}\\,[C_{ij}]$ may be interpreted causally.\n\n\n\n\n\nThe qualification is repeated for equations (4) and (6) and in the note to Table S7. Read with the displayed equation, it more naturally proposes a conditional average causal response $\\tau_0+\\beta c$, with eligible composition in a baseline shared by treatment and control and an appropriate residual mean or orthogonality condition. This need not mean identical realized treatment effects in every school. The repeated descriptions of average effects also support a weaker marginal-curve reading, and the conditioning set and residual condition are not explicit. On balance I prefer the qualified conditional-model reading. My comment on this entry states that the linear model was meant to express a common baseline treatment effect while preserving efficiency with few schools. The counterexample below satisfies marginal average linearity but violates the stronger identifying response restriction. It shows what does not follow from randomization alone, rather than an error within that model. Neither the example nor the observed specification differences establishes whether the restriction holds in these schools.\n\n\\emph{What the design supplies.} Let $D$ indicate that a school is treated, let $C$ be the proportion of seeds who are social referents, and let $q=C^E$ be that proportion in the seed-eligible group. Let $\\mu$ be the mean of $C$ under the stratified seed draw, given the eligible group. The supplement writes $\\mathrm{E}[C_{ij}]\\propto C^E_{ij}$, and $\\mu=q$ when all strata are sampled at the same rate. The interaction decomposes as\n\\[\nDC=D\\mu+D(C-\\mu).\n\\]\nThe second component is a mean-zero seed innovation, conditional on school assignment. The first is randomized through school assignment and carries treatment variation associated with fixed eligible pools. A main effect for $q$ does not remove $D\\mu$. Additional moderation of the treatment intercept can therefore enter the printed composition coefficient. Under a common treatment contrast, however, both components have coefficient $\\beta$; the decomposition alone establishes no bias under that sufficient model.\n\nThe authors' code fits the displayed terms. Its syntax does not establish the causal meaning or empirical adequacy of the response approximation.\n\nIn \\texttt{1. Sanitize and generate perms, probs.R}, lines 141--146 compute $C$ and $q$ separately.\n\nIn \\texttt{4. School results-with-correction.R}, lines 203--209 regress on $DC$ with $q$ as a main effect, without $Dq$ or centering $C$.\n\n\n\nThe mathematical evidence is an exact design-alone counterexample in rational arithmetic and conditional identification arguments. I also examined the authors' scripts and ran them on the public release of the data (ICPSR 37070, version 2). In my calculations, compositions are measured as fractions; a slope per percentage point is one hundredth of the slope reported.\n\n\\emph{The example.} A block has four schools whose seed-eligible groups have compositions $q=(1/10,1/10,3/10,3/10)$. Each school has 40 seed-eligible students in four strata of ten. Each stratum contains one social referent in the first two schools and three in the last two. Two of the four schools are treated, each pair with probability $1/6$. In a treated school five students are drawn from each stratum, which gives 20 seeds. The potential outcomes of school $i$ are\n\\[\nY_i(0)=\\frac{3}{10},\\qquad Y_i(1,c)=\\frac{3}{10}-q_i \\quad\\text{for every composition } c \\text{ of the seed group.}\n\\]\nThe composition of the seed group has no effect on any school. The average effect of treatment is $-1/5$ at every $c$ feasible in all four schools, and it is linear in $c$ with slope $0$. The design-mean relation holds: $\\mathrm{E}[C_i\\mid D_i=1]=q_i$, and $\\operatorname{Var}(C_i\\mid D_i=1)=q_i(1-q_i)/36$. The marginal average causal curve is linear, but the sufficient conditional-response restriction does not hold: at the common feasible composition $c=0$, the treatment effects are $-1/10$ and $-3/10$ according to eligible-pool type. The exact check enumerates all 454 collapsed states of school assignment and seed composition.\n\nConsider the regression of $Y$ on block effects, $D$, $q$ and $DC$. After the block means are removed, the expected cross-products of the columns $(D,q,DC)$ per block, and their expected cross-products with $Y$, are\n\\[\nG=\\begin{pmatrix}1&0&1/5\\\\ 0&1/25&1/50\\\\ 1/5&1/50&151/2400\\end{pmatrix},\\qquad\nh=\\begin{pmatrix}-1/5\\\\ -1/50\\\\ -17/300\\end{pmatrix}.\n\\]\nSolving $G\\,(\\tau_0,\\gamma,\\beta)^{\\top}=h$, I obtain\n\\[\n(\\tau_0,\\gamma,\\beta)=\\Bigl(-\\frac{3}{31},\\,-\\frac{15}{62},\\,-\\frac{16}{31}\\Bigr).\n\\]\nThe value $-16/31$ is also the limit in probability of the least-squares coefficient on $DC$ when independent copies of the block are pooled. It is not claimed to be the expectation of that coefficient in a finite sample. With $\\tau_0=-1/5$ and treatment-intercept deviations $\\eta_i=1/5-q_i$, the projection identity below has numerator $-1/150$ and denominator $31/2400$. Their ratio is $-16/31$. The nonzero coefficient is the contribution of baseline treatment moderation, which the qualified identifying model excludes or restricts. It cannot refute identification under a common treatment contrast or the weaker zero-numerator condition.\n\n\\begin{audittable}{llrr}{Coefficient on the composition term in the example, from the expected normal equations with block effects. In the second and third outcomes the composition of the seed group has slope $1/5$.}\nOutcome & Regressors & Coefficient & True slope \\\\\n\\hline\n$3/10-Dq$ & $D$, $q$, $DC$ (as printed) & $-16/31$ & $0$ \\\\\n$3/10-Dq$ & $D$, $q$, $DC$, $Dq$ & $0$ & $0$ \\\\\n$3/10-Dq$ & $D$, $q$, $D(C-q)$ & $0$ & $0$ \\\\\n$3/10-Dq+DC/5$ & $D$, $q$, $DC$ (as printed) & $-49/155$ & $1/5$ \\\\\n$3/10-Dq+DC/5$ & $D$, $q$, $D(C-q)$ & $1/5$ & $1/5$ \\\\\n$3/10-D/5-q/2+DC/5$ & $D$, $q$, $DC$ (as printed) & $1/5$ & $1/5$ \\\\\n\\end{audittable}\n\nThe last row conforms to the displayed model, and the printed regression recovers its slope. An exactly linear untreated mean is unnecessary for the following expected-equations result: arbitrary fixed untreated school outcomes are allowed.\n\n\\emph{A sufficient identifying moment.} Fix the school population, eligible pools, analysis membership and pretreatment controls. Let $M$ remove the mean within each included block, and write $d=MD$, $z=M(DC)$ and $V=M[q,X]$ for a full-rank basis of the fixed nuisance controls. Suppose school-treatment probabilities are equal, $M\\mu$ lies in the span of $V$, and residual variation is positive. Let\n\\[\nR=z-\\rho d-Vg\n\\]\nbe the residual from projecting $z$ on $d$ and $V$, with coefficients chosen from design-expected cross-products. These are not assignment-specific fitted projection coefficients. The expected normal equations give $\\mathbb{E}[R]=0$, $\\mathbb{E}[R^{\\top}d]=0$, $\\mathbb{E}[R^{\\top}V]=0$ and $\\mathbb{E}[R^{\\top}z]=\\mathbb{E}[R^{\\top}R]>0$.\n\nWithin the common-slope class, write\n\\[\nY_i(0)=a_i,\\qquad Y_i(1,c)-Y_i(0)=\\tau_0+\\eta_i+\\beta c,\n\\]\nwhere $a_i$ and $\\eta_i$ are arbitrary fixed numbers. Since $\\mathbb{E}[R]=0$, arbitrary untreated levels contribute nothing to the expected residual cross-product. Eliminating the other regressors from the expected normal equations gives\n\\[\n\\beta^*-\\beta=\n\\frac{\\mathbb{E}[\\sum_i R_iD_i\\eta_i]}{\\mathbb{E}[\\sum_i R_i^2]}.\n\\]\nZero numerator identifies $\\beta$ under these conditions; a common school treatment intercept is a stronger sufficient case. School treatment effects need not be constant for the numerator to vanish. This is identification by expected equations and consistency under independent repetition of bounded blocks with limiting full-rank moments, not finite-sample design unbiasedness. Fixed membership and controls are assumptions of this argument, not conclusions about the actual analysis sample.\n\nA separate sufficient route is a full conditional causal-mean model with the corresponding residual mean condition. If the conditional expectation of $Y$ given the design matrix is that matrix times its coefficient vector, the expected normal equations identify those coefficients; with realized full rank, least squares is conditionally unbiased under that model. This route does not require the controls to span $\\mu$. It concerns a conditional average response, allowing residual heterogeneity, and is distinct from imposing a common realized treatment intercept in every school.\n\n\\emph{Optional alternatives.} To allow arbitrary fixed treatment intercepts, suppose that $Y_i(0)=a_i$ and $Y_i(1,c)=a_i+t_i+bc$, with common within-school slope $b$. Keep analysis membership fixed and nuisance regressors independent of the seed draw. Put $U_i=D_i(C_i-\\mu_i)$. Then\n\\[\nY_i=a_i+D_i(t_i+b\\mu_i)+bU_i.\n\\]\nConditional on school assignment, $U_i$ has mean zero. After fixed block demeaning it is orthogonal in expectation to the other terms and to seed-independent nuisance regressors, including block effects, $D$, $q$, $D\\mu$ and fixed covariates. Its expected normal equation identifies $b$ when residual innovation variation is positive. Thus replacing $DC$ by $D(C-\\mu)$, or retaining $DC$ and adding $D\\mu$, provides an alternative expected-equations identification strategy under these assumptions. Independent bounded-block repetition gives consistency, not finite-sample design unbiasedness. These alternatives are optional when the original causal-response model holds. They need not agree in finite samples when fitted with different nuisance spaces.\n\nIn the authors' design the seeds are drawn within strata, with a fixed number from each. If school $i$ has $N_{ih}$ seed-eligible students, $K_{ih}$ social referents and $n_{ih}$ seeds in stratum $h$, and $M_i=\\sum_h n_{ih}$, then\n\\[\n\\mu_i=\\frac{1}{M_i}\\sum_h n_{ih}\\frac{K_{ih}}{N_{ih}},\n\\]\n\\[\n\\operatorname{Var}(C_i\\mid D_i=1)=\\frac{1}{M_i^{2}}\\sum_h n_{ih}\\frac{K_{ih}}{N_{ih}}\\Bigl(1-\\frac{K_{ih}}{N_{ih}}\\Bigr)\\frac{N_{ih}-n_{ih}}{N_{ih}-1}.\n\\]\nA stratum with one student contributes nothing to the variance. The eligible-pool composition $q$ can replace $\\mu$ in these alternatives only when the two agree; equal sampling fractions suffice. With heterogeneous composition slopes, the estimand and its weights must be stated. These common-slope arguments do not establish a plain average of school-specific slopes. On the public data $\\mu_i=q_i$ in 55 of the 56 schools. In one control school the two differ by $0.0018$, because its strata are sampled at different rates.\n\n\\emph{Specification sensitivity on the public data.} I ran the authors' scripts for the school-level regressions on the public data, with an adapter that decodes and renames its variables as the scripts expect, and I varied only the composition specification. The covariates and the rule for including schools are the authors'. The regression for disciplinary reports has 49 schools.\n\n\\begin{audittable}{lrrr}{Sensitivity of the composition coefficient for disciplinary reports of peer conflict, on the public data (ICPSR 37070, version 2), 49 schools. The standard error is the robust (HC0) standard error of the authors' script, and the $P$ value is two-sided with a normal reference, as in the article. Table S7 prints $-0.85$ and $0.46$ for the first row.}\nRegressors & Coefficient & Standard error & $P$ \\\\\n\\hline\n$D$, $DC$, $q$ (as printed) & $-0.841$ & $0.463$ & $0.069$ \\\\\n$D$, $DC$, $Dq$, $q$ & $-0.479$ & $0.440$ & $0.277$ \\\\\n$D$, $D(C-\\mu)$, $q$ & $-0.393$ & $0.476$ & $0.409$ \\\\\n\\end{audittable}\n\nWith $D\\mu$ in place of $Dq$ the second row is unchanged, since $\\mu=q$ in every treated school. On the public data, the second row is $-0.0048$ reports per student per percentage point. As an illustration on the public data, using the authors' construction of the curve of Fig.~2D gives $0.095$ reports per student at a composition of 20\\% from the first row, a reduction of 57.3\\% from the fitted control mean, and $0.125$ and 43.7\\% from the second row with $D\\mu$. The article prints $0.08$ and 60\\% (see \\fid{P037-N010}).\n\nThese values have three limits. First, they are values on the public data. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. On the public data five of the eight coefficients of Table S7 and seven of its eight robust standard errors agree with the printed values to two decimals; the coefficient $-0.841$ rounds to $-0.84$, not to the printed $-0.85$. Second, the three rows differ in what they assume. The differences between them show that the estimate depends on the specification. They do not show which specification is correct, and they do not show that the printed estimate is biased. If the slope differs between schools, the second and third rows generally target a weighted average of the school slopes, and not their plain average. Third, these $P$ values use the stated normal reference and are reported as part of that sensitivity calculation. These sensitivity calculations do not include the bootstrap. The comparison does not establish an incorrect published $P$ value or select a uniquely correct replacement coefficient; test calibration is considered separately in \\fid{P037-N006}.\n\n\\emph{What follows.} The composition-specific causal interpretations of equations (2), (4) and (6), Table S7, Fig.~2, and the projection of $0.08$ disciplinary events per student and a $60\\%$ reduction at $20\\%$ social referents remain conditional on the response approximation or another justified identification strategy. The example and specification differences do not establish failure within that model or empirical misspecification. The same condition reaches the Abstract's ``stronger'' statement and the Discussion's recommendation insofar as they rely on the school-level composition result; separate peer-exposure evidence also bears on the role of social referents.\n\nThe between-school randomization and separate overall-treatment regressions (script 4, lines 158 and 162) do not use this composition-identification step. The released script also uses predictions from the interaction regression for adjusted overall summaries (lines 253--279 and 286). Even if the composition interpretation failed, it would not follow that an averaged overall treatment contrast was wrong. The peer-exposure analyses use a different identification step and are not invalidated here.\n\nA clarification could make the existing qualification explicit: under the valid causal-response approximation described in the SI Appendix, the interaction coefficient estimates the response to seed composition. Randomization and the eligible-pool control alone do not guarantee that approximation; residual treatment heterogeneity must satisfy the corresponding identifying mean or moment condition. On the preferred reading this states the scope of an existing substantive assumption. Under the competing marginal-only reading it would be an additional restriction. These results require no change to the fitted specification, published numbers or policy conclusion. I did not establish an identification error under the qualified model, and its empirical adequacy remains unresolved." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E001", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Six lines of text followed by a displayed definition by cases. In the text, a red box marks the symbol z sub i in the sentence The crossproduct of z sub i and theta sub i, z prime theta sub i, is equal to the number of student i's peers that are assigned to be seeds. The display defines f(z, theta sub i) by five cases: d C (Control exposure), d S (School exposure), d NI (Non social referent seed exposure), d I (Social referent seed exposure) and d D (Direct treatment condition). A red box marks the condition of the third case, I((z circle gamma) prime theta sub i = 0) s sub i = 1.", "caption": "SI Appendix, p. 17 (PDF p. 17): the definitions that precede the exposure mapping, and the mapping. The marked condition for $d_{NI}$ has neither the factor $(1-z_i)$ nor a requirement that some peer is a seed. The marked $z_i$ in the text above stands where the vector $\\mathbf{z}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E002.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T7", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T7" }, { "id": "P037-T8", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T8" } ], "headline": "The condition printed for the nonreferent exposure $d_{NI}$ lacks the factors $(1-z_i)$ and $\\mathbf{I}(\\mathbf{z}'\\theta_i>0)$, so the five cases overlap; with both factors the cases partition the assignments, and the released code computes the corrected condition.", "id": "P037-E002", "location": "SI Appendix, p. 17 (PDF p. 17 of the SI Appendix), display after “The exposure mapping is:”, third case ($d_{NI}$), and the sentence beginning “The crossproduct of” above it; compare the main text, p. 568 (PDF p. 3)", "pdf_page": 17, "printed_page": "17", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T7", "P037-T8" ], "text": { "as_printed": "``The crossproduct of \\hl{$z_i$} and $\\theta_i$, $\\mathbf{z}'\\theta_i$, is equal to the number of student $i$'s peers that are assigned to be seeds.''\n\nThe exposure mapping is:\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_C\\ (\\text{Control exposure}): & (1-s_i)=1\\\\\nd_S\\ (\\text{School exposure}): & (1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i=0)s_i=1\\\\\nd_{NI}\\ (\\text{Non social referent seed exposure}): & \\hl{\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0)s_i=1}\\\\\nd_I\\ (\\text{Social referent seed exposure}): & (1-z_i)\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i>0)s_i=1\\\\\nd_D\\ (\\text{Direct treatment condition}): & z_i=1\n\\end{cases}\n\\]", "corrected": "``The crossproduct of \\hl{$\\mathbf{z}$} and $\\theta_i$, $\\mathbf{z}'\\theta_i$, is equal to the number of student $i$'s peers that are assigned to be seeds.''\n\nLet $s_i$ indicate that the school of student $i$ is treated, let $z_i$ indicate that student $i$ is assigned to be a seed, let $\\theta_i$ be the row of student $i$ in the network adjacency matrix, and let $\\gamma$ be the vector of social referent indicators, as on p.~17. The exposure mapping is\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_C\\ (\\text{Control exposure}): & (1-s_i)=1\\\\\nd_S\\ (\\text{School exposure}): & (1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i=0)s_i=1\\\\\nd_{NI}\\ (\\text{Non social referent seed exposure}): & \\hl{(1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i>0)}\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0)s_i=1\\\\\nd_I\\ (\\text{Social referent seed exposure}): & (1-z_i)\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i>0)s_i=1\\\\\nd_D\\ (\\text{Direct treatment condition}): & z_i=1\n\\end{cases}\n\\]\nFor every assignment that the design can generate, exactly one of the five conditions holds.", "evidence": "I base this finding on two exact counterexamples, a derivation, a reading of the released code and a recalculation on the public data. Let $t_i=\\mathbf{z}'\\theta_i$ denote the number of seed peers of student $i$, and let $r_i=(\\mathbf{z}\\circ\\gamma)'\\theta_i$ denote the number of seed peers who are social referents. For $(s_i,z_i,t_i,r_i)=(1,0,0,0)$ the printed conditions for $d_S$ and $d_{NI}$ both hold, and for $(1,1,0,0)$ the printed conditions for $d_{NI}$ and $d_D$ both hold. Both configurations can occur under the design, since a treated school can have all of its other seeds outside the peers of student $i$.\n\nWith the corrected condition, exactly one case holds for every assignment that the design can generate. A control school has no seeds. Then $s_i=0$ and $z_i=0$, and only $d_C$ holds. In a treated school, only $d_D$ holds when $z_i=1$. If $z_i=0$, then only $d_S$ holds when $t_i=0$, and when $t_i>0$ the case is $d_{NI}$ if $r_i=0$ and $d_I$ if $r_i>0$. Since $\\gamma$ and $\\theta_i$ have entries in $\\{0,1\\}$, I have $0\\le r_i\\le t_i$, and the cases are exhaustive and disjoint. Enumerating the 133 configurations $(s_i,z_i,t_i,r_i)$ with at most ten peers, I find exactly one corrected case in each.\n\nThe released code computes the corrected condition, with $t_i$ and $r_i$ counted in a within-school graph that includes nominations in either direction (see \\fid{P037-N032}). Let $b_i=\\mathbf{I}(t_i>0)$ and $q_i=\\mathbf{I}(r_i>0)$. The exposure script sets the indicator of exposure to some seed to $(1-z_i)b_i$ and the indicator of exposure to a social referent seed to $(1-z_i)q_i$ (lines 196 and 199 for the simulated assignments, lines 217 and 222 for the observed one), and the network script takes their difference as the nonreferent exposure (lines 203 and 233). Since $q_i\\le b_i$, I have $b_i-q_i=\\mathbf{I}(t_i>0)\\mathbf{I}(r_i=0)$, and the difference is\n\\[\n(1-z_i)(b_i-q_i)=(1-z_i)\\,\\mathbf{I}(\\mathbf{z}'\\theta_i>0)\\,\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0),\n\\]\nwhich is zero in a control school, where no student is a seed. The code estimates the probability of nonreferent exposure by subtracting the two simulated frequencies (network script, line 188), giving the frequency of the corrected condition. Using the authors' scripts with an adapter for the public data release (ICPSR 37070 V2), I find that the code's indicator equals the corrected condition for all 24,191 retained students under the observed assignment and for all 241,910,000 pairs of a student and an assignment in 10,000 simulated assignments, for both definitions of a social referent (indegree and betweenness), and the two probability estimates differ by less than $10^{-15}$. On the public data the printed condition, read literally, would add 5,754 students with school exposure and 665 seeds to the nonreferent category under the observed assignment (indegree definition).", "problem": "As printed, the condition for $d_{NI}$ says only that the school is treated and that no social referent peer of student $i$ is a seed. It does not exclude a student who is a seed, and it does not require that at least one peer is a seed. A student in a treated school who is not a seed and has no seed peers satisfies the conditions for both $d_S$ and $d_{NI}$, and a seed none of whose seed peers is a social referent satisfies the conditions for both $d_{NI}$ and $d_D$. The five cases do not define a function. The main text (p.~568) states the category in words: students in treated schools ``for whom at least one peer is a seed but no seed is a social referent''. In the sentence above the display, the product $\\mathbf{z}'\\theta_i$ is called the crossproduct of $z_i$ and $\\theta_i$, where the vector $\\mathbf{z}$ is meant.", "what_survives": "The main text states the intended categories in words. The released code has both missing factors in the observed exposure, in the simulated exposures and in the estimated probability of nonreferent exposure, and correcting the display leaves these quantities unchanged. No reported estimate is corrected by this finding. I have not independently reproduced every published estimate, and cannot establish equality with the authors' original data: their own data file is not public, and ICPSR reports changes to some released variables." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E002", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Nine lines of text from page 18 of the SI Appendix. The first line holds the definition pi sub i of d sub k equals the sum over z in Omega of I(f(z, theta sub i) = d sub k) times Pr(Z = z). The fifth line holds the formula pi-hat sub i of d sub k equals I[ one over J times the sum from l = 1 to J of f(z superscript J, theta sub i) = d sub k ]. One red box marks the opening I[ of the indicator, and a second red box marks z superscript J inside the sum.", "caption": "SI Appendix, p. 18 (PDF p. 18), second paragraph: the exact definition of $\\pi_i(d_k)$ in the first line, and the simulated version in the fifth line. The first mark is on the indicator, which stands outside the average; the second is on $\\mathbf{z}^J$, the summand’s assignment, which does not depend on the index $\\ell$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E003.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T9", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T9" }, { "id": "P037-T30", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T30" } ], "headline": "In the formula for the simulated exposure probability, the indicator belongs inside the average and the summand should use the $\\ell$th simulated assignment, not the $J$th; the released code averages the draw-specific indicators.", "id": "P037-E003", "location": "SI Appendix, p. 18 (PDF p. 18 of the SI Appendix), second paragraph, formula after “according to the experimental design:”; the same formula on p. 14 (PDF p. 14)", "pdf_page": 18, "printed_page": "18", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T30", "P037-T9" ], "text": { "as_printed": "``We can achieve an arbitrarily precise estimate of $\\pi_i$ via simulated random assignments of $\\mathbf{Z}$ according to the experimental design:\n\\[\n\\hat\\pi_i(d_k)=\\hl{\\mathbf{I}[}\\frac{1}{J}\\sum_{\\ell=1}^{J}f(\\hl{\\mathbf{z}^{J}},\\theta_i)=d_k],\n\\]\nwhere $J$ is the number of simulated random assignments, and \\hl{$z^J$} denotes the \\hl{$J$th} simulated random assignment vector.''", "corrected": "On pp.~14 and 18:\n\\[\n\\hat\\pi_i(d_k)=\\frac{1}{J}\\sum_{\\ell=1}^{J}\\hl{\\mathbf{I}[}f(\\hl{\\mathbf{z}^{\\ell}},\\theta_i)=d_k],\n\\]\nwhere $J$ is the number of simulated random assignments, \\hl{$\\mathbf{z}^{\\ell}$} denotes the \\hl{$\\ell$th} simulated random assignment vector, the assignments are drawn independently according to the experimental design, and $f$ is the exposure mapping as corrected in \\fid{P037-E002}. Then $\\hat\\pi_i(d_k)$ is unbiased for $\\pi_i(d_k)$, with variance $\\pi_i(d_k)(1-\\pi_i(d_k))/J$.", "evidence": "The evidence is a check of pp.~14 and 18, a numerical counterexample, a derivation, a reading of the released code and a recalculation on the public data. Suppose that two simulated assignments induce exposures coded 0 and 1, and that $d_k$ is the exposure coded 1. The proportion of assignments with exposure $d_k$ is $1/2$. The printed formula uses the last assignment in both terms and gives 1. With $\\mathbf{z}^{\\ell}$ in place of $\\mathbf{z}^{J}$ and the indicator left outside the average, I obtain $\\mathbf{I}[(0+1)/2=1]=0$.\n\nFor the corrected formula, let $\\mathbf{z}^{1},\\ldots,\\mathbf{z}^{J}$ be independent draws from the design, and set $X_\\ell=\\mathbf{I}[f(\\mathbf{z}^{\\ell},\\theta_i)=d_k]$. Each $X_\\ell$ has mean $\\pi_i(d_k)$ and variance $\\pi_i(d_k)(1-\\pi_i(d_k))$, which implies that $\\hat\\pi_i(d_k)$ has mean $\\pi_i(d_k)$ and variance $\\pi_i(d_k)(1-\\pi_i(d_k))/J$.\n\nThe released code performs this averaging. Its probability script forms a 0/1 indicator of each exposure for each simulated assignment (lines 60 and 170--206) and averages each indicator over the assignments (lines 208--214). A reconstruction with an adapter on the public data (ICPSR 37070 V2) executed these expressions for all 10,000 assignments and 24,191 retained students. Every indicator is 0 or 1, the five basic exposures partition every assignment, and each average equals the number of assignments with the exposure divided by 10,000. In this reconstruction, the simulated probabilities of control exposure lie between 0.4881 and 0.5125 and are fractional for every student, where the indicator of the last assignment would be 0 or 1.", "problem": "As printed, the right-hand side is the indicator that an average of exposure labels equals $d_k$. The labels are categories, and an average of categories is not defined. A single indicator takes only the values 0 and 1, so it is not the simulated frequency and cannot become arbitrarily precise for a probability strictly between 0 and 1 as $J$ grows. The summand carries the superscript $J$ and does not depend on the index $\\ell$: every term uses the last simulated assignment. The preceding definition states the quantity that is meant, the probability that an assignment induces exposure $d_k$. The same formula is printed on p.~14.", "what_survives": "The exact definition $\\pi_i(d_k)=\\sum_{\\mathbf{z}\\in\\Omega}\\mathbf{I}(f(\\mathbf{z},\\theta_i)=d_k)\\Pr(\\mathbf{Z}=\\mathbf{z})$, printed before the formula, is correct and fixes the meaning. The released code averages the indicators of the separate assignments, and correcting the two displays changes no output of that code. No reported estimate is shown to change. The authors' original data file and saved simulations are unavailable, ICPSR reports changes to released variables, and the reconstruction uses the current R runtime, so I cannot recover the historical probabilities exactly. This check does not cover other simulation choices or outcome analyses (see \\fid{P037-N031} for the school draws)." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E003", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Four lines of text from page 19 of the SI Appendix, beginning Consistency and asymptotic normality directly follow from the law of large numbers and a Lindeberg-type central limit theorem given a growing sequence of independent schools of finite size, as holds by construction given our between-school randomization. One red box marks the words independent schools, and a second red box marks the words as holds by construction.", "caption": "SI Appendix, p. 19 (PDF p. 19). The marked words state that the schools are independent and that this holds by construction; the design assigns two of the four schools in each block to treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E004.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T10", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T10" }, { "id": "P037-T11", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T11" } ], "headline": "The schools are not independent by construction, because schools in the same randomization block of four are dependent; the independent units are the blocks, and the argument for consistency and asymptotic normality should be made for block totals under stated conditions.", "id": "P037-E004", "location": "SI Appendix, p. 19 (PDF p. 19 of the SI Appendix), paragraph beginning “Consistency and asymptotic normality”; the design is described on p. 5 and in the caption of Fig. S1 (PDF p. 24)", "pdf_page": 19, "printed_page": "19", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [ { "id": "P045-N018", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N018", "work_id": "P045" } ], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T10", "P037-T11" ], "text": { "as_printed": "``Consistency and asymptotic normality directly follow from the law of large numbers and a Lindeberg-type central limit theorem given a growing sequence of \\hl{independent schools} of finite size, \\hl{as holds by construction} given our between-school randomization.''", "corrected": "``Consistency and asymptotic normality follow from the law of large numbers and a Lindeberg-type central limit theorem \\hl{for the totals of the four-school randomization blocks}, given a growing sequence of \\hl{independently randomized blocks} of finite size \\hl{and the conditions below. Schools within a block are dependent.}''\n\nA sufficient set of conditions, for the fixed population above, an exposure $d_k$ and in the notation of the evidence, is the following, with positive constants $m$, $c$, $\\eta$, $K$ and $v$ that do not depend on the number of blocks $B$. For a contrast, impose the exposure-specific conditions for both exposures.\n\\begin{enumerate}\n\\item The randomizations of different blocks are independent, including the draws of seeds within their schools. Each student's exposure depends only on the randomization of that student's block.\n\\item Each block has at most $m$ retained students, and $N_B\\ge cB$.\n\\item For every retained student, $\\pi_i(d_k)\\ge\\eta>0$, $|y_i(d_k)|\\le K$ and $|g_i|\\le K$.\n\\item The variance $s_B^2=\\sum_{b=1}^{B}\\operatorname{Var}(X_b)$ satisfies $s_B^2\\ge vB$ with $v>0$; for a contrast, the same holds for the block totals of the contrast.\n\\item The regression is linear in covariates $x_i$ of fixed dimension and bounded norm and is fitted by inverse-probability-weighted least squares on the retained students with exposure $d_k$. The sums over all retained students satisfy $N_B^{-1}\\sum_i x_ix_i^\\top\\to Q$ for a positive definite matrix $Q$ and $N_B^{-1}\\sum_i x_iy_i(d_k)\\to w$, and $\\beta_*=Q^{-1}w$.\n\\item The simulated probabilities satisfy $\\max_i|\\hat\\pi_i(d_k)-\\pi_i(d_k)|=o_p(B^{-1/2})$.\n\\end{enumerate}\nUnder conditions 1--4, the estimator with exact probabilities and fixed predictions is consistent and asymptotically normal as $B\\to\\infty$; conditions 5 and 6 extend the conclusion to the fitted regression and to the simulated probabilities. The conditions are sufficient and are not claimed to be necessary, and they are not checked here for a sequence that contains the 56 schools of the study. Condition 5 gives a sufficient linear case of the regression displayed on p.~19; the regression of the released code is the subject of \\fid{P037-N033}. Regrouping the proof does not change a variance calculation under constant effects that simulates the blocked randomization.\n\nTo make the released variance sampler follow the design, line 58 of \\texttt{1. Sanitize and generate perms, probs.R} should read:\n\n\\texttt{sch.perms <- genperms(schtreat, \\hl{blockvar = schrb}, clustvar = schid, maxiter = niter)}\n\nThe exposure draws must then be regenerated and passed through scripts 1--3 before recomputing the network standard errors, intervals and $p$-values. This code repair is separate from regrouping the asymptotic proof.", "evidence": "The evidence is an exact enumeration, a derivation and a test of the released school sampler on the public data. Let $T_A,T_B,T_C,T_D$ denote the treatment indicators of the four schools of a block. The six allocations of two treated schools are equally likely; three of them treat school $A$, three treat school $B$, and one treats both. I have $\\Pr\\{T_A=1\\}=\\Pr\\{T_B=1\\}=1/2$ and $\\Pr\\{T_A=1,T_B=1\\}=1/6$, where independence would require $1/4$, and $\\operatorname{Cov}(T_A,T_B)=-1/12$. The same covariance holds for every pair of schools in the block and for the control-exposure indicators $1-T_A$ and $1-T_B$. The dependence does not rest on equal probabilities for the six allocations: the sum $T_A+T_B+T_C+T_D=2$ has variance zero, while the sum of four independent indicators of mean $1/2$ has variance $1$. The released script for the experimental randomization permutes two labels of one arm and two labels of the other within each block of four (\\texttt{7. Blocking, randomization, and summary statistics.R}, lines 83--90), and on the public data each of the 14 blocks has four schools, two of them treated.\n\nThe simulation script, \\texttt{1. Sanitize and generate perms, probs.R}, line 58, calls \\texttt{genperms} without \\texttt{blockvar=schrb}, drawing 28 of 56 schools without the original blocks; exact counting gives covariance $-1/220$ for every distinct school pair. In a 10,000-draw test on the public data (ICPSR 37070 V2), every draw from this call violated at least one original block, while adding \\texttt{blockvar=schrb} gave no violations. Script 1 saves the first 1,000 exposure draws (lines 232--242), script 2 loads them (line 16), and script 3 uses them for the network standard errors and Wald bands (lines 237--264 and 325--328). The public data reproduce the published 24,191 pre-analysis records and the 56-school, 14-block allocation, but ICPSR warns that variables were changed and the authors' original analysis files are unavailable.\n\nFor the corrected statement, at each number of blocks $B$, fix the schools, within-school networks, covariates, potential outcomes and retained student sets. Fix an exposure $d_k$, and write $I_i=\\mathbf{1}\\{D_i=d_k\\}$ and $\\pi_i=\\pi_i(d_k)$. Let $S_b$ denote the retained students of block $b$, and let $N_B=\\sum_{b=1}^B|S_b|$. For fixed predictions $g_i$, define\n\\[\nh_i=g_i+\\frac{I_i}{\\pi_i}\\bigl(y_i(d_k)-g_i\\bigr),\\qquad X_b=\\sum_{i\\in S_b}h_i,\\qquad \\widetilde\\mu_B=\\frac{1}{N_B}\\sum_{b=1}^{B}X_b .\n\\]\nSince $\\mathbb{E}[I_i]=\\pi_i$, I have $\\mathbb{E}[h_i]=y_i(d_k)$, and $\\widetilde\\mu_B$ is unbiased for $\\mu_B=N_B^{-1}\\sum_i y_i(d_k)$. Each block total $X_b$ depends only on the randomization of block $b$, and condition 1 of the corrected statement implies that the $X_b$ are independent. Under conditions 2 and 3, I have $|h_i|\\le H$ with $H=K+2K/\\eta$, and $|X_b|\\le mH$. Using these bounds and the independence of the block totals, I obtain\n\\[\n\\operatorname{Var}(\\widetilde\\mu_B)=\\frac{1}{N_B^2}\\sum_{b=1}^{B}\\operatorname{Var}(X_b)\\le\\frac{m^2H^2}{c^2B}.\n\\]\nThis bound and Chebyshev's inequality imply consistency. Under condition 4, for every $\\epsilon>0$ I have $|X_b-\\mathbb{E}[X_b]|\\le 2mH<\\epsilon s_B$ once $B$ is large. Then every term of the Lindeberg sum vanishes, and $N_B(\\widetilde\\mu_B-\\mu_B)/s_B$ converges in distribution to a standard normal variable. For a contrast between two exposures, $h_i$ is replaced by the difference of its values for the two exposures.\n\nWith fitted coefficients $\\widehat\\beta$, and with $g_i=x_i^\\top\\beta_*$ in $\\widetilde\\mu_B$, the regression estimator with exact probabilities differs from $\\widetilde\\mu_B$ by\n\\[\n\\Bigl[\\frac{1}{N_B}\\sum_i\\Bigl(1-\\frac{I_i}{\\pi_i}\\Bigr)x_i\\Bigr]^\\top\\bigl(\\widehat\\beta-\\beta_*\\bigr).\n\\]\nUnder conditions 1--3 and 5, the bracket is $O_p(B^{-1/2})$ and $\\widehat\\beta-\\beta_*=o_p(1)$. I conclude that the difference is $o_p(B^{-1/2})$, which is negligible beside $s_B/N_B\\ge\\sqrt{v}/(m\\sqrt{B})$. Under condition 6, replacing $\\pi_i$ by the simulated $\\hat\\pi_i$ changes the estimator by $o_p(B^{-1/2})$ as well.", "problem": "The design randomizes schools within blocks of four: in each of the 14 blocks, two schools are assigned to treatment and two to control (SI Appendix, p.~5, and the caption of Fig.~S1). The treatment indicators of two schools in a block are dependent, and so are their control-exposure indicators $1-s_i$. The schools are not independent by construction, and the law of large numbers and the central limit theorem cannot be applied to them as the sentence does. The independent units are the blocks. The argument can be made for block totals, under regularity conditions that the sentence does not state.", "what_survives": "Regrouping the asymptotic proof alone leaves the exposure definitions, the estimand and the point estimates unchanged. The finding does not show that the estimator is inconsistent or that a reported standard error, interval or $p$-value is wrong. The normal approximation behind the Wald intervals and $p$-values of the social-influence analysis (SI Appendix, p.~20, and Fig.~3) needs the argument for blocks and its conditions. A variance simulation under constant effects that follows the allocation of two schools in each block of four keeps the dependence within blocks. The released school sampler instead omits the blocks; its numerical effect on the public data is measured in \\fid{P037-N031}. Regenerating the simulated probabilities may change the retained set and fitted estimates. This correction does not affect the school-climate regressions." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E004", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Table S6, Average school-level climate effects caused by anti-conflict groups, with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict, and rows School level main effect, Heteroskedasticity Robust SE, Bayesian Bootstrap SE and Wild Bootstrap SE. A red box marks the row label Bayesian Bootstrap SE, whose entries are (0.01), (0.05), (0.01) and (0.03). In the note below the table, a red box that runs over a line break marks the words applying weights to each school drawn from the standard uniform distribution.", "caption": "SI Appendix, Table S6 and its note (PDF p. 41; the page is printed sideways and is shown upright). The marks are on the row label “Bayesian Bootstrap SE” and on the words of the note that describe the weights as drawn from the standard uniform distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005.png" }, { "alt": "Table S7, Average school-level climate effects caused by anti-conflict groups, interacted with proportion of social referent seeds in the group, with the same four columns as Table S6. Under School effect with 0 percent Social Referents and under Interaction: Proportion of Referents in Treatment group there are three rows of standard errors each. Red boxes mark the two row labels Bayesian Bootstrap SE, whose entries are (0.03), (0.11), (0.02), (0.05) and (0.22), (0.80), (0.13), (0.35). In the note below the table, a red box marks the words applying weights to each school drawn from the standard uniform distribution.", "caption": "SI Appendix, Table S7 and its note (PDF p. 42; the page is printed sideways and is shown upright). The marks are on the two rows labelled “Bayesian Bootstrap SE” and on the same description of the weights in the note.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005-b.png" }, { "alt": "Dot plot with twelve rows in three groups: Table S6, school level main effect; Table S7, school effect with 0 percent social referents; Table S7, interaction with proportion of referents. Each group has the rows Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict. The horizontal axis is the ratio of the standard deviation with uniform weights to the standard deviation with Dirichlet weights, from 0.55 to 1.03. A blue vertical line at 1 is labelled Bayesian bootstrap (Dirichlet weights). The twelve orange dots lie between 0.683 and 0.730, each with a narrow orange bar extending two Monte Carlo standard errors on each side and its value printed beside it: 0.683, 0.711, 0.693, 0.720 in the first group; 0.710, 0.715, 0.712, 0.723 in the second; 0.720, 0.706, 0.716, 0.730 in the third.", "caption": "The ratio of the standard deviation of each refitted coefficient under uniform school weights to its standard deviation under Dirichlet$(1,\\ldots,1)$ weights, for the twelve entries of Tables S6 and S7, from 50,000 paired refits per regression on the public data (ICPSR 37070 V2). The vertical line at 1 is the Bayesian-bootstrap reference. Orange bars extend two Monte Carlo standard errors on each side of a dot; each standard error is at most 0.003.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005-weights.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T4", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T4" }, { "id": "P037-T5", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T5" }, { "id": "P037-T16", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T16" } ], "headline": "The twelve entries labelled “Bayesian Bootstrap SE” in Tables S6 and S7 were computed with an independent uniform weight for each school, which is not the Bayesian bootstrap; in a reconstruction on the public data, uniform-weight dispersions are 27% to 32% smaller than the corresponding dispersions under the Bayesian bootstrap. The rows should be relabelled as dispersions under uniform weights, or recomputed with Dirichlet$(1,\\ldots,1)$ weights.", "id": "P037-E005", "location": "SI Appendix, Table S6 (PDF p. 41 of the SI Appendix), note and row “Bayesian Bootstrap SE”; the same note and two rows of Table S7 (PDF p. 42)", "pdf_page": 41, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T16", "P037-T4", "P037-T5" ], "text": { "as_printed": "Notes to Tables S6 and S7: ``\\hl{Bayesian bootstrap standard errors} generated using 5,000 replicates, \\hl{applying weights to each school drawn from the standard uniform distribution}.''\n\nRow label, once in Table S6 and twice in Table S7: ``\\hl{Bayesian Bootstrap SE}''.", "corrected": "The three rows are labelled ``\\hl{Uniform-weight perturbation SD}'', their entries stay as printed, and the notes to Tables S6 and S7 read: ``\\hl{Uniform-weight perturbation standard deviations} generated using 5,000 replicates, applying \\hl{independent $U(0,1)$ weights to the schools}.'' The entries are dispersions under uniform weights, and they are not Bayesian-bootstrap standard errors.\n\nIf Bayesian-bootstrap standard errors are wanted instead, the twelve entries are recomputed as follows. For replicate $r=1,\\ldots,R$, with $R=5{,}000$ for the revised notes below, draw independent standard exponential variables $E_1^{(r)},\\ldots,E_n^{(r)}$ for the $n$ schools of the regression, set $p_s^{(r)}=E_s^{(r)}/\\sum_t E_t^{(r)}$, and compute\n\\[\n\\widehat b^{(r)}=\\arg\\min_b\\sum_{s=1}^{n}p_s^{(r)}\\bigl(y_s-x_s^\\top b\\bigr)^2,\\qquad \\widehat{\\mathrm{SE}}=\\Bigl[\\frac{1}{R-1}\\sum_{r=1}^{R}\\bigl(\\widehat b^{(r)}_k-\\bar b_k\\bigr)^2\\Bigr]^{1/2},\n\\]\nwhere $k$ is the coefficient of interest and $\\bar b_k$ is the mean of the $\\widehat b^{(r)}_k$. The weights $p^{(r)}$ then have the Dirichlet$(1,\\ldots,1)$ law, and the notes read: ``Bayesian bootstrap standard errors generated using 5,000 replicates, applying weights to \\hl{the schools with the Dirichlet$(1,\\ldots,1)$ law, obtained by dividing independent standard exponential draws by their sum}.'' The column ``Dirichlet'' of the table above gives estimates of this dispersion on the public data with $R=50{,}000$, and their conditional Monte Carlo standard errors. The use of the method for the blocked design would have to be justified.", "evidence": "The evidence is an exact counterexample, a reading of the released code and a recalculation on the public data. Let $U_1,\\ldots,U_{56}$ be independent and uniform on $(0,1)$, and set $P_i=U_i/\\sum_j U_j$. Let $E_1,\\ldots,E_{56}$ be independent standard exponential variables, and set $B_i=E_i/\\sum_j E_j$, so that $(B_1,\\ldots,B_{56})$ has the Dirichlet$(1,\\ldots,1)$ law. Consider the event $p_1/(p_1+p_2)<1/4$. Since $P_1/(P_1+P_2)=U_1/(U_1+U_2)$, the event holds for $P$ exactly when $U_10$, the weighted least-squares coefficients satisfy\n\\[\n\\widehat b(cw)=(cX^\\top WX)^{-1}cX^\\top Wy=\\widehat b(w).\n\\]\nA refit with uniform weights has the same coefficients whether or not the weights are divided by their sum.\n\nFor independent weights, to first order the covariance of the refitted coefficients is $\\sigma^2/\\mu^2$ times the unadjusted heteroskedasticity-robust covariance (HC0) of the least-squares coefficients, where $\\mu$ and $\\sigma^2$ denote the mean and variance of a weight. The factor is $1/3$ for $U(0,1)$ weights and $1$ for standard exponential weights. This is a statement about the linearized coefficients. It is not a correction for the sample at hand, and it does not justify multiplying the printed entries by $\\sqrt{3}$.\n\nI ran the two loops of the released code in a reconstruction on the public data (ICPSR 37070 V2), with adapted inputs and 5,000 replicates as in the code, and then refitted each regression 50,000 times with uniform weights and with Dirichlet weights built from the same draws ($U$ for the uniform weight and $-\\log U$ for the exponential one). The table reports the results, and the figure shows the ratios.\n\n\\begin{audittable}{llrrrr}{The twelve entries labelled ``Bayesian Bootstrap SE''. Printed: the entries of Tables S6 and S7. Uniform: standard deviation of the coefficient over 5,000 replicates of the released code's loop on the public data. Dirichlet: standard deviation over 50,000 refits with Dirichlet$(1,\\ldots,1)$ weights on the public data, with its conditional Monte Carlo standard error in parentheses. Ratio: uniform over Dirichlet in 50,000 paired refits. The values in the last three columns are computed from the public data; they do not establish exact values for the authors' own data file.}\nCoefficient & Outcome & Printed & Uniform & Dirichlet & Ratio \\\\\nS6, main effect & Talk & 0.01 & 0.0125 & 0.0182 (0.000059) & 0.683 \\\\\n & Norms & 0.05 & 0.0539 & 0.0761 (0.000247) & 0.711 \\\\\n & Wristband & 0.01 & 0.0081 & 0.0115 (0.000036) & 0.693 \\\\\n & Conflict & 0.03 & 0.0252 & 0.0349 (0.000110) & 0.720 \\\\\nS7, 0\\% referents & Talk & 0.03 & 0.0250 & 0.0352 (0.000115) & 0.710 \\\\\n & Norms & 0.11 & 0.1107 & 0.1515 (0.000495) & 0.715 \\\\\n & Wristband & 0.02 & 0.0160 & 0.0222 (0.000069) & 0.712 \\\\\n & Conflict & 0.05 & 0.0510 & 0.0692 (0.000220) & 0.723 \\\\\nS7, interaction & Talk & 0.22 & 0.2117 & 0.2986 (0.000957) & 0.720 \\\\\n & Norms & 0.80 & 0.8183 & 1.1336 (0.003750) & 0.706 \\\\\n & Wristband & 0.13 & 0.1370 & 0.1863 (0.000602) & 0.716 \\\\\n & Conflict & 0.35 & 0.3575 & 0.4821 (0.001569) & 0.730 \\\\\n\\end{audittable}\n\nThe twelve ratios lie between 0.683 and 0.730, so the standard deviations under uniform weights are 27.0\\% to 31.7\\% smaller. The public data reproduce the printed tables closely but not exactly. Nine of the twelve point estimates and eleven of the twelve robust standard errors round to the printed values, and each of the twelve standard deviations under uniform weights is compatible with the printed entry once rounding and two Monte Carlo standard errors are allowed (eight of them with rounding alone). The robust covariance of the code is HC0.", "problem": "The Bayesian bootstrap assigns random weights to the units; the weights sum to one and have the Dirichlet$(1,\\ldots,1)$ law. Dirichlet weights are obtained from independent draws of the standard exponential distribution, each divided by the sum of the draws, or, for $n$ schools, as the $n$ gaps formed by $n-1$ ordered independent $U(0,1)$ draws together with the endpoints $0$ and $1$. The notes to Tables S6 and S7 describe a different rule, a weight for each school drawn from the standard uniform distribution, and the released code computes that rule. In each of 5,000 replicates it draws one independent $U(0,1)$ weight for each school, passes the weights to the regression unchanged, and keeps the coefficient; the entry is the standard deviation of the 5,000 coefficients (\\texttt{4. School results-with-correction.R}, lines 165--175 for Table S6 and 212--226 for Table S7). The code has no ordering, no gaps and no logarithm. Uniform weights do not have the Dirichlet law, and they do not acquire it when they are divided by their sum. The notes describe the computation correctly, and the name of the rows is wrong. The four entries of Table S6 and the eight entries of Table S7 are dispersions under uniform weights, and they are not standard errors from a Bayesian bootstrap. In the comparison using 50,000 paired refits per regression on the public data, each of the twelve uniform-weight dispersions is smaller than its counterpart with Dirichlet weights, by 27\\% to 32\\%. Read as Bayesian-bootstrap standard errors, the rows understate the dispersion under that method on the public data, and they cannot serve as a check of the robust standard errors by that method.", "what_survives": "This finding does not require changing the point estimates of Tables S6 and S7, the rows of heteroskedasticity-robust standard errors or the rows of standard errors from the wild bootstrap. For school-level effects, the main text computes confidence intervals and $P$ values from robust standard errors (p.~568), and the intervals of Fig.~2 use heteroskedasticity-robust standard errors (caption, p.~569); neither uses the rows in question. The student-level Wald tests use randomization-based variance estimates (p.~569); those analyses and the headline estimates do not depend on these rows. What changes is the reading of the twelve entries, which cannot be cited as standard errors from a Bayesian bootstrap. The values with Dirichlet weights in the table are Monte Carlo estimates on the public data. They are not replacements for the printed entries, which would require the authors' own data file; that file is not public, and ICPSR states that some variables of the release were changed. The comparison is specific to these regressions. It establishes no general ordering of the two dispersions and no coverage property of either weighting under the blocked design." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E005", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of eight lines of text beginning 'The survey that all students took at the beginning of the year'. Red boxes mark the sentence 'Norms questions were on a scale of 0 to 5, with answers ranging from almost nobody to almost everybody.'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence gives one response scale for the descriptive and the prescriptive norm questions.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006.png" }, { "alt": "Crop of the heading 'Descriptive norms scale (asked at pretest and posttest)' and the first two items of the list. A red box marks the lines 'How often do you see students...' and '(0 = Never, 1 = 1-2 times / month, 2 = About 1 time /week, 3 = 2-3 times / week, 4 = Every day)'.", "caption": "SI Appendix, p. 8. The supplement’s own statement of the descriptive-norm question and of its scale from 0 (Never) to 4 (Every day), which the sentence on p. 567 does not describe.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-b.png" }, { "alt": "Crop of four lines. A red box marks '(Yes/No)' after 'Talking to friends item: This year, friends talked with me about how to change student behavior'. The next item, 'Orange wristband report item: I wore an orange Roots wristband this past week', is also followed by '(Yes/No)', which is not marked.", "caption": "SI Appendix, p. 10. The marked answer format is wrong for the talking item, which had four answers coded 0 to 3. The wristband item below it is also printed with a Yes/No answer.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-c.png" }, { "alt": "Crop of the heading 'Prescriptive norms scale', the stem 'How many students think it is...' and items 1 to 9 of the list. Red boxes mark item 3, 'Not funny to mess with or pick on others'; item 4, 'Not OK to say mean things to girl because of what she does with boys'; item 5, 'Not funny to post online or text mean things'; item 6, 'Not OK to use racial, ethnic jokes'; and item 8, 'OK to tell adults about conflict'.", "caption": "SI Appendix, p. 9, list of the prescriptive-norm items. The marked items (3–6 and 8) have the opposite orientation to the questionnaire statements.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-d.png" }, { "alt": "Crop of Table S3, 'Factor loadings for Prescriptive norm items', with the columns Survey Item, Wave 1 and Wave 2. Red boxes mark the consecutive row labels 'Item 7' and 'Item 9', between which there is no Item 8, and the last row label, 'Item 13', which has a Wave 2 loading of minus 0.38 and no Wave 1 entry.", "caption": "SI Appendix, Table S3, p. 38. The rows keep the item numbers of the questionnaire: there is no Item 8, and Item 13 corresponds to item 12 of the list on p. 9.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-e.png" } ], "formal_release": null, "formal_statements": [], "headline": "The descriptive-norm items had a frequency scale from 0 to 4 and the talking item a scale from 0 to 3, not the single 0–5 scale and the Yes/No answer that are printed; the supplement’s lists of items also need a key to the item numbers of Tables S3 and S4.", "id": "P037-E006", "location": "Methods and Materials, p. 567 (PDF p. 2), right column, paragraph beginning “The survey that all students took”; SI Appendix, pp. 8–10 (survey wording) and pp. 38–39 (Tables S3 and S4)", "pdf_page": 2, "printed_page": "567", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Main text, p.~567: \\ldots\\ students' estimates of descriptive norms (\\hl{how many students at the school} participate in various forms of conflict), and prescriptive norms (how many students at the school think it is desirable to participate in various forms of conflict). \\hl{Norms questions were on a scale of 0--5}, with answers ranging from ``almost nobody'' to ``almost everybody.''\n\nSI p.~10: Talking to friends item: ``This year, friends talked with me about how to change student behavior'' \\hl{(Yes/No)}\n\nSI p.~9, prescriptive norms scale, ``How many students think it is\\ldots'', items 3--6 and 8 of the list:\n\\begin{itemize}\n\\item 3. \\hl{Not funny} to mess with or pick on others\n\\item 4. \\hl{Not OK} to say mean things to girl because of what she does with boys\n\\item 5. \\hl{Not funny} to post online or text mean things\n\\item 6. \\hl{Not OK} to use racial, ethnic jokes\n\\item 8. \\hl{OK} to tell adults about conflict\n\\end{itemize}\n\nTable S3 (SI p.~38) has the rows Item 1 to Item 7 and Item 9 to Item 13. Table S4 (SI p.~39) has the rows Item 1 to Item 12 and Item 14.", "corrected": "Main text, p.~567: Descriptive-norm questions asked \\hl{how often students saw the listed behaviors at school (0 = Never, 1 = 1--2 times/month, 2 = About 1 time/week, 3 = 2--3 times/week, 4 = Every day)}. Prescriptive-norm questions asked how many students at the school \\hl{endorsed the listed statements, with six answers coded 0--5 (Almost nobody, A few people, About 25\\%, About 50\\%, About 75\\%, Almost everyone)}.\n\nSI p.~10: Talking to friends item: ``This year, \\hl{my} friends talked with me about how to change \\hl{the way students at this school act around each other}'' \\hl{(0 = Not true, 1 = A little bit, 2 = Some, 3 = Very true)}\n\nSI p.~9: Items 3--6 and 8 of the list are summaries in the opposite orientation to the questionnaire wording. In the questionnaire, item 6 reads ``It's okay to use jokes to insult someone else's race or ethnicity'', and the item listed as 8 says that it is not okay to tell an adult at school about student conflicts. \\hl{The supplied scripts compute the index as the first principal component of the covariance matrix of the selected input variables, on complete cases, without reversing individual items. The outcome analyses use this component with its sign reversed as a whole. The loadings of Table S3 refer to these input variables, up to the overall sign of the component.} For item 8 of the list at the second wave, the heading in the codebook of the public release reads ``okay'', against the questionnaire of both waves and the investigators' codebook. I report this difference of labels and draw no conclusion from it about the data.\n\nTables S3 and S4: The item numbers are those of the questionnaire. Items 9--13 of Table S3 are items 8--12 of the list on SI p.~9, and Item 14 of Table S4 is item 13 of the list on SI p.~8. Prescriptive item 8 and descriptive item 13 of the questionnaire, on working hard at school, appear in neither the lists nor the tables, and the scripts leave them out of the indices.", "evidence": "I checked the pages against the questionnaire and the two codebooks of the public release (ICPSR 37070, version 2). Both waves of the questionnaire print the five frequency answers for the descriptive items and six answers, from ``Almost nobody'' to ``Almost everyone'', for the prescriptive items. The second wave prints the talking statement with the answers Not true, A little bit, Some and Very true. The investigators' codebook marks the prescriptive items 3--6 and 9 of the questionnaire for reverse coding and lists the supplement's wording as the intended labels of the reversed items. These notes explain the wording of the list. They do not show that the items were recoded.\n\nI then read the authors' analysis scripts and ran the parts that bear on these items on the public data with an adapter for the released file. The scripts select the input variables by their questionnaire numbers and do not reverse any single item (\\texttt{2. Cleaning.R}, lines 56--61). The index is the first principal component of the covariance matrix of the selected input variables, on complete cases (\\texttt{5. Factor loadings and description.R}), and the scripts for the outcome analyses reverse the sign of the whole prescriptive component. On the public data all 48 loadings of Tables S3 and S4 agree with the printed values to two decimals, after one choice of overall sign for each component. The talking item keeps its four scores. On the public data, the adjusted school-level effect on talking is $0.0874$ points of the 0--3 score (robust HC0 standard error $0.0176$) in 56 schools, and Table S6 prints $0.09$ and $0.02$.", "problem": "The main text describes both sets of norm items with one question and one response scale. The question and the scale are those of the prescriptive items. The descriptive items asked ``How often do you see students\\ldots'', with five answers coded from 0 (Never) to 4 (Every day), as SI p.~8 states. The talking item had four answers coded 0 to 3, as the axes of Figs.~2A and 3A say. This implies that the effect of 0.09 on talking in Table S6 is in points of that score. It is not a difference of nine percentage points in the proportion answering Yes.\n\nIn the supplement's list of prescriptive items, items 3--6 and 8 have the opposite orientation to the questionnaire wording, and the list does not say so. The two lists also renumber the items after two questions on schoolwork were left out, while Tables S3 and S4 keep the numbers of the questionnaire. Items 8--12 of the list on SI p.~9 are Items 9--13 of Table S3, and item 13 of the list on SI p.~8 is Item 14 of Table S4.", "what_survives": "These documentation corrections require no change to the reproduced loadings of Tables S3 and S4 or the talking effect and robust standard error in Table S6, which agree on the public data at the printed precision. The authors' own input files are not public and ICPSR states that some variables of the release were changed, so I cannot establish their original inputs or coding before the supplied scripts; I did not check other estimates here. I did not examine here whether the final norms score has the direction that the article states; the orientation of the index is discussed in \\fid{P037-N023}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E006", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Table S1, Summary statistics for each school block and for overall sample: fourteen school blocks and a Total Sample row, each with a row of means and a row of dispersions, in sixteen columns, among them Home language, School size and percent White. A red box marks the column heading Home language. A second red box marks 0.40, the Total Sample mean of that column. In the note below the table, two red boxes mark the words speak only English (1) vs. other languages (0) across a line break.", "caption": "SI Appendix, Table S1 and the first lines of its note (PDF p. 36; the page is printed sideways and is shown upright). The marks are on the column heading “Home language”, on its Total Sample mean 0.40, and on the words of the note that define the column as the proportion who speak only English (1) vs. other languages (0).", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E007.png" } ], "formal_release": null, "formal_statements": [], "headline": "The note to Table S1 reverses the coding of the home-language column: the entries are the share of students who speak a language other than English at home, alone or together with English, and not the share who speak only English.", "id": "P037-E007", "location": "SI Appendix, Table S1 (PDF p. 36 of the SI Appendix; the page carries no number), column “Home language” and its definition in the note", "pdf_page": 36, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Note to Table S1: ``home language'' represents the average proportion of students who reported that they speak \\hl{only English (1) vs. other languages (0)} at home in the pre-randomization student surveys;\n\nColumn ``Home language'', means for school blocks 1 to 14: 0.42, 0.35, 0.44, 0.20, 0.41, 0.23, 0.71, 0.35, 0.84, 0.11, 0.31, 0.28, 0.41, 0.49. Total Sample: 0.40, with 0.24 in the second row.", "corrected": "Note to Table S1: ``home language'' represents the average proportion of students who reported \\hl{speaking a language other than English at home, either alone or together with English (1), versus speaking only English (0)}, in the pre-randomization student surveys.\n\nThe entries of the column stay as printed. If the share who speak only English is wanted instead, the means are replaced by their complements, taken before rounding, and the standard deviations stay as printed. On the public data the complements for school blocks 1 to 14 are 0.58, 0.65, 0.56, 0.80, 0.59, 0.77, 0.29, 0.65, 0.16, 0.89, 0.69, 0.72, 0.59 and 0.51, and 0.60 for the Total Sample.", "evidence": "The evidence is a reading of the released code and a recalculation on the public data. I ran the relevant recoding and aggregation lines on the public data (ICPSR 37070 V2), with explicit translation of factor and missing-value codes. All 14 block means, all 14 block standard deviations and the two entries of the Total Sample row agree with the printed column at two decimals. The standard deviations are computed with divisor 4 within a block and 56 for the whole sample. On the public data, the mean over the 56 schools of the share who speak another language at home is 0.3968, the mean of the share who speak only English is 0.6032, and the standard deviation is 0.2383. In the public file, 14,342 students answer English only, 706 another language only and 7,671 English and another language, and 1,752 answers are missing. If Table S1's participants form a subset of the public release with unchanged language answers, the printed definition also conflicts with the public counts without reference to the code: by the Cauchy--Schwarz inequality, the block means and standard deviations of the column and of school size allow at most 9,273 English-only answers among the 22,732 survey participants of Table S1, and any 22,732 of the 24,471 released records contain at least 12,603.\n\nFor the regressions, let $x$ denote the language control and $Z$ the other columns of the design. Since the intercept and the block fixed effects span the constant, there exists a vector $c$, zero outside those columns, such that $Zc=\\mathbf{1}$, and I have $Z\\theta+x\\gamma=Z(\\theta+c\\gamma)+(1-x)(-\\gamma)$. This identity implies that replacing $x$ by $1-x$ on the same analysis rows leaves the column space, the fitted values, the residuals and the treatment coefficients unchanged. In the two adjusted regressions for peer conflict at the school level, on the public data (49 schools), the treatment and interaction coefficients change by less than $4\\times 10^{-16}$ and their robust standard errors by less than $1.2\\times 10^{-15}$.", "problem": "Read with the note, the column says that on average across schools 40\\% of students speak only English at home. The released code computes the column from the opposite indicator. The cleaning script codes a student as 0 for English only and as 1 for another language, alone or together with English (\\texttt{2. Cleaning.R}, lines 42--48), and the script that produces Table S1 averages this indicator within schools and then over the four schools of each block (\\texttt{7. Blocking, randomization, and summary statistics.R}, lines 112--125 and 159--195). The entries of the column are shares of students who speak another language at home, and the definition in the note is reversed.", "what_survives": "The numbers of the column stay as printed, and the correction of the note changes no computation. The adjusted regressions at the school level and the network regression use the same indicator, of another language at home, as an additive control (\\texttt{4. School results-with-correction.R}, lines 162 and 208; \\texttt{3. Network results.R}, line 216). If the control were replaced by its exact English-only complement on the same analysis rows, with unchanged regression weights, the treatment coefficients, the coefficients of the interaction with composition and their robust standard errors would not change; this follows from the identity above for the additive regressions, and I checked it numerically only for the two regressions for peer conflict. The row ``Home language'' of Fig.~S2 is computed from the three-category answer (0, 1, 2), not from this indicator, and the correction does not bear on it. The row ``Home language is English'' of Table S2 is coded separately, with English only as 1. The authors' own data file is not public, and ICPSR notes that some variables in the public release were changed; the agreement of all 30 printed entries does not show that the public file is identical to it." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E007", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Six lines of text from page 6 of the SI Appendix. Two red boxes mark the parenthesis (schools with less than 550 students) across a line break, after the words no more than 64 seed-eligible students. The next sentence reads: For schools with between 200 and 550, we scaled linearly between 40 and 64 seed-eligible students, rounding to the nearest 4.", "caption": "SI Appendix, p. 6 (PDF p. 6), first paragraph. The marked parenthesis attaches the cap of 64 seed-eligible students to schools with less than 550 students; the sentence after it scales the number from 40 at 200 students to 64 at 550.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E008.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T12", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T12" } ], "headline": "The parenthesis attached to the cap of 64 seed-eligible students names the wrong range of school sizes: the upper branch of the rule applies at 550 students and holds for larger schools.", "id": "P037-E008", "location": "SI Appendix, “Study design”, p. 6 (PDF p. 6 of the SI Appendix), first paragraph, sentence beginning “such that we had no fewer than 40”; compare the main text, p. 567 (PDF p. 2)", "pdf_page": 6, "printed_page": "6", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T12" ], "text": { "as_printed": "``such that we had no fewer than 40 seed-eligible students (schools with fewer than 200 students), and no more than 64 seed-eligible students (schools with \\hl{less than} 550 students). For schools with between 200 and 550, we scaled linearly between 40 and 64 seed-eligible students, rounding to the nearest 4.''", "corrected": "``\\ldots and no more than 64 seed-eligible students (schools with \\hl{at least} 550 students).''\n\nWith explicit endpoints, the published rule is: 40 seed-eligible students for a school of at most 200 students, 64 for a school of at least 550 students, and for sizes in between the linear interpolation between 40 and 64, rounded to the nearest multiple of four.", "evidence": "I checked the page, recalculated the printed rule and checked the public data (ICPSR 37070 V2). Under the printed rule, a school of 100 or of 200 students has 40 seed-eligible students, a school of 375 has 52, and a school of 550 or of 715 has 64.\n\nThe authors' analysis code does not compute the eligible group; it reads eligibility from the data. I ran its filtering and classification lines with an adapter on the public data, retaining 24,191 students in 56 schools and 728 seeds in the 28 treatment schools, matching the published counts. The public school-size variable INCSIZE reproduces all 14 block means of school size in Table S1. With that variable, each of the 14 schools of at least 550 students has 64 seed-eligible students, and each of the nine schools of at most 200 students has 40. Of the 33 schools in between, 14 have the number that the printed rule predicts and 19 do not.", "problem": "The sentence after the parenthesis scales the number of seed-eligible students linearly from 40 at 200 students to 64 at 550. The upper branch fixes the number at 64 for schools with at least 550 students; rounding can also give 64 below 550. Table S1 has a block of schools whose mean size is 715. The parenthesis should name the schools with at least 550 students.", "what_survives": "The rule is recoverable from the sentence after the parenthesis, and correcting the wording leaves the released analysis unchanged. The released analysis code reads eligibility as given and does not reconstruct its selection. The checks on the public data do not establish an implementation or randomization failure, but do not confirm the printed interpolation for every school of intermediate size. The selection routine is absent, and I could not identify the enrollment figures used at selection; ICPSR reports changes to some released variables. The main text's ``designed to represent 15\\% of the school population'' (p.~567) states the aim of the design and needs no change. The seed-eligible students are not 15\\% of each school, since 40 of 200 students is 20\\% and 64 of 550 is $11.6\\%$; on the public data, using INCSIZE as the denominator and averaging school shares across the 28 treatment schools, the printed rule gives $15.0\\%$ and the eligible groups in the data $14.8\\%$." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E008", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Fig. 3, four panels with its caption at the right. Each panel shows four dots with dotted vertical 95 percent confidence intervals, for the exposures 'No exposure from seed or school', 'Treatment school, no exposure to seed student', 'Exposed to non-referent seed student' and 'Exposed to social referent seed student'. Red boxes mark the rightmost dot of panel B, which lies slightly above 0.0, and the rightmost dot of panel D, which lies a little below 0.2.", "caption": "Fig. 3 and its caption, p. 570 (PDF p. 5). The two marked dots are the estimates for students exposed to a social referent seed in panel B (perceived social norm against conflict) and panel D (disciplinary records of peer conflict), where the mean gap from Table S5 is largest within each of those panels.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E009.png" }, { "alt": "Crop of Table S5, 'Causal social influence effects from seed students', with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict and four exposure rows. Red boxes mark, in the row 'Exposed to social referent seed student', the Prescriptive norms entry 0.007 with standard error (0.113) and the Peer conflict entry 0.170 with standard error (0.037). A third red box marks the words 'Figure 4' in the last sentence of the note, 'A graphical version of this table is presented in Figure 4 in the main text of the paper.'", "caption": "SI Appendix, Table S5, p. 40 (supplement PDF p. 40), shown upright; the table is printed sideways on the page. The marked entries are the estimates and standard errors for students exposed to a social referent seed, for prescriptive norms and for peer conflict. The marked words of the note are its reference to the corresponding figure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E009-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Fig. 3 and Table S5 give different values for the same exposure means and standard errors: 10 of the 16 means and 11 of the 16 standard errors differ by more than rounding and the drawing allow, and the authors' code, run on the public data, reproduces the figure and not the table.", "id": "P037-E009", "location": "Fig. 3, panels A–D, p. 570 (PDF p. 5); SI Appendix, Table S5, p. 40 (supplement PDF p. 40)", "pdf_page": 5, "printed_page": "570", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{audittable}{lrr}{Table S5 (SI p.~40), first row, two of its four columns; standard errors in parentheses.}\n & Prescriptive norms & Peer conflict \\\\\nExposed to social referent seed student & \\hl{0.007} & \\hl{0.170} \\\\\n & \\hl{(0.113)} & \\hl{(0.037)} \\\\\n\\end{audittable}\n\nTable S5, note: ``A graphical version of this table is presented in \\hl{Figure 4} in the main text of the paper.''\n\nFig.~3 (p.~570) plots the four exposure means for each outcome with 95\\% confidence intervals and prints no numbers.", "corrected": "Table S5 should be regenerated from the estimates that Fig.~3 plots. On the public data, the authors' table script gives:\n\n\\begin{audittable}{lrr}{Table S5 as printed, and as the authors' table script gives it on the public data (ICPSR 37070 V2): estimate (standard error). The outcomes are the four columns of the table, Friends talk about conflict (Talk), Prescriptive norms (Norms), Wristband and Peer conflict (Conflict); the exposures are its four rows. Highlighted values differ from the printed ones.}\nOutcome and exposure & As printed & On the public data \\\\\nTalk, social referent seed & 0.747 (0.032) & 0.747 (\\hl{0.033}) \\\\\nTalk, non-referent seed & 0.696 (0.051) & \\hl{0.704} (\\hl{0.050}) \\\\\nTalk, treatment school, no seed & 0.695 (0.084) & 0.695 (\\hl{0.080}) \\\\\nTalk, no seed or school & 0.624 (0.020) & \\hl{0.625} (0.020) \\\\\nNorms, social referent seed & 0.007 (0.113) & \\hl{0.030} (\\hl{0.106}) \\\\\nNorms, non-referent seed & $-0.468$ (0.163) & \\hl{$-0.456$} (\\hl{0.151}) \\\\\nNorms, treatment school, no seed & $-0.825$ (0.254) & \\hl{$-0.823$} (\\hl{0.258}) \\\\\nNorms, no seed or school & 0.141 (0.075) & \\hl{0.127} (\\hl{0.071}) \\\\\nWristband, social referent seed & 0.232 (0.012) & \\hl{0.229} (0.012) \\\\\nWristband, non-referent seed & 0.124 (0.016) & \\hl{0.123} (0.016) \\\\\nWristband, treatment school, no seed & 0.095 (0.025) & \\hl{0.096} (\\hl{0.026}) \\\\\nWristband, no seed or school & 0.002 (0.008) & \\hl{0.004} (0.008) \\\\\nConflict, social referent seed & 0.170 (0.037) & \\hl{0.183} (\\hl{0.035}) \\\\\nConflict, non-referent seed & 0.190 (0.052) & \\hl{0.185} (\\hl{0.049}) \\\\\nConflict, treatment school, no seed & 0.152 (0.072) & \\hl{0.148} (\\hl{0.077}) \\\\\nConflict, no seed or school & 0.200 (0.027) & \\hl{0.196} (\\hl{0.026}) \\\\\n\\end{audittable}\n\nThese values come from the public data and not from the authors' own files. \\hl{Before rounding,} they agree with all 32 values of Fig.~3 within the precision of the drawing. Their samples are 2,457, 2,273, 2,847 and 2,495 students for the four outcomes, and the note's ``subpopulation of 2,451 students'' does not describe them (\\fid{P037-N025}). If the authors' files give another final output, the figure and the table should both be regenerated from it and the difference explained. In the note, ``Figure 4'' should read ``\\hl{Figure 3}'' (\\fid{P037-E010}).", "evidence": "The evidence is a recalculation from the published PDF and a run of the authors' code on the public data.\n\nFrom the PDF, I read each dot and interval of Fig.~3 from the vector drawing and calibrated it on the grid lines of its panel; the standard error is the length of the interval divided by $2 \\times 1.96$. A value counts as a mismatch only if the figure, with $\\pm 0.1$ PDF point allowed for each coordinate, and the table, with rounding to three decimals allowed, cannot coincide. By this count, 10 of the 16 means and 11 of the 16 standard errors do not match.\n\nOn the public data (ICPSR 37070 V2), I ran the authors' code for the four outcomes of Fig.~3: 10,000 simulated assignments for the exposure probabilities, the first 1,000 of them for the standard errors, the seed of the code, the random sampler that R used before version 3.6, the sign convention for principal components that it used before version 3.5, and social referents defined by indegree. All 16 unrounded means and all 16 unrounded standard errors agree with Fig.~3 within the precision of the drawing; the largest differences are about $0.00002$ between a mean and its plotted interval's midpoint and $0.000004$ between a standard error and the value inferred from its interval's width. Rounded to three decimals as the table script rounds them, 14 of the 16 means and 12 of the 16 standard errors differ from the printed Table S5. The table below shows panel B and the referent row of panel D.\n\n\\begin{audittable}{lrrrrrr}{Means and standard errors read from Fig.~3, computed by the authors' code on the public data, and printed in Table S5 (S5).}\nPanel and exposure & Fig.~3 mean & Public data & S5 mean & Fig.~3 SE & Public data & S5 SE \\\\\nB, control school & $0.127$ & $0.127$ & $0.141$ & $0.071$ & $0.071$ & $0.075$ \\\\\nB, treatment school, no seed & $-0.823$ & $-0.823$ & $-0.825$ & $0.258$ & $0.258$ & $0.254$ \\\\\nB, nonreferent seed & $-0.456$ & $-0.456$ & $-0.468$ & $0.151$ & $0.151$ & $0.163$ \\\\\nB, social referent seed & $0.030$ & $0.030$ & $0.007$ & $0.106$ & $0.106$ & $0.113$ \\\\\nD, social referent seed & $0.183$ & $0.183$ & $0.170$ & $0.035$ & $0.035$ & $0.037$ \\\\\n\\end{audittable}\n\nThe public data are not the authors' own analysis files, which are not available, and ICPSR states that some variables in the release were changed. The run shows that the released code and the public data reproduce the figure. It does not show which output the authors took as final, or why the printed table differs.", "problem": "Table S5 and Fig.~3 are two displays of one set of estimates: the covariate-adjusted mean of each outcome under each of the four exposure conditions, with standard errors computed under constant effects. The note to the table calls the figure its graphical version, and the figure it means is Fig.~3 (\\fid{P037-E010}). In the authors' code the two displays have one source. The script that draws the figure plots each mean with an interval of $\\pm 1.96$ standard errors and saves the same means and standard errors, and the script that writes the table reads the saved values and rounds them, with no further estimation. The two displays should then agree to the rounding of the table. They do not. For students exposed to a social referent seed, the mean is about $0.030$ in Fig.~3B against $0.007$ in the table, and about $0.183$ in Fig.~3D against $0.170$. The interval in Fig.~3B corresponds to a standard error of about $0.106$, against $0.113$. At least one of the two displays does not show the output it is said to show, and the publication does not say which.", "what_survives": "In every panel the four point estimates have the same order in the figure and in the table, including the larger means under referent exposure in panels A to C, and no stated conclusion is shown to reverse. The largest gap, $0.007$ against $0.030$, is a fraction of the standard error of that mean, which is $0.11$. Since the marginal standard errors of the table do not determine the variance of a contrast, the mismatch does not show that a reported test changes. For the contrast between referent and nonreferent exposure on norms, the difference on the public data is $0.486$, with standard error $0.189$ and $P = 0.0101$, where the simulation error of $P$, conditional on the estimated exposure probabilities and fitted residuals, is about $0.0017$. The crossing of $0.01$ is within that error, and it does not establish that the printed $P < 0.01$ changes. The school-level results of Figs.~1 and 2 and Tables S6 and S7 are separate estimates." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E009", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of a paragraph of five lines. A red box marks 'Points 1 - 6' in the sentence 'Points 1 - 6 follow a rough chronological introduction of the program’s principles'.", "caption": "SI Appendix, p. 2. The marked range should read 1–5: the list that follows has five principles.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010.png" }, { "alt": "Crop of the note of six lines under Table S5. A red box marks 'Figure 4' in the last sentence, 'A graphical version of this table is presented in Figure 4 in the main text of the paper.'", "caption": "SI Appendix, note to Table S5, p. 40. The page is printed sideways and is shown here turned upright. The marked reference should be to Figure 3.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-b.png" }, { "alt": "Crop of four lines of text. A red box marks 'Table 1' in the sentence 'See Table 1 for variable descriptions.'", "caption": "SI Appendix, p. 25, text under Fig. S2. The marked reference should be to Table S1.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-c.png" }, { "alt": "Crop of references 30 and 31. A red box marks 'Hartwick H' in 'Barki H, Hartwick H (2004) Conceptualizing the construct of interpersonal conflict. Int J Conflict Manage 15(3):216-244.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 31. The marked author is J. Hartwick.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-d.png" }, { "alt": "Crop of references 38 and 39. A red box marks '(2011)' in 'Marwick A, Boyd D (2011) It's just drama: Teen perspectives on conflict and aggression in a networked era. J Youth Stud 17(9):1187-1204.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 39. The marked year should be 2014.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-e.png" }, { "alt": "Crop of references 47 to 49. A red box marks '47(1)' in 'Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. J Am Stat Assoc 47(1):663-685.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 48. The marked issue number should be 260.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-f.png" } ], "formal_release": null, "formal_statements": [], "headline": "Six slips of cross-reference and citation: the supplement lists five principles, not six; the graph of Table S5 is Fig. 3 of the article; the variable descriptions are in Table S1; and references 31, 39 and 48 each have one wrong detail.", "id": "P037-E010", "location": "SI Appendix, p. 2 (PDF p. 2 of the supplement), paragraph beginning “Each principle described below”; also SI p. 25 (text under Fig. S2), SI p. 40 (note to Table S5), and the reference list of the article, p. 571 (PDF p. 6), references 31, 39 and 48", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "SI p.~2: Points 1 - \\hl{6} follow a rough chronological introduction of the program’s principles\n\nSI p.~40, note to Table S5: A graphical version of this table is presented in Figure \\hl{4} in the main text of the paper.\n\nSI p.~25: See Table \\hl{1} for variable descriptions.\n\np.~571: 31. Barki H, Hartwick \\hl{H} (2004) Conceptualizing the construct of interpersonal conflict. \\emph{Int J Conflict Manage} 15(3):216--244.\n\np.~571: 39. Marwick A, Boyd D (\\hl{2011}) ‘It’s just drama’: Teen perspectives on conflict and aggression in a networked era. \\emph{J Youth Stud} 17(9):1187--1204.\n\np.~571: 48. Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. \\emph{J Am Stat Assoc} 47(\\hl{1}):663--685.", "corrected": "SI p.~2: Points 1--\\hl{5} follow a rough chronological introduction of the program’s principles\n\nSI p.~40, note to Table S5: A graphical version of this table is presented in Figure \\hl{3} in the main text of the paper.\n\nSI p.~25: See Table \\hl{S1} for variable descriptions.\n\np.~571: 31. Barki H, Hartwick \\hl{J} (2004) Conceptualizing the construct of interpersonal conflict. \\emph{Int J Conflict Manage} 15(3):216--244.\n\np.~571: 39. Marwick A, Boyd D (\\hl{2014}) ‘It’s just drama’: Teen perspectives on conflict and aggression in a networked era. \\emph{J Youth Stud} 17(9):1187--1204.\n\np.~571: 48. Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. \\emph{J Am Stat Assoc} 47(\\hl{260}):663--685.", "evidence": "I checked the pages and, for the three references, the publishers' bibliographic information. The page for reference 31 names Jon Hartwick as second author. The page for reference 39 reads \\emph{Journal of Youth Studies}, volume 17, 2014, issue 9, pages 1187--1204. The page for reference 48 reads \\emph{Journal of the American Statistical Association}, volume 47, 1952, issue 260, pages 663--685.\n\nFor the two cross-references of the supplement I also read the authors' analysis scripts and ran their graph and table assembly steps with distinct test values in place of estimates. All sixteen mean inputs and sixteen standard-error inputs pass to Table S5 with the exposure order reversed. The scripts for Fig.~S2 and for Table S1 give fifteen corresponding covariate labels. The targets are Fig.~3 and Table S1. This check follows the inputs and labels, and it reproduces no estimate.", "problem": "The supplement lists five guiding principles; the list ends with principle 5 on SI p.~4. The article has three figures, and the graph of Table S5 is Fig.~3. The fifteen covariates of Fig.~S2 are described in the note to Table S1 (SI p.~36). In the reference list, the second author of reference 31 is J.~Hartwick, reference 39 was published in 2014, and reference 48 is in issue 260 of volume 47.", "what_survives": "Each intended target can be identified, and no estimate or conclusion depends on these items. The reference on SI p.~13 to ``the interaction model in (4)'' is not counted as an error, since equation (4) on SI p.~15 repeats the interaction model (2). This finding says nothing about the correctness of the numerical results." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E010", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of seven lines of the abstract in bold type. A red box marks 30 percent in the sentence 'Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by 30 percent over 1 year.'", "caption": "Abstract, p. 566 (PDF p. 1). The marked figure is the reduction in disciplinary reports, which the correction of 2018 replaces by 25%.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001.png" }, { "alt": "Crop of six lines of text. A red box marks 30 percent in the passage 'In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a 30 percent reduction in peer conflict reports.'", "caption": "Results, p. 569 (PDF p. 4), left column, first paragraph. The marked 30% is the quotient $0.06/0.20$ of the two rounded numbers in the same passage.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001-b.png" }, { "alt": "Crop of seven lines of text. A red box marks 30 percent in 'our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated 30 percent over 1 y.' The next sentence reads 'To put this in perspective, our estimates imply that the intervention reduced the total number of disciplinary events from 2,695 events to 2,012 events across the 11,938 students in treatment schools.'", "caption": "Discussion, p. 571 (PDF p. 6), left column, first paragraph. Three lines below the marked figure are the totals 2,695 and 2,012, which give a reduction of 25.3%.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001-c.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T6", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T6" }, { "id": "P037-T17", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T17" } ], "headline": "The reduction in disciplinary reports of peer conflict is 25%, not 30%; the authors' correction of 2018 makes this change in the four places where 30% is printed.", "id": "P037-N001", "location": "Abstract, p. 566 (PDF p. 1), line 11; also the Significance statement, p. 566, line 8; Results, p. 569 (PDF p. 4), left column, first paragraph; Discussion, p. 571 (PDF p. 6), left column, first paragraph", "pdf_page": 1, "printed_page": "566", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T17", "P037-T6" ], "text": { "as_printed": "Abstract: Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by \\hl{30\\%} over 1 year.\n\nSignificance: \\ldots\\ our intervention reduced overall levels of conflict by an estimated \\hl{30\\%}.\n\nResults, p.~569: In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a \\hl{30\\%} reduction in peer conflict reports.\n\nDiscussion, p.~571: \\ldots\\ our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated \\hl{30\\%} over 1 y. To put this in perspective, our estimates imply that the intervention reduced the total number of disciplinary events from 2,695 events to 2,012 events across the 11,938 students in treatment schools.", "corrected": "Abstract: Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by \\hl{25\\%} over 1 year.\n\nSignificance: \\ldots\\ our intervention reduced overall levels of conflict by an estimated \\hl{25\\%}.\n\nResults, p.~569: In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year \\hl{(the raw control-school mean in the 56-school calculation, rounded)}. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools \\hl{(a covariate-adjusted decrease, rounded)}, a \\hl{25\\%} reduction in peer conflict reports \\hl{when the mean predictions under control and under treatment are both taken, unrounded, from the covariate-adjusted interaction regression and averaged over the same 49 schools}.\n\nDiscussion, p.~571: \\ldots\\ our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated \\hl{25\\%} over 1 y.\n\nThe replacement of 30 by 25 in the four places is the correction of 2018. The three phrases added to the Results are mine. They state the basis of each number, since $0.06/0.20$ is not $25\\%$. On the public data the two mean predictions are $0.2225$ and $0.1660$, and the reduction is $25.4\\%$.", "evidence": "The evidence is a recalculation from the published numbers, a check of the pages against the correction of 2018 (PNAS 115(15):E3598), and a calculation with the authors' analysis scripts on the public release of the data (ICPSR 37070, version 2).\n\n\\emph{The published numbers.} The Discussion estimates 2,695 disciplinary events without the intervention and 2,012 with it, for the same 11,938 students. The reduction is\n\\[\n\\frac{2695-2012}{2695}=\\frac{683}{2695}\\approx 25.3\\%.\n\\]\nThis lies between $24.5\\%$ and $25.5\\%$, and it rounds to $25\\%$. It equals $30\\%$ only if $10\\cdot 683=3\\cdot 2695$, that is, $6830=8085$. If each total was rounded to a whole number of events, the reduction lies between $1364/5389\\approx 25.31\\%$ and $152/599\\approx 25.38\\%$. The quotient $0.06/0.20$ equals $30\\%$ exactly, and the printed figure is not a slip of division.\n\n\\emph{The scripts and the public data.} The authors' script \\texttt{4. School results-with-correction.R} computes the percentages of the correction. Line 208 fits the covariate-adjusted regression with the interaction of treatment and seed-group composition. Lines 253--256 average its predictions under control and under treatment over the 49 schools of the analysis, and line 286 divides their difference by the average prediction under control. I ran these lines on the public data, with an adapter that maps its variables to the inputs the scripts expect. The average predictions are $0.222537$ under control and $0.165975$ under treatment, and\n\\[\n\\frac{0.222537-0.165975}{0.222537}\\approx 25.42\\%.\n\\]\nThe same calculation gives $28.68\\%$ for the quotient of the adjusted difference and the raw control mean $0.197241$ (line 270), and $22.87\\%$ for the quotient of the raw difference and the raw control mean (line 289). These round to the 25\\%, 29\\% and 23\\% of the correction. The raw means use all 56 schools, as discussed in \\fid{P037-N009}.\n\nThe regression of Table S6 is a different fit (line 162, without the interaction). On the public data its coefficient is $-0.057272$, and the difference of the average predictions of the interaction regression is $-0.056562$. Both round to the printed $-0.06$. On the public data, I reproduce the Discussion's totals from the regression of Table S6: its average predictions over the 49 schools, multiplied by 11,938, are $2695.4$ and $2011.7$, a reduction of $25.37\\%$. The three reductions $25.34\\%$ (printed totals), $25.37\\%$ and $25.42\\%$ are different quantities, and each rounds to $25\\%$.\n\n\\begin{audittable}{lrr}{The reduction in disciplinary reports on each basis. The second column is from the article and the correction of 2018: the first row is the printed $0.06/0.20$, the third row is the correction, and the last row is $(2695-2012)/2695$. The third column is computed with the authors' script on the public data (ICPSR 37070, version 2), not on the authors' own files; its last row uses the totals $2695.4$ and $2011.7$.}\nBasis & As published & On the public data \\\\\n\\hline\nAdjusted effect over raw control mean, both rounded & 30\\% & 30\\% \\\\\nThe same quotient, unrounded & 29\\% & 28.68\\% \\\\\nAdjusted effect over adjusted control mean & 25\\% & 25.42\\% \\\\\nRaw effect over raw control mean & 23\\% & 22.87\\% \\\\\nTotals in the Discussion & 25.3\\% & 25.37\\% \\\\\n\\end{audittable}\n\nOn the public data, the authors' rule for selecting records keeps 24,191 students in 56 schools, 49 schools enter the regressions of disciplinary reports, and the treatment schools have 11,938 students. The recomputed quantities agree with the corresponding published values at their printed precision. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. The values above are values on the public data. They are not the unrounded values of the authors' files.", "problem": "The article reports the effect on disciplinary reports as a 30\\% reduction in four places. The figure is the quotient $0.06/0.20$ of the two numbers in the Results. According to the correction of 2018, the first is the covariate-adjusted estimate of the effect and the second is the unadjusted mean of the control schools, and both are rounded. The numerator and the denominator are on different bases. The correction states that the same quotient is 29\\% without rounding, that the reduction is 25\\% when the effect and the control mean are both covariate-adjusted, and that it is 23\\% when both are unadjusted. From the estimated totals in the Discussion, I also obtain 25\\% after rounding. A citation of the original figure overstates the reduction by five percentage points, one sixth of the printed 30\\%.", "what_survives": "This correction leaves the estimate of 0.06 fewer disciplinary reports per student (Table S6: $-0.06$, standard error $0.03$), the totals in the Discussion, the direction of the effect and the other analyses unchanged. The reduction is five percentage points smaller than printed. The correction requires no new causal assumption or estimator. The correction of 2018 also contains a sentence on statistical significance, which is the subject of \\fid{P037-N008}; the school-level tests are the subject of \\fid{P037-N006}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N001", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of six lines of text. Red boxes mark the sentence 'Each school's entire school body took a survey at the same time on a given day (n = 24,191 students).'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence gives 24,191 as the number of students who took the survey.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003.png" }, { "alt": "Crop of five lines of the note under Table S1. Red boxes mark the clause 'school size is the average school population as measured by the number of students who took the pre-randomization student surveys'.", "caption": "SI Appendix, note to Table S1, p. 36. The page is printed sideways and is shown here turned upright. The marked clause defines school size as the number of students who took the survey before randomization.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003-b.png" }, { "alt": "Crop of the first five columns of Table S1: School Block, percent White (self-ID), Home language, Friends like house and School size. Red boxes mark the column heading 'School size' and the Total Sample mean 405.93. The School size means of blocks 1 to 14 are 368.25, 389.75, 226.50, 715.00, 122.00, 528.50, 335.50, 492.75, 320.25, 318.50, 512.75, 468.75, 481.25 and 403.25.", "caption": "SI Appendix, Table S1, p. 36, first five columns, turned upright. For each block of four schools the marked column gives the mean school size and, on the line below, its dispersion. The fourteen means sum to 5,683. The marked entry 405.93 is the mean over the 56 schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "The article’s 24,191 is reproduced on the public data as the number of student records in the analysis, including students without a baseline survey response; it is not an established count of students who took the survey before randomization, and the school-size means of Table S1 imply a total of 22,732.", "id": "P037-N003", "location": "Methods and Materials, p. 567 (PDF p. 2), right column, paragraph beginning “We measured social connections”; compare SI Appendix, Table S1 and its note, p. 36", "pdf_page": 2, "printed_page": "567", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Main text, p.~567: Each school's \\hl{entire school body took a survey} at the same time on a given day \\hl{($n = 24{,}191$ students)}.\n\nSI p.~36, note to Table S1: ``school size'' is the average school population \\hl{as measured by the number of students who took the pre-randomization student surveys};", "corrected": "Main text, p.~567: \\hl{A school-wide baseline survey was administered on a given day at each school. The analysis sample contained 24,191 student records, including students without a baseline survey response.}\n\nOn the same page, ``The survey that all students took at the beginning of the year'' should read ``\\hl{The baseline survey}''.\n\nSI p.~36, note to Table S1: ``school size'' is \\hl{the variable \\texttt{incsize}, whose published block means imply a total of 22,732 over the 56 schools. The public release defines it as the school population included in the study, limited to the participating grades where applicable. Whether it also equals the number of students who took the survey cannot be established without the authors' files or a documented indicator of response.}", "evidence": "I recalculated the total from Table S1, read the authors' analysis scripts, and ran the parts that bear on these counts on the public data.\n\n\\emph{Table S1.} Each of the 14 blocks has four schools, and the implied number of survey takers is four times the sum of the block means in the column ``School size''. The means are 368.25, 389.75, 226.50, 715.00, 122.00, 528.50, 335.50, 492.75, 320.25, 318.50, 512.75, 468.75, 481.25 and 403.25. Their sum is $5683$, and\n\\[\n4\\cdot 5683=22732,\\qquad 24191-22732=1459 .\n\\]\nRounding does not account for the difference. Each printed mean is a multiple of $1/4$, and errors of at most $0.005$ in the 14 means change the total by at most $4\\cdot 14\\cdot 0.005=0.28$. The total agrees with the mean of the Total Sample row, since $22732/56\\approx 405.93$.\n\n\\emph{The scripts and the public data.} The script \\texttt{1. Sanitize and generate perms, probs.R} (lines 33--35) keeps the records with a known school block and a roster number other than 999. On the public data this rule keeps 24,191 records in 56 schools. The script \\texttt{7. Blocking, randomization, and summary statistics.R} takes the school size of Table S1 from the variable \\texttt{incsize}. Its block means on the public data equal the fourteen printed means, and its total is 22,732. Of the 24,191 records in the public data, 22,810 have a survey type (\\texttt{SURTYP}) recorded and 22,829 have at least one observed baseline field from \\texttt{GENDER} through \\texttt{CRUNT}, excluding \\texttt{AGE\\_NEW}, \\texttt{GENM} and \\texttt{GRM}; 1,362 have none under this criterion. The ratio $22732/24191\\approx 94.0\\%$ is a ratio of the two published counts. It is not an established response rate.\n\nIn a reconstruction on the public data, the network has all 24,191 records as vertices. The 1,362 records without an observed baseline answer under this criterion have no outgoing nomination links. Of these, 817 are nominated by other students, and 146 have a seed among their peers when a nomination in either direction counts (see \\fid{P037-N032}).", "problem": "The sentence says that every student took the survey and reports their number as 24,191. Applying the authors' sample-selection code to the public data gives 24,191 student records kept for the analysis. The rule that keeps them asks for a known school block and a roster number other than 999, and it does not ask for a baseline response. Table S1 defines school size as the number of students who took the survey before randomization, and its block means imply 22,732 such students, 1,459 fewer. SI p.~6 refers to ``students who failed to take the survey'', and SI p.~5 reports 24,191 as the total sample of the study. The count of the main text is not an established count of survey takers.\n\nThe original number of students who answered the baseline survey is not established. On the public release of the data (ICPSR 37070, version 2), the school-size variable of Table S1 sums to 22,732, and the codebook of the release defines it as the school population included in the study, not as a count of completed surveys. Two indicators of a baseline response in the public data count 22,810 and 22,829 records.", "what_survives": "No estimate is shown to change. The design of 56 schools in 14 blocks is not contradicted, and the Abstract's ``56 schools with 24,191 students'' need not count completed surveys. The public data agree with the published 24,191, with the 56 schools and with all fourteen school-size means of Table S1. The authors' own input files are not public, ICPSR reports changes to some variables, and the number of completed baseline surveys in the original files is not established. The code retains students without a baseline response in the network and includes their available outcomes in school-level means; network regressions have further inclusion criteria. The nominations that they would have made are not observed, and I did not estimate the effect of their absence on any result." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N003", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of six lines of text. A red box marks 'seed group' in 'controlling for the proportion of students in the seed group who were social referents'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), left column. The marked words should read “seed-eligible group”.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N005.png" } ], "formal_release": null, "formal_statements": [], "headline": "The control variable of the interaction regression is the proportion of social referents in the seed-eligible group, not in the seed group.", "id": "P037-N005", "location": "Methods and Materials, p. 568 (PDF p. 3), left column, paragraph beginning “The randomization of schools and seeds”", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\ldots\\ we use linear regression interacting school-level treatment with seed group composition and controlling for the proportion of students in the \\hl{seed group} who were social referents.", "corrected": "\\ldots\\ we use linear regression interacting school-level treatment with seed group composition and controlling for the proportion of students in the \\hl{seed-eligible group} who were social referents.", "evidence": "I checked the page against the supplement and against the authors' analysis scripts. Equations (2), (4) and (6) on SI pp.~13, 15 and 16 contain $\\beta C_{ij}D_{ij}$ and $\\gamma C^E_{ij}$, and the note to Table S7 (SI p.~42) says that a control is included for the expected value of the seed group composition.\n\nThe script \\texttt{1. Sanitize and generate perms, probs.R} (lines 143--146) computes two proportions of social referents for each school, one among students assigned to the seed group and one among seed-eligible students. The script \\texttt{4. School results-with-correction.R} (lines 140--141 and 203--209) interacts treatment with the first and uses the second as the control. Using an adapter with these scripts on the public data (ICPSR 37070, version 2), I confirmed the seed-eligible control in both regression specifications for the three direct outcomes; the two proportions differ in 43 of 56 schools.", "problem": "As printed, the sentence names the same variable twice: the proportion of social referents in the seed group is the seed group composition. The control in the regression is the proportion of social referents in the seed-eligible group, $C^E_{ij}$. The next sentence of the paragraph contrasts the two proportions.", "what_survives": "A word was dropped, and no estimate changes on account of it, since the released code uses the control that the supplement specifies. The authors' own input files are not public, ICPSR warns that some released variables were changed, and this check does not reproduce every estimate. On the public data, five of the six coefficients of Table S7 that I compared agree at the printed precision, and the interaction for peer conflict is $-0.84$ where the table prints $-0.85$. The regression's causal interpretation remains conditional on the response approximation stated in the SI Appendix. That model scope is separate from this omitted-word correction: restoring the control name neither verifies nor refutes the approximation." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N005", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of seven lines of text. Red boxes mark the sentence 'We computed confidence intervals and P values using robust SEs under a normal approximation.'", "caption": "Methods and Materials, p. 568 (PDF p. 3), right column. The marked sentence describes the school-level confidence intervals and $P$ values. It does not name the robust estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006.png" }, { "alt": "Crop of seven lines of text beginning 'We first turn to the question of school climate'. Red boxes mark 'significantly reduced' and '(P < 0.05, heteroskedasticity-robust Wald test)' in the sentence 'Figs. 1 and 2 reveal that the treatment significantly reduced average levels of disciplinary reports of peer conflict in treatment compared with control schools (P < 0.05, heteroskedasticity-robust Wald test).'", "caption": "Results, p. 569 (PDF p. 4), left column, first paragraph. The marked words report the main effect on disciplinary reports as significant at the 0.05 level.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-b.png" }, { "alt": "Crop of seven lines of the legend of Fig. 2. Red boxes mark '(P = 0.07, heteroskedasticity-robust Wald test)' in the sentence 'We find that a higher proportion of social referent seeds assigned to the anticonflict group increases the size of the overall school effect on disciplinary reports of peer conflict (P = 0.07, heteroskedasticity-robust Wald test).'", "caption": "Legend of Fig. 2, p. 569 (PDF p. 4). The marked words give the $P$ value of the interaction between treatment and the proportion of social referent seeds.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-c.png" }, { "alt": "Crop of Table S6, 'Average school-level climate effects caused by anti-conflict groups', with the columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict, and its note. The row School level main effect reads 0.09, minus 0.01, 0.16 and minus 0.06. Red boxes mark the entry (0.03) of the row Heteroskedasticity Robust SE and the entry (0.03) of the row Wild Bootstrap SE in the column Peer conflict. The note says that wild bootstrap standard errors were generated using 5,000 replicates, multiplying residuals by draws from the standard normal distribution.", "caption": "SI Appendix, Table S6, p. 41. The page is printed sideways and is shown here turned upright. The two marked standard errors of the peer-conflict column estimate the same conditional standard deviation: the wild bootstrap of the authors' script has conditional variance equal to the robust variance and adds no finite-sample adjustment. Its simulated standard error is subject to Monte Carlo variation.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-d.png" }, { "alt": "Dot plot with five rows and a horizontal axis from 0 to 18 percent; a dashed vertical line marks the nominal level of 5 percent. Each row has a blue circle for the main effect of treatment (Table S6) and an orange square for the interaction of treatment and referent share (Table S7). HC0 with a normal reference, the test of the article, under normal null errors: 13.7 and 16.0 percent. The same test under blocked re-randomization: 13.0 and 12.4 percent. HC1 with a t reference: 5.2 and 6.2 percent. HC2 with a t reference: 5.4 and 5.7 percent. HC3 with a normal reference: 2.0 and 1.9 percent. A note says 49 schools, 19 and 20 estimated coefficients, 100,000 draws of normal null errors and 30,000 draws of blocked re-randomization.", "caption": "Share of draws in which a true null hypothesis is rejected at the nominal 5% level, on the school-level design reconstructed from the public data (ICPSR 37070, version 2). The first two rows are the test of the article. The other rows are variants that are not established as calibrated for the hypotheses of the article. Values are properties of the two null experiments described under Evidence, not rejection probabilities for the authors' own files.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-rejection.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T18", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T18" }, { "id": "P037-T37", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T37" } ], "headline": "The HC0 normal-reference school tests reject a true null more often than their nominal 5% level in specified null experiments on the public-data design; a calibrated replacement for the original hypotheses remains unestablished.", "id": "P037-N006", "location": "Methods and Materials, p. 568 (PDF p. 3), right column, sentence beginning “We computed confidence intervals”; Results, p. 569 (PDF p. 4), first paragraph; legend of Fig. 2, p. 569; SI Appendix, Tables S6 and S7, pp. 41–42", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": [ "P037-T18", "P037-T37" ], "text": { "as_printed": "Main text, p.~568: We computed confidence intervals and $P$ values using \\hl{robust SEs under a normal approximation}.\n\nResults, p.~569: Figs. 1 and 2 reveal that the treatment \\hl{significantly reduced} average levels of disciplinary reports of peer conflict in treatment compared with control schools \\hl{($P < 0.05$, heteroskedasticity-robust Wald test)}.\n\nLegend of Fig.~2, p.~569: We find that a higher proportion of social referent seeds assigned to the anticonflict group increases the size of the overall school effect on disciplinary reports of peer conflict \\hl{($P = 0.07$, heteroskedasticity-robust Wald test)}.\n\nTable S6 (SI p.~41), column ``Peer conflict'': School level main effect $-0.06$; Heteroskedasticity Robust SE $(0.03)$; Wild Bootstrap SE $(0.03)$. Table S7 (SI p.~42), column ``Peer conflict'', interaction: $-0.85$; Heteroskedasticity Robust SE $(0.46)$; Wild Bootstrap SE $(0.47)$.", "corrected": "Main text, p.~568: Confidence intervals and $P$ values use \\hl{HC0 standard errors, without a finite-sample adjustment}, under a normal approximation. \\hl{In null experiments on the public-data design with 49 schools and 19 or 20 estimated coefficients, this test rejects a true null hypothesis more often than its nominal level.}\n\nNotes to Tables S6 and S7: \\hl{The heteroskedasticity-robust standard errors are HC0. The wild bootstrap multiplies the unscaled residuals by standard normal draws. Given the data, its variance equals HC0, and it adds no finite-sample adjustment.}\n\nResults, p.~569: Figs. 1 and 2 \\hl{show lower} average levels of disciplinary reports of peer conflict in treatment compared with control schools. \\hl{On the public data (ICPSR 37070, version 2) the estimate is $-0.057$ reports per student, with HC0 standard error $0.035$ and two-sided $P=0.099$ under a normal reference; it is not significant at the 0.05 level.}\n\nLegend of Fig.~2, p.~569: A higher proportion of social referent seeds assigned to the anticonflict group is associated with a larger estimated overall school effect on disciplinary reports of peer conflict (\\hl{on the public data, coefficient $-0.84$, HC0 standard error $0.46$, $P=0.069$ under a normal reference}).\n\nBoth $P$ values come from a test that, at the nominal 5\\% level, rejects a true null hypothesis in 12 to 16\\% of draws in null experiments on the public-data design. As sensitivity calculations on the public data, HC1 with a normal reference gives $P=0.20$ and $0.16$, HC2 with a $t$ reference gives $0.21$ and $0.19$, and a re-randomization of the schools within blocks, under the hypothesis of no effect on any school, gives $0.20$ and $0.15$. None of these is established as a calibrated test of a zero average effect or of a zero interaction. The standard errors reported with the two estimates above remain HC0; neither the additional rounding digit nor the sensitivity calculations establish improved sampling-uncertainty estimation. The values on the authors' own files cannot be given here.", "evidence": "I checked the authors' script, derived the covariance identities below, calculated the estimates with the script on the public release of the data (ICPSR 37070, version 2), and examined two null experiments on the school-level design reconstructed from that release.\n\n\\emph{The bootstrap reproduces HC0.} Let $X$ be the matrix of regressors, of full column rank, and write $A=(X^{\\top}X)^{-1}X^{\\top}$, $H=XA$ and $e=(I-H)y$ for the residuals. The bootstrap of the script regresses $X\\widehat b+\\operatorname{diag}(e)v$ on $X$, where the entries of $v$ are independent with mean 0 and variance 1. Its coefficient is $b^{*}=\\widehat b+A\\operatorname{diag}(e)v$, and hence\n\\[\n\\operatorname{Var}(b^{*}\\mid X,y)=A\\operatorname{diag}(e^{2})A^{\\top}=V_{\\mathrm{HC0}} .\n\\]\n\n\\emph{HC0 is too small in this design.} Suppose that $y=Xb+u$, where, conditional on $X$, the errors $u$ are independent with mean zero and common variance $\\sigma^2$, and let $h_{ii}$ be the $i$th diagonal entry of $H$. Then $\\mathbb{E}[e_i^2\\mid X]=\\sigma^2(1-h_{ii})$, so\n\\[\n\\mathbb{E}[V_{\\mathrm{HC0}}\\mid X]=\\sigma^{2}A\\operatorname{diag}(1-h)A^{\\top},\\qquad \\operatorname{Var}(\\widehat b\\mid X)=\\sigma^{2}AA^{\\top}.\n\\]\nFor the $j$th coefficient, the expected HC0 variance divided by the true variance is\n\\[\n\\frac{\\sum_i A_{ji}^{2}(1-h_{ii})}{\\sum_i A_{ji}^{2}} .\n\\]\nOn the public data this ratio is $0.619$ for the treatment coefficient of the regression of Table S6, where the mean of $h_{ii}$ is $19/49$, and $0.556$ for the interaction coefficient of the regression of Table S7, where it is $20/49$.\n\n\\emph{The estimates on the public data.} The public-data reconstruction applies the authors' rule for selecting records, builds the network and the outcome as their scripts do, and fits their two covariate-adjusted regressions. The rule selects 49 schools, 26 control and 23 treatment, as in the article (the rule is the subject of \\fid{P037-N007}). The main effect is $-0.0573$ and the interaction coefficient is $-0.841$. The script prints estimates, standard errors and intervals, and no $P$ value; I computed each two-sided $P$ value here from the estimate and its standard error.\n\n\\begin{audittable}{lrrrr}{Standard errors and two-sided $P$ values on the public data (ICPSR 37070, version 2), 49 schools, for the main effect of treatment on disciplinary reports (estimate $-0.0573$; Table S6 prints $-0.06$ with standard error $0.03$) and for the interaction with the proportion of social referent seeds (estimate $-0.841$; Table S7 prints $-0.85$ with standard error $0.46$). The first row is the test of the article. The other rows are sensitivity calculations, not corrected values. In the last row the schools are assigned again within blocks, 30,000 times, under the hypothesis of no effect on any school.}\nVariance and reference & SE, main & $P$, main & SE, interaction & $P$, interaction \\\\\n\\hline\nHC0, normal (the article) & $0.0347$ & $0.099$ & $0.463$ & $0.069$ \\\\\nHC1, normal & $0.0444$ & $0.197$ & $0.601$ & $0.162$ \\\\\nHC2, $t$ (30 and 29 d.f.) & $0.0451$ & $0.214$ & $0.621$ & $0.186$ \\\\\nHC3, normal & $0.0594$ & $0.335$ & $0.850$ & $0.322$ \\\\\nHC0 statistic, re-randomization & & $0.202$ & & $0.152$ \\\\\n\\end{audittable}\n\n\\emph{The null experiments.} The first experiment draws independent standard normal errors as the outcome, 100,000 times for each of the two fixed designs, so that every coefficient is truly zero. The test of the article rejected at the nominal 5\\% level in 13.7\\% of draws for the main effect and in 16.0\\% for the interaction (Monte Carlo standard errors of about 0.11 percentage points). The second experiment assigns two of the four schools of each block to treatment again, 30,000 times over the 56 schools, holding the outcomes, the covariates, the draws of the seeds and the set of 49 schools fixed, as they would be if treatment had no effect on any school. The test rejected in 13.0\\% and 12.4\\% of assignments (Monte Carlo standard errors of about 0.19 percentage points). The last row of the table estimates the share of assignments in which the HC0 statistic was at least as large in absolute value as the observed one, using $(1+\\text{count})/(30000+1)$.\n\nIn the first experiment the variants also miss the nominal level. HC1 with a $t$ reference rejects in 5.2\\% and 6.2\\% of draws, HC2 with a $t$ reference in 5.4\\% and 5.7\\%, and HC3 with a normal reference in 2.0\\% and 1.9\\%. The experiment has normal errors of equal variance, and it does not show how these variants behave for the randomization of the study. The re-randomization calculation is for the hypothesis of no effect on any school, with the outcomes, the covariates, the draws of the seeds and the set of 49 schools held fixed. That hypothesis is stronger than a zero average effect, and stronger than a zero interaction when treatment has some effect.\n\n\\emph{What the printed tables alone allow.} An effect of $-0.0649$ with HC0 standard error $0.0251$ rounds to the entries $-0.06$ and $0.03$ of Table S6. After the rescaling of the standard error by $\\sqrt{49/(49-k)}$ (HC1), its two-sided normal $P$ value is $0.043$ with $k=19$ coefficients and $0.047$ with $k=20$, the two numbers of coefficients on the public data. The rescaling alone does not remove significance for every pair of values that rounds to the printed entries. With the rescaling and a $t$ reference with $49-k$ degrees of freedom, every such pair has $P>0.05$; the smallest values are $0.051$ and $0.055$.\n\n\\emph{Influence-score terminology and current use.} The raw empirical OLS influence covariance is exactly HC0. With $n$ schools, design row $x_i^{\\top}$ and $\\widehat\\phi_i=(X^{\\top}X/n)^{-1}x_i e_i=nA_{\\cdot i}e_i$,\n\\[\n\\frac{1}{n^2}\\sum_i\\widehat\\phi_i\\widehat\\phi_i^{\\top}=A\\operatorname{diag}(e^2)A^{\\top}=V_{\\mathrm{HC0}}.\n\\]\nAn identified current DoubleML implementation also uses an unadjusted empirical influence-score second moment: its nonclustered variance helper computes $\\operatorname{mean}(\\psi_i^2)/(N\\operatorname{mean}(\\psi'_i)^2)$, and its official variance guide uses normal inference under regularity conditions. The sources are the \\href{https://docs.doubleml.org/stable/guide/se_confint.html}{captured stable guide} and \\href{https://github.com/DoubleML/doubleml-for-py/blob/6884b5f9816f180324d854a4d81b22e9d334f5e0/doubleml/utils/_estimation.py}{the pinned Python source}. They establish documented continued use, without measuring its prevalence or identifying the stable documentation with a particular numeric package release. Cross-fitted DoubleML scores generally differ from these in-sample OLS residuals; their shared second-moment form transfers no finite-sample calibration to this school design.\n\nThe corrected passage below retains HC0. Reporting its public-data main-effect standard error as $0.035$ rather than $0.03$ gives an extra rounding digit; it does not provide a more accurate estimator of sampling uncertainty.\n\n\\emph{Limits.} The values above are values on the public data. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. On the public data the coefficient and standard error of Table S6 agree with the printed $-0.06$ and $0.03$, and the standard error and $P$ value of the interaction round to the printed $0.46$ and $0.07$. The interaction coefficient is $-0.841$, which rounds to $-0.84$ where Table S7 prints $-0.85$. The rejection rates are properties of the two experiments on the public-data design. They are not the rejection probability of the study under its own unknown outcomes, and the coverage of the bands of Fig.~2 was not examined.", "problem": "The article says that the school-level confidence intervals and $P$ values use ``robust SEs under a normal approximation'', and it does not name the estimator. The authors' script \\texttt{4. School results-with-correction.R} (lines 162--163 and 208--209) uses the heteroskedasticity-robust variance with no finite-sample adjustment, known as HC0. The wild bootstrap of Tables S6 and S7 (lines 178--195 and 228--248) multiplies the unscaled residuals by standard normal draws. Given the data, its variance is exactly HC0. The row ``Wild Bootstrap SE'' is therefore not a second check on the row ``Heteroskedasticity Robust SE''.\n\nThe regressions for disciplinary reports have 49 schools and many coefficients: block effects for 14 blocks, treatment, the composition terms and covariates. On the public data the regression of Table S6 has 19 estimated coefficients and that of Table S7 has 20, which leaves 30 and 29 residual degrees of freedom. Under a linear model with independent mean-zero errors of equal variance, fitting shrinks the residuals: their expected mean square is $30/49$ and $29/49$ of the error variance, whereas the expected HC0 variance is 0.62 of the variance of the treatment coefficient and 0.56 of the variance of the interaction coefficient. A normal reference does not correct for this, and the test rejects too often in the null experiments below.\n\nBoth results on disciplinary reports rest on this test: the main effect ($P<0.05$) and the interaction with the proportion of social referent seeds ($P=0.07$). In null experiments on the design reconstructed from the public data, the test rejects a true null hypothesis at the nominal 5\\% level in 12 to 16\\% of draws. The two printed $P$ values are weaker evidence than the same values from a calibrated test. On the public data, the test of the article gives $P=0.099$ for the main effect and $P=0.069$ for the interaction, and about $0.20$ and $0.15$ when the schools are re-randomized under the hypothesis of no effect on any school.\n\nThe description in the Methods is accurate as far as it goes, and it claims no exact size. Whether the printed $P$ values are the values of this test on the authors' own files remains unknown, because those files are not public. No replacement test is established as calibrated for the two hypotheses of the article.", "what_survives": "If the variance estimator or the reference distribution is changed, the least-squares estimates, their direction, the assignment of schools and the measurement of the networks do not change. The analysis of peer-to-peer influence computes its variance from simulated assignments and does not use this test. The correction of 30\\% to 25\\% is a separate matter (\\fid{P037-N001}).\n\nThe evidence for the two results on disciplinary reports is weaker than printed. On the public data the main effect is not significant at the 0.05 level under the test of the article ($P=0.099$), in agreement with the last sentence of the correction of 2018; the disagreement between that sentence and the article's $P<0.05$ is the subject of \\fid{P037-N008}. The interaction keeps $P=0.07$ under a test that rejects a true null hypothesis in 12 to 16\\% of draws. The treatment bands of Fig.~2 use the same robust variance (their construction is the subject of \\fid{P037-N026}). No printed claim is shown to be false on the authors' own files. No test here is established as calibrated for a zero average effect and for a zero interaction in this design; the variants in the table weaken the evidence and are not such a test." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N006", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the first nine lines of the legend of Fig. 1, 'Overall school climate results: distribution of disciplinary events throughout school networks, comparing treatment, and control schools.' A red box marks the words 'the 49 schools that provided administrative data (26 in control, 23 in treatment)', which run over two lines of the sentence 'Visualization of the effect of treatment on disciplinary reports of peer conflict among the 49 schools that provided administrative data'.", "caption": "Legend of Fig. 1, p. 568 (PDF p. 3). The marked words describe the 49 schools of the conflict analyses as the schools that provided administrative data, 26 in control and 23 in treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N007.png" }, { "alt": "Crop of six lines of the Methods and Materials. A red box marks the parenthesis '(Administrative data were available for 49 of the 56 schools, and we restricted our attention to these schools for analyses of conflict-related events.)', which runs over three lines. The text before it says that behavior was tracked with the schools' administrative records on peer conflict-related disciplinary events across the entire year.", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked parenthesis says that administrative data were available for 49 of the 56 schools and that the analyses of conflict-related events are restricted to them.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N007-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T19", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T19" }, { "id": "P037-T20", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T20" }, { "id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22" }, { "id": "P037-T38", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T38" } ], "headline": "The authors' code selects the 49 schools of the conflict analyses by a rule on the outcome (a school mean of the peer-conflict count above $0.02$), not by whether a school provided administrative data, which is the article’s description; on the public data 52 schools have disciplinary records, and the three further schools that the rule removes are treatment schools.", "id": "P037-N007", "location": "Legend of Fig. 1, p. 568 (PDF p. 3); Methods and Materials, p. 567 (PDF p. 2), right column; SI Appendix, p. 10", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T19", "P037-T20", "P037-T22", "P037-T38" ], "text": { "as_printed": "Fig.~1, legend: ``Visualization of the effect of treatment on disciplinary reports of peer conflict among \\hl{the 49 schools that provided administrative data} (26 in control, 23 in treatment).''\n\nMethods and Materials, p.~567: ``(\\hl{Administrative data were available for 49 of the 56 schools}, and we restricted our attention to these schools for analyses of conflict-related events.)''", "corrected": "Fig.~1, legend: ``Visualization of the effect of treatment on disciplinary reports of peer conflict among \\hl{the 49 schools whose mean peer-conflict count per student exceeded 0.02} (26 in control, 23 in treatment).''\n\nMethods and Materials, p.~567: ``(\\hl{Each student's peer-conflict count is the sum of 17 disciplinary category counts, with a missing category counted as zero. We restricted the analyses of conflict-related events to the 49 of the 56 schools whose mean count per student exceeded 0.02.})''\n\nOn the public data this rule selects 49 schools (26 control, 23 treatment), and 52 schools (26 in each arm) have recorded disciplinary fields; the three further schools removed are treatment schools with positive counts. The sentence of the SI Appendix (p.~10) that seven schools provided no data needs evidence of what each school supplied. Of the seven schools removed by the code on the public data, four have no disciplinary entry and three have recorded fields.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data. In the cleaning script (\\texttt{2. Cleaning.R}, lines 35--38), the count is the sum of the 17 categories after missing values are set to zero; the list of categories is the subject of \\fid{P037-N018}. The school-level script (\\texttt{4. School results-with-correction.R}, lines 151--153) keeps the schools whose mean count exceeds $0.02$, and it fits the regressions of Tables S6 and S7 on them. The network script (\\texttt{3. Network results.R}, lines 167--173) applies the same rule. The script for the check within treatment schools (Fig.~S5) removes the schools with mean below $0.02$, which selects the same schools here, since no school mean equals $0.02$. The rule is not applied to the survey outcomes.\n\nI ran these lines on the public data, after converting its variable names and codes to the form that the scripts read. The authors' filter on the records keeps 24,191 students in 56 schools, as published.\n\n\\begin{audittable}{lrr}{Schools of the public data in each arm, and the sum of the peer-conflict count over their students.}\n & Control & Treatment \\\\\nSchools randomized & 28 & 28 \\\\\nSchools with recorded disciplinary fields & 26 & 26 \\\\\nSchools with no disciplinary entry & 2 & 2 \\\\\nSchools with mean count above $0.02$ (kept by the code) & 26 & 23 \\\\\nSchools with recorded fields that the rule removes & 0 & 3 \\\\\nSum of the count, all schools & 2,213 & 1,862 \\\\\nSum of the count, schools kept & 2,213 & 1,849 \\\\\n\\end{audittable}\n\nThe counts of the three treatment schools that have records and fail the rule sum to 13. Two of these schools have ordinary amounts of other disciplinary records: $0.51$ and $0.41$ per student, which exceed the rates of 32 and of 28 of the 49 retained schools. Nearly all of these records (245 of 255, and all 228) are generic rule violations, and the third school has one other record. These totals do not show whether the three schools reported peer conflict completely. The public data also do not show which seven schools originally provided no data.\n\nI then fitted the covariate-adjusted regressions of Tables S6 and S7 on the public data twice, changing only the schools included.\n\n\\begin{audittable}{lrrrrrr}{Covariate-adjusted school regressions on the public data under the two rules of inclusion: the main effect of treatment (the model of Table S6) and the interaction of treatment with the proportion of social referents (the model of Table S7), each with its heteroskedasticity-robust (HC0) standard error and two-sided normal $P$ value.}\nSchools included & Main effect & SE & $P$ & Interaction & SE & $P$ \\\\\n49 schools (mean count above $0.02$) & $-0.0573$ & $0.0347$ & $0.099$ & $-0.841$ & $0.463$ & $0.069$ \\\\\n52 schools (recorded disciplinary fields) & $-0.0713$ & $0.0323$ & $0.027$ & $-0.469$ & $0.417$ & $0.260$ \\\\\n\\end{audittable}\n\nThese $P$ values use the article's HC0 normal test. In null experiments on the 49-school public-data design, that test rejects a true null too often (\\fid{P037-N006}). The row for 52 schools is a check of sensitivity and not a corrected estimate.\n\nA constructed example shows that the further condition does not follow from randomization, even when every record is complete. Suppose that a block of four schools has control means $0$, $0.03$, $0.06$ and $0.09$ per student and a constant effect of $+0.03$, and that two of the four schools are treated at random. The first school passes the rule only when it is treated. Over the six assignments, the difference in means between the retained treatment and control schools is $-0.03$, $0$, $0.03$, $-0.015$, $0.03$ and $0.075$. The average is $0.015$, half the effect.\n\nThe public data reproduce the published 24,191 students, the 49 schools and their division into 26 and 23, and the $-0.06$ and $0.03$ of Table S6. For the interaction of Table S7, I find $-0.84$ against the printed $-0.85$, with the printed standard error $0.46$. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The values above are values for the public data, and not the values that the authors would obtain from their own files.", "problem": "The article says that the analyses of disciplinary reports of peer conflict (the conflict analyses) use the 49 schools for which administrative data were available, and the SI Appendix (p.~10) says that seven of the 56 schools provided no data. The authors' code does not select schools in this way. It forms each student's peer-conflict count as the sum of 17 disciplinary categories, with a missing category counted as zero, and it keeps a school when the mean of this count over the school's students is greater than $0.02$. This rule is a condition on the outcome. On the public data (ICPSR 37070 V2) the rule selects 49 schools, 26 control and 23 treatment, and the printed count is not shown to be wrong. The same data have disciplinary records for 52 schools, 26 in each arm. Four schools, two in each arm, have no disciplinary entry. The three further schools that the rule removes are all treatment schools. Each has a recorded count for every student, and their school means, $0.0018$, $0.0145$ and $0.0199$, are positive and below $0.02$. On the public data, ``provided administrative data'' does not describe the selection.\n\nThere are two consequences. First, a rule that keeps a school only when its outcome is large enough can depend on the assignment, since the treatment may change the count. A sufficient additional condition for interpreting the conflict estimates causally is that school inclusion and reporting remain unchanged across assignments, together with the paper's model conditions; an analysis that accounts for selection could also justify that interpretation. The observed data cannot check inclusion under other assignments. Second, the estimates depend on the rule. When the three schools are admitted on the public data, the main effect moves from $-0.057$ to $-0.071$, and the interaction with the proportion of social referents moves from $-0.84$ ($P = 0.07$) to $-0.47$ ($P = 0.26$).", "what_survives": "The number 49 and its division into 26 control and 23 treatment schools are recovered by the authors' rule on the public data. The assignment of the 56 schools is unaffected, the rule is not applied to the survey outcomes (talking about conflict, norms, wristbands), and the main effect on disciplinary reports is negative under both rules. No published estimate is shown to be wrong. What changes is the description of the sample in every analysis of disciplinary reports (Figs.~1, 2D and 3D, the peer-conflict columns of Tables S5--S7, and the checks of the SI Appendix that use the same rule). Interpreting the conflict estimates causally needs the additional justification stated above. Whether seven schools originally provided no data, and whether that failure was unrelated to treatment as the SI Appendix says, cannot be decided without the original records of the schools." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N007", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the opening of the Results section. Red boxes mark the words 'significantly reduced' and the parenthesis '(P < 0.05, heteroskedasticity-robust Wald test)' in the sentence 'Figs. 1 and 2 reveal that the treatment significantly reduced average levels of disciplinary reports of peer conflict in treatment compared with control schools'. The next sentences give 0.20 disciplinary events per student in control schools and an average decrease of 0.06.", "caption": "Results, first paragraph, p. 569 (PDF p. 4). The marked words report the overall reduction in disciplinary reports as significant, with $P < 0.05$ from a heteroskedasticity-robust Wald test.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N008.png" }, { "alt": "Crop of the last five lines of the published correction. After the instruction to replace the number 30 with 25, the final sentence reads 'The estimated overall effects on administrative reports of conflict remain statistically insignificant at the alpha=0.05 level.' Two red boxes mark the words 'remain statistically insignificant at the alpha=0.05 level'.", "caption": "2018 correction, p. E3598 (PDF p. 1), last sentence. The marked words say that the overall effects on administrative reports of conflict remain insignificant at the $0.05$ level.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N008-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T3", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T3" }, { "id": "P037-T15", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T15" } ], "headline": "The article reports the overall reduction in disciplinary reports as significant ($P < 0.05$), the 2018 correction says that the same effect remains insignificant at $0.05$, and the sentence of the Results was never withdrawn; a conventional two-sided Wald test on the public data gives $P = 0.099$, supporting nonsignificance for that reconstruction.", "id": "P037-N008", "location": "Results, first paragraph, p. 569 (PDF p. 4); 2018 correction, p. E3598 (PDF p. 1), last sentence", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T15", "P037-T3" ], "text": { "as_printed": "Article, p.~569: ``Figs. 1 and 2 reveal that the treatment \\hl{significantly reduced} average levels of disciplinary reports of peer conflict in treatment compared with control schools ($\\hl{P < 0.05}$, heteroskedasticity-robust Wald test).''\n\n2018 correction, p.~E3598: ``The estimated overall effects on administrative reports of conflict \\hl{remain} statistically insignificant at the $\\alpha=0.05$ level.''", "corrected": "Article, p.~569: ``Figs. 1 and 2 \\hl{show lower} average levels of disciplinary reports of peer conflict in treatment compared with control schools\\hl{; a reconstruction on the public data gives a reduction that is not statistically significant at the 0.05 level} (\\hl{two-sided} heteroskedasticity-robust Wald test).'' On the public data the estimate is $-0.057$ reports per student, with standard error $0.035$ and two-sided $P = 0.099$.\n\n2018 correction, last sentence: ``The estimated overall effects on administrative reports of conflict \\hl{are} statistically insignificant at the $\\alpha=0.05$ level \\hl{in the two-sided analysis reconstructed on the public data; this replaces the statement of significance ($P < 0.05$) in the Results of the article}.''\n\nIf the $P < 0.05$ of the article came from another test, for instance a one-sided test, the test and its inputs should be stated.", "evidence": "The evidence is a check of the pages, a reading of the authors' code and a recalculation on the public data. Both quotations are as printed, and the correction specifies no replacement test. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 162--163 regress the school mean of the peer-conflict count on treatment, block fixed effects, the proportion of social referents among the seed-eligible students and three school covariates, with HC0 standard errors. The script prints no $P$ value. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read.\n\n\\begin{audittable}{lrrrrr}{The overall effects of Table S6 and their heteroskedasticity-robust standard errors (SE), as printed and from the authors' code on the public data.}\nOutcome & Printed & SE & Public data & SE & Schools \\\\\nFriends talk about conflict & $0.09$ & $0.02$ & $0.0874$ & $0.0176$ & 56 \\\\\nPrescriptive norms & $-0.01$ & $0.08$ & $-0.0142$ & $0.0781$ & 56 \\\\\nWristband & $0.16$ & $0.01$ & $0.1647$ & $0.0114$ & 56 \\\\\nPeer conflict & $-0.06$ & $0.03$ & $-0.0573$ & $0.0347$ & 49 \\\\\n\\end{audittable}\n\nFor peer conflict on the public data, I obtain\n\\[\nz \\simeq \\frac{-0.0572720}{0.0347214} \\simeq -1.649, \\qquad P = 2\\bigl\\{1 - \\Phi(|z|)\\bigr\\} \\simeq 0.099,\n\\]\nand the 95\\% interval is $[-0.125, 0.011]$. A separate calculation of the same regression by matrix algebra agrees to ten decimals. The code uses another fit for the corrected percentage, the interaction model averaged over the 49 schools; on the public data its contrast is $-0.0566$ with standard error $0.0321$ and $P = 0.078$. A one-sided $P$ value for the public-data coefficient of Table S6 would be $0.0495$, but neither the article nor the code specifies a one-sided test.\n\nThe rounded entries of Table S6 do not decide the question by themselves: an estimate of $-0.057$ with standard error $0.028$ or $0.034$ prints as $-0.06$ and $0.03$, and its $P$ value is $0.042$ or $0.094$.\n\nThe public data reproduce all eight entries of the table above at the printed precision, and the published 24,191 students, 56 schools and 49 schools of the conflict analyses. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The $P$ value of $0.099$ is the value for the public data, and the $P$ value of the original run is not recovered.", "problem": "Both sentences describe the overall school-level effect of the intervention on disciplinary reports of peer conflict. The article reports this effect as a significant reduction, with $P < 0.05$ from a heteroskedasticity-robust Wald test. The correction changes the size of the reduction from 30\\% to 25\\% and ends by saying that the overall effects remain insignificant at the $0.05$ level. It names no new test, and the four passages that it lists for amendment do not include the sentence that reports significance. For one test, the two statements cannot both hold. The word ``remain'' is also inaccurate, since the article did not report the overall effect as insignificant. The corrected publication still carries ``significantly reduced ($P < 0.05$)'' in the Results beside a correction that says the opposite, and the abstract, the Significance statement, the Discussion and the legend of Fig.~2 can still be read as establishing the overall reduction at the $0.05$ level.\n\nThe authors' code identifies the analysis; its reconstruction on the public data (ICPSR 37070 V2) supports nonsignificance under a conventional two-sided Wald test. The code fits the regression of Table S6 on the 49 schools of the conflict analyses and prints the coefficient of treatment with its heteroskedasticity-robust (HC0) standard error. On the public data the estimate is $-0.0573$ reports per student with standard error $0.0347$, and the two-sided normal Wald $P$ value is $0.099$. This supports the correction's nonsignificance statement for that reconstruction.", "what_survives": "The point estimate ($-0.06$ reports per student, standard error $0.03$) is reproduced on the public data, and so is the corrected 25\\% ($25.4\\%$). Nothing here bears on the randomization, the intervention or the data. What changes is the sentence of the Results; readings of the abstract, the Significance statement, the Discussion and the legend of Fig.~2 as establishing the overall reduction at the $0.05$ level need the same qualification for the public-data analysis. Whether this test is reliable with 49 schools is the subject of \\fid{P037-N006}. The results on the other school-climate outcomes, on the composition of the seed group and on peer-to-peer influence rest on separate estimates and tests." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N008", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of five lines of the first paragraph of the Results: 'a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a 30 percent reduction in peer conflict reports.' A red box marks the number 0.20.", "caption": "Results, first paragraph, p. 569 (PDF p. 4). The marked number is the mean number of disciplinary events per student that the article reports for control schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N009.png" }, { "alt": "Crop of six lines of the published correction, from 'mated average treatment effect (-0.06) divided by the rounded unadjusted control group mean (0.20)' to 'The original estimate of 30 percent was reported in the Abstract (line 11), the Significance'. A red box marks the sentence 'Using both unadjusted estimates, we have an estimate of 23 percent.', which runs over two lines and follows 'Using both covariate adjusted estimates, we have an estimate of 25 percent.'", "caption": "2018 correction, p. E3598 (PDF p. 1). The marked sentence gives 23% as the reduction computed from both unadjusted estimates.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N009-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The control mean of $0.20$ in the Results and the “unadjusted” 23% of the 2018 correction are computed in the authors' code over all 56 schools, including the seven schools outside the conflict analyses, whose counts are zero or nearly zero; on the 49-school sample selected by the code, the public data give raw means of $0.21$ and $0.18$ and a raw reduction of 13.6%.", "id": "P037-N009", "location": "Results, first paragraph, p. 569 (PDF p. 4); 2018 correction, p. E3598 (PDF p. 1)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": { "as_printed": "Article, p.~569: ``In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average \\hl{0.20} times per year.''\n\n2018 correction, p.~E3598: ``Using both covariate adjusted estimates, we have an estimate of 25\\%. Using both \\hl{unadjusted estimates}, we have an estimate of \\hl{23\\%}.''", "corrected": "Article, p.~569: ``In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year \\hl{(the unadjusted mean over all 28 control schools, after missing disciplinary fields were coded as zero)}.''\n\n2018 correction: ``Using both unadjusted estimates \\hl{over all 56 schools, including the seven excluded from the conflict analyses, after missing disciplinary fields were coded as zero,} we have an estimate of 23\\%.''\n\nOn the 49-school sample selected by the code, I obtain raw control and treatment means of $0.2124$ and $0.1836$ on the public data, weighting schools equally. These round to $0.21$ and $0.18$ and imply a raw reduction of 13.6\\%, or 14\\% to the nearest whole percent. These are a different comparison from the printed $0.20$ and 23\\%. The adjusted 25\\% uses interaction-model predictions over the 49 schools, not the raw 56-school control mean.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 151--153 select schools whose mean exceeds $0.02$ (\\fid{P037-N007}), yielding 49 schools on the public data, and the fits on lines 162 and 208 use them. Lines 264, 273 and 289 compute the raw means and the raw percentage without the rule. Lines 253--256 and 286 compute the adjusted percentage from the interaction model, by averaging its predictions over the 49 schools with treatment set to zero and to one. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read. Each mean weights the schools equally.\n\n\\begin{audittable}{lrrr}{Mean peer-conflict count per student in control and treatment schools on the public data: raw means over all 56 schools, raw means over the 49 schools of the analyses, and the adjusted means that the code uses for the corrected 25\\%.}\n & Raw, 56 schools & Raw, 49 schools & Adjusted, 49 schools \\\\\nControl schools & 28 & 26 & 26 \\\\\nTreatment schools & 28 & 23 & 23 \\\\\nControl mean & $0.1972$ & $0.2124$ & $0.2225$ \\\\\nTreatment mean & $0.1521$ & $0.1836$ & $0.1660$ \\\\\nDifference & $-0.0451$ & $-0.0288$ & $-0.0566$ \\\\\nReduction & 22.9\\% & 13.6\\% & 25.4\\% \\\\\n\\end{audittable}\n\nOn the public data, the model of Table S6 has fitted rates $0.2258$ and $0.1685$. Multiplying them by the 11,938 students across all 28 treatment schools, I obtain the totals 2,695 and 2,012 of the Discussion after rounding. These totals are fitted values and not raw means. The public data reproduce the published 24,191 students, the 26 control and 23 treatment schools, the $-0.06$ of Table S6 and the rounded $0.20$, 29\\%, 25\\% and 23\\%; the interaction is $-0.84$ against the printed $-0.85$. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The decimals above are values for the public data; the exact historical values and excluded-school membership cannot be established without the authors' original files.", "problem": "The article restricts the analyses of disciplinary reports to 49 schools (p.~567); the authors' code selects 49 schools on the public data by a rule on the school mean. The lines of the code that compute the raw means of the two arms do not apply the restriction. They average over all 56 schools. On the public data (ICPSR 37070 V2), missing disciplinary fields are coded as zero: four of the seven excluded schools have no recorded fields for peer conflict, and three have positive counts. The two excluded control schools have no recorded event, and the five excluded treatment schools have 13 events in all. Including them lowers the treatment mean more than the control mean. On the public data the means over the 56 schools are $0.197$ and $0.152$, a reduction of 22.9\\%, which reproduces the printed $0.20$ and 23\\%. Over the 49 schools the means are $0.212$ and $0.184$, a reduction of 13.6\\%. The correction places the 23\\% beside the adjusted 25\\% as the same comparison without adjustment. On the 49-school sample selected by the code in the public data, the raw reduction is about half of the adjusted one.", "what_survives": "The adjusted estimate and the corrected 25\\% are reproduced at published precision on the public data, as are the 29\\% of the correction and the totals of the Discussion. No defect in the regressions follows from this sample mismatch. What changes is the interpretation of the raw control mean of the Results and the raw percentage of the correction: the code computes them over all 56 schools, rather than its 49-school conflict-analysis sample." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N009", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the end of the second paragraph of the Results. In the sentence 'When 20 percent of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37 percent), we estimate that each student was disciplined on average 0.08 times during the year, a reduction of 60 percent compared with control schools and twice the average effect of the anticonflict groups', a red box marks the number 0.08.", "caption": "Results, second paragraph, p. 569 (PDF p. 4). The marked number is the level of disciplinary reports per student that the sentence reports at 20% social referents.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N010.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T13", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T13" } ], "headline": "At 20% social referents the fitted model predicts about $0.095$ disciplinary reports per student, not $0.08$.", "id": "P037-N010", "location": "Results, second paragraph, p. 569 (PDF p. 4), sentence beginning “When 20%”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T13" ], "text": { "as_printed": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37\\%), we estimate that each student was disciplined on average \\hl{0.08} times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.", "corrected": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds \\ldots, we estimate that each student was disciplined on average \\hl{about 0.095} times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.\n\nThis is the smallest change; to two decimals the level is $0.10$. A fuller statement would report the fitted control level on the public data, about $0.223$, a reduction of about 57\\%, and an effect of roughly $2.2$ times the average decrease in disciplinary events per student. The parenthesis on the range of the proportions is the subject of \\fid{P037-N027}. In the first paragraph of the Results, label $0.20$ as the unadjusted control mean over all 28 control schools and state that the corrected 25\\% uses an adjusted control level of about $0.2225$ from the interaction regression on the public data; see \\fid{P037-N001} and \\fid{P037-N009}.", "evidence": "The evidence is a recalculation on the public data (ICPSR 37070 V2) with the authors' code through an adapter for that release, and a recalculation from the published table and figure. In the school-level script (\\texttt{4. School results-with-correction.R}), line 208 fits the interaction model on the 49 schools of the conflict analyses, lines 253--255 average its predictions for control, and lines 311--312 and 332--333 compute the plotted level in treatment schools. On the public data I obtain $\\widehat{\\tau}_0 \\approx 0.04083$, $\\widehat{\\beta} \\approx -0.84134$ and a fitted control level of about $0.22254$, and the level at $c = 0.20$ is\n\\[\n0.22254 + 0.04083 + 0.20 \\times (-0.84134) \\approx 0.0951 .\n\\]\nThe difference from control is $-0.1274$, and the reduction is $0.1274/0.2225 \\approx 57.3\\%$. The main effect of Table S6 on the public data is a decrease of $0.0573$, and the decrease at $c = 0.20$ is $2.2$ times as large.\n\nThe published figure agrees. In the vector drawing of Fig.~2D, the control line is at about $0.2225$ and the treatment line is approximately $0.2634 - 0.8414\\,c$, giving about $0.0951$ at $c = 0.20$; before rounding, the two calculations differ by less than $0.000002$ in level.\n\nRounding in Table S7 does not close the gap. By SI equations (4) and (6), the difference between the treatment and control predictions at proportion $c$ is $\\widehat{\\tau}_0 + \\widehat{\\beta} c$, and Table S7 prints $\\widehat{\\tau}_0 = 0.04$ and $\\widehat{\\beta} = -0.85$ for peer conflict. If $\\widehat{\\tau}_0 \\in [0.035, 0.045]$ and $\\widehat{\\beta} \\in [-0.855, -0.845]$, then the effect at $c = 0.20$ lies in $[-0.136, -0.124]$. Since the control level lies in $[0.222, 0.223]$, the treatment level lies in $[0.086, 0.099]$. A number printed as $0.08$ lies in $[0.075, 0.085]$, and the two intervals are disjoint.\n\nThe values from the code are values for the public data. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The agreement with Fig.~2D holds at the precision of the drawing. On the public data the code does not show how the printed $0.08$ arose.", "problem": "The sentence reports a prediction of the fitted model of Fig.~2D and Table S7 of the SI Appendix, at a proportion $c = 0.20$ of social referents in the seed group. The predicted level in treatment schools is the fitted control level plus the treatment effect $\\widehat{\\tau}_0 + \\widehat{\\beta} c$. From the line of Fig.~2D and from the authors' code on the public data, I find a level of about $0.095$, and no values that round to the entries of Table S7 correspond to a level that rounds to $0.08$. The other two numbers of the sentence hold as approximations: by the same calculation, the reduction is about 57\\% and the effect is about $2.2$ times the average decrease.", "what_survives": "The fitted effect at 20\\% referents (about $-0.13$ reports per student), the lines of Fig.~2D, the projected totals of the Discussion and the corrected 25\\% are unchanged, and the effect at 20\\% referents is about twice the average effect, as stated. Only the printed level changes. The slope of Fig.~2D differs from that of Table S7 in the second decimal (\\fid{P037-N011}); with either, the level lies between $0.086$ and $0.099$." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N010", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Fig. 2, four panels with its caption at the right. Each panel plots a horizontal red line for control schools and a sloped black line for treatment schools against the proportion of social referent seeds in the anti-conflict group, with ticks from 0.05 to 0.20, each with dashed curves labeled as 95 percent confidence intervals. Red boxes mark panel B, 'Perceived social norm against conflict', and panel D, 'Peer conflict (disciplinary records)', where the black line falls from about 0.24 to about 0.09. A third red box marks the words 'Table S7' at the end of the caption.", "caption": "Fig. 2 and its caption, p. 569 (PDF p. 4). The marked panels, B (perceived social norm against conflict) and D (disciplinary records of peer conflict), are the two whose lines do not match Table S7, to which the caption refers.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N011.png" }, { "alt": "Crop of Table S7, 'Average school-level climate effects caused by anti-conflict groups, interacted with proportion of social referent seeds in the group', with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict. Red boxes mark, in the Prescriptive norms column, the school effect with 0 percent social referents, 0.05, and the interaction -0.53 with its robust standard error (1.06); and, in the Peer conflict column, the interaction -0.85. A fourth red box marks the sentence of the note 'A graphical version of this table is presented in Figure 2'.", "caption": "SI Appendix, Table S7, p. 42 (supplement PDF p. 42), shown upright; the table is printed sideways on the page. The marked entries are the four that differ from Fig. 2 and from the authors' code on the public data, and the marked sentence of the note calls Fig. 2 the graphical version of the table.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N011-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T13", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T13" } ], "headline": "Four entries of Table S7 do not match Fig. 2, which the SI Appendix presents as the graphical version of the table; the authors' code on the public data reproduces the fitted lines and treatment bands and gives $0.04$, $-0.51$ and $1.07$ where the table prints $0.05$, $-0.53$ and $1.06$ for prescriptive norms, and $-0.84$ where it prints $-0.85$ for peer conflict.", "id": "P037-N011", "location": "Fig. 2, panels B and D, and its caption, p. 569 (PDF p. 4); SI Appendix, Table S7 and its note, p. 42 (supplement PDF p. 42)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T13" ], "text": { "as_printed": "Fig.~2, caption: ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of social referent seeds, and are surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit. \\ldots\\ See SI Appendix, \\hl{Table S7} for regression tables.''\n\n\\begin{audittable}{lrr}{Table S7 (SI p.~42), two of its four columns and the rows for the estimates and their heteroskedasticity-robust standard errors.}\n & Prescriptive norms & Peer conflict \\\\\nSchool effect with 0\\% Social Referents & \\hl{0.05} & 0.04 \\\\\nHeteroskedasticity Robust SE & (0.15) & (0.07) \\\\\nInteraction: Proportion of Referents in Treatment group & \\hl{-0.53} & \\hl{-0.85} \\\\\nHeteroskedasticity Robust SE & \\hl{(1.06)} & (0.46) \\\\\n\\end{audittable}\n\nTable S7, note: ``\\hl{A graphical version of this table is presented in Figure 2} in the main text of the paper.''", "corrected": "\\begin{audittable}{lrr}{Table S7, the same rows and columns, with the values from the authors' code on the public data.}\n & Prescriptive norms & Peer conflict \\\\\nSchool effect with 0\\% Social Referents & \\hl{0.04} & 0.04 \\\\\nHeteroskedasticity Robust SE & (0.15) & (0.07) \\\\\nInteraction: Proportion of Referents in Treatment group & \\hl{-0.51} & \\hl{-0.84} \\\\\nHeteroskedasticity Robust SE & \\hl{(1.07)} & (0.46) \\\\\n\\end{audittable}\n\nThese four values agree with the corresponding estimates recovered from Fig.~2 as published. They are values for the public data and should be checked against the authors' own data before they are printed as the entries of the table.\n\nUse the same interaction fit for the table and figure, retaining the 49-school restriction for every peer-conflict covariance calculation on the public data. To retain the predicted-mean labels, calculate each band's variance as $\\hl{a_g(c)^\\top V a_g(c)}$, where $V$ is that fit's full heteroskedasticity-robust covariance matrix and $a_g(c)$ is the prediction design vector averaged over the included schools for group $g$ at proportion $c$. The pointwise interval is the predicted mean plus or minus $1.96\\sqrt{a_g(c)^\\top V a_g(c)}$. The bands of Fig.~2 are also addressed in \\fid{P037-N026}.", "evidence": "The evidence is a recalculation from the published PDF and a recalculation on the public data with the authors' code. In each panel I read the lines and the band around the treatment line from the vector drawing and calibrated them on the axis ticks. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 208--209 fit the model of Table S7, on 56 schools for the survey outcomes and on 49 schools for peer conflict. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read.\n\n\\begin{audittable}{lrrrr}{The four entries of Table S7 that differ: the printed value, the value read from Fig.~2, the value from the authors' code on the public data (Code), and that value at two decimals (Rounded). SE is the heteroskedasticity-robust standard error.}\nEntry & Table S7 & Fig.~2 & Code & Rounded \\\\\nNorms, effect with 0\\% referents & $0.05$ & $0.042$ & $0.0421$ & $0.04$ \\\\\nNorms, interaction & $-0.53$ & $-0.511$ & $-0.5107$ & $-0.51$ \\\\\nNorms, SE of the interaction & $1.06$ & $1.069$ & $1.0692$ & $1.07$ \\\\\nPeer conflict, interaction & $-0.85$ & $-0.841$ & $-0.8413$ & $-0.84$ \\\\\n\\end{audittable}\n\nThe standard error in the third row is read from the curvature of the band: if $h(c)$ is the half-width of the dashed band around the treatment line, then $\\{h(c)/1.96\\}^2$ is a quadratic in $c$ whose leading coefficient is the variance of the slope. The other 12 estimates and robust standard errors of Table S7 agree with the code on the public data at the printed precision. For the interaction of peer conflict, the fit on the 49 schools has standard error $0.4626$, which is the printed $0.46$, and a two-sided normal $P$ value of $0.069$, which is the $P = 0.07$ of the caption. The band of panel D implies the smaller standard error $0.394$, which comes from the fit on 56 schools.\n\nThe code on the public data matches the treatment band at all 178 plotted proportions in each panel to within $0.000032$ outcome units. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The values in the fourth column are values for the public data, and the source of the four printed entries is not known.", "problem": "For each outcome, Table S7 reports the treatment effect when no seed is a social referent, $\\widehat{\\tau}_0$, and the slope $\\widehat{\\beta}$ of the effect in the proportion $c$ of social referents. The note to the table calls Fig.~2 its graphical version. Since the figure plots the predicted means in control and in treatment schools, the gap between the two lines at proportion $c$ is $\\widehat{\\tau}_0 + \\widehat{\\beta} c$. Panels A and C agree with the table at the printed precision. In panel B the gap has intercept $0.042$ and slope $-0.511$, against the printed $0.05$ and $-0.53$, and in panel D the slope is $-0.841$ against the printed $-0.85$. Rounding in the table and in the drawing does not account for these differences. The authors' code on the public data (ICPSR 37070 V2) reproduces the lines and treatment bands of all four panels, and its values are those of the figure. The figure and the code on the public data agree, and four entries of the table differ from them in the second decimal: the three marked entries for prescriptive norms and the interaction for peer conflict.\n\nThe band of panel D is a separate matter. On the public data, the code draws it using the uncertainty of a treatment-control contrast from a fit on all 56 schools around a line fitted on the 49 schools of the conflict analyses, and it is not the interval for the predicted mean that the caption describes; this is the subject of \\fid{P037-N026}. For that reason the band of panel D cannot be used to recover the reported test of the interaction. It is narrower than the same-fit contrast interval at low referent proportions, but not throughout the plotted range.", "what_survives": "Panels A and C, and 12 of the 16 estimates and robust standard errors of Table S7, agree at the printed precision. The interaction for peer conflict is negative in the figure, in the table and on the public data, and the $P = 0.07$ of the caption is reproduced. The qualitative reading of the fitted lines is unchanged. I draw no conclusion here about the average effect or its significance; the bootstrap rows of Table S7 were not included in this comparison. The prediction at 20\\% referents that the Results take from Fig.~2D is the subject of \\fid{P037-N010}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N011", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of seven lines of the right column of the Results. The sentence 'These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms' ends with the citation '(SI Appendix, Figs. S3 and S6 and Tables S3 and S4)', which is marked with a red box.", "caption": "Results, p. 569 (PDF p. 4), right column. The marked citation accompanies the statement that prescriptive and descriptive norms moved in the same manner; it does not include Fig. S5.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N013.png" } ], "formal_release": null, "formal_statements": [], "headline": "The citation for the statement that descriptive norms moved like prescriptive norms points to four items of the SI Appendix, and none of them shows that comparison; Fig. S5, the only display of descriptive-norm contrasts in the publication, is not cited.", "id": "P037-N013", "location": "Results, p. 569 (PDF p. 4), right column, sentence beginning “These statements about the undesirability of conflict behaviors”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms (SI Appendix, \\hl{Figs. S3 and S6 and Tables S3 and S4}).", "corrected": "These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms (SI Appendix, Figs. S3\\hl{, S5,} and S6 and Tables S3 and S4).\n\nThe four exposure means for descriptive norms and their contrasts are not published. In a reconstruction on the public data using the Rounding sampler, they are the values of the table above: means of $-0.006$, $-0.958$, $-0.362$ and $-0.113$ for control schools, no seed exposure, nonreferent exposure and referent exposure, with contrasts of $0.846$ ($P = 0.009$) for referent against no seed exposure and $0.249$ ($P = 0.30$) for referent against nonreferent exposure.", "evidence": "I checked the cited pages. In Fig.~S5, the contrasts for peers of social referent seeds are positive for both measures: $0.432$ ($p = 0.044$) for prescriptive norms and $0.635$ ($p = 0.013$) for descriptive norms. Fig.~S5 compares students within treatment schools, and it does not reproduce the four exposure means of Table S5.\n\nI also ran the descriptive-norm option of the authors' code on the public data (ICPSR 37070 V2), using an adapter for the released variables and reconstructed exposure inputs. The exposure probabilities were rebuilt from 10,000 simulated assignments with seed 123, the variances from the first 1,000 of them, and the principal components were given the signs of Tables S3 and S4. The table below shows the result with the Rounding sampler that R used before version 3.6. With the current Rejection sampler, the two contrasts are $0.875$ ($P = 0.007$) and $0.277$ ($P = 0.22$), on 2,101 students. In both runs the mean under referent exposure is above the means of the other two exposures in treatment schools, which is the ordering of the prescriptive-norm means, and the difference between referent and nonreferent exposure is not significant.\n\n\\begin{audittable}{lrr}{Descriptive-norm index under the four exposure conditions: a reconstruction with the authors' code on the public data, using the Rounding sampler, 2,094 students. The standard errors are randomization-based, and $P$ is from the two-sided Wald test of the code.}\nQuantity & Estimate & Standard error \\\\\nMean, control school & $-0.006$ & $0.102$ \\\\\nMean, treatment school, no seed exposure & $-0.958$ & $0.293$ \\\\\nMean, nonreferent seed exposure & $-0.362$ & $0.196$ \\\\\nMean, social referent seed exposure & $-0.113$ & $0.145$ \\\\\nReferent minus no seed exposure ($P = 0.009$) & $0.846$ & $0.325$ \\\\\nReferent minus nonreferent exposure ($P = 0.30$) & $0.249$ & $0.239$ \\\\\n\\end{audittable}\n\nThese values are computed from the public data. The corresponding values from the authors' own data files are unknown: those files are not public, and ICPSR states that some variables in the release were changed. Two comparisons show how close the public data come. All 48 loadings of Tables S3 and S4 are reproduced at the printed precision. The prescriptive-norm means from the same run have a largest absolute difference of approximately $0.023$ from Table S5, on 2,273 students where the note to Table S5 states 2,451.", "problem": "None of the four cited items separately identifies an effect on descriptive norms. Fig.~S3 shows scree plots, and Tables S3 and S4 report factor loadings; these describe how the two norm indices were built. Fig.~S6 repeats the analyses of Figs.~2 and 3 with another measure of centrality, and it has no separately identified result for descriptive norms. Table S5, which reports the main exposure estimates, has no column for descriptive norms. The only explicitly identified post-treatment contrasts for descriptive norms in the publication are in Fig.~S5 (SI pp.~30--32), under the alternative specification, and Fig.~S5 is not cited. The authors' analysis code has an option that applies the main four-exposure analysis to the descriptive-norm index, but the script that writes Table S5 reads only the four outcomes of that table, and the publication has no estimate from that option.", "what_survives": "The statement is not shown to be false. The publication documents the parallel between the two measures only under the alternative specification of Fig.~S5. On the public data, the authors' code shows the same qualitative ordering under the main four-exposure analysis, although the difference between referent and nonreferent exposure is not significant for descriptive norms. The estimates from the authors' own files are not available. No reported estimate changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N013", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Two excerpts of a numbered list, separated by a grey horizontal rule. Above the rule: 1. Inappropriate language towards student (written or verbal), marked by a red box, and 2. Verbal altercation with student. Below the rule: 13. Inciting violence/planning fight; 14. Inappropriate language towards student (written or verbal), marked by a red box; and 15. Verbal altercation with student.", "caption": "SI Appendix, list of disciplinary codes: items 1 and 2 from p. 10 (above the grey rule) and items 13 to 15 from p. 11 (below it). The marked item 14 repeats the marked item 1, and the list then runs through the same thirteen categories a second time.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N018.png" } ], "formal_release": null, "formal_statements": [], "headline": "In the list of disciplinary codes, thirteen categories are printed twice, as items 1–13 and again as items 14–26, and two categories that the released code sums are missing. The outcome is the sum of seventeen counts, each entered once.", "id": "P037-N018", "location": "SI Appendix, “Codes for administrative disciplinary action (counts summed for each individual and school)”, pp. 10–12 (PDF pp. 10–12 of the SI Appendix), numbered list, items 1–28", "pdf_page": 10, "printed_page": "10–12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "``1. Inappropriate language towards student (written or verbal)'' \\ldots ``13. Inciting violence/planning fight'', then ``\\hl{14. Inappropriate language towards student (written or verbal)}'' \\ldots ``26. Inciting violence/planning fight'', then ``27. Number of H.I.B. incidents as offender'' and ``28. Number of H.I.B. incidents as target''.", "corrected": "\\hl{Items 1--13 are deleted}. Items 14--26 keep their wording, which is the more specific of the two versions. \\hl{``Taking or abusing other student property'' and ``Online harassment'' are added} before the two H.I.B. counts, and the list is renumbered 1--17. The outcome for student $i$ is\n\\[\nP_i=\\sum_{j=4}^{16}c_{ij}+c_{i,46}+c_{i,47}+o_i+t_i,\n\\]\nwhere $c_{ij}$ denotes the count of the code \\texttt{de}$j$, $o_i$ the count \\texttt{hibo} of original item 27 and $t_i$ the count \\texttt{hibt} of original item 28, and a missing count is taken as zero. Each of the seventeen counts enters once. The wording of original item 28 is corrected in \\fid{P037-N019}.", "evidence": "The evidence is a check of pp.~10--12, a reading of the released code and a recalculation on the public data. Items 1 and 14, items 2 and 15, and each later pair up to items 13 and 26 name the same category. The investigators' codebook has one variable for each of the thirteen categories (\\texttt{de4} to \\texttt{de16}), one for each of the two H.I.B. counts (\\texttt{hibo} and \\texttt{hibt}), and the variables \\texttt{de46} and \\texttt{de47} with the labels quoted above. The script \\texttt{2. Cleaning.R}, lines 36--38, selects these seventeen variables once each, sets missing entries to zero and sums them for each student. I ran these lines on the public data after adapting the column names and filter variables and applying the first script's filter (line 33), which retains 24,191 of the 24,471 public records and removes 280, the number that the script's own comment states. The thirteen categories sum to 2,281, the two unlisted categories to 44 and 36, and the two H.I.B. counts to 811 and 903. The total is 4,075. The fifteen distinct counts named in the printed list sum to 3,995; the total difference is 80, and each retained student's difference equals \\texttt{de46} plus \\texttt{de47}.", "problem": "Items 14--26 repeat the thirteen categories of items 1--13 in the same order. The wording differs in three places: item 16 reads ``Bullying / H.I.B. report (H.I.B.: harassment, intimidation, and bullying New Jersey state report)'' where item 3 reads ``Bully/HIB'', item 22 has ``can be one way or two way'', and item 23 adds ``NOT pushing/hitting/shoving/tackling''. The heading says that the counts are summed, and the list has 28 entries for 15 distinct counts. The released code sums 17 counts. Besides the thirteen categories and the two H.I.B. counts, it includes two categories that the list does not name, ``Taking or abusing other student property'' (\\texttt{de46}) and ``Online harassment'' (\\texttt{de47}).", "what_survives": "The code enters each of the thirteen repeated categories once, and the repetition in the list is not a double count in the outcome. Correcting the list to the seventeen counts leaves the outcome and every estimate computed from it unchanged. The check does not show that the two unlisted categories should have been left out, and I did not compute how the estimates change without them. The totals are for the public data (ICPSR 37070 V2). The authors' own data file is not public, and ICPSR states that some variables of the release were changed; I cannot confirm that the counts in the two files are equal." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N018", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Four lines of a numbered list: 25. Sexual harassment, inappropriate sexual conduct (inc. sexual behavior in school); 26. Inciting violence/planning fight; 27. Number of H.I.B. incidents as offender; 28. Number of H.I.B. incidents as target. A red box marks the word incidents in item 28.", "caption": "SI Appendix, p. 12 (PDF p. 12), end of the list of disciplinary codes. The marked word gives the unit of item 28 as incidents; the codebook of the released data says that the variable counts writeups, and the released code adds it to the outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019.png" }, { "alt": "Two excerpts from one column of text, separated by a grey horizontal rule. Above the rule, a red box that runs over three lines marks the words each student in the school was officially disciplined for peer conflict on average 0.20 times per year. Below the rule, a red box that runs over two lines marks the words each student was disciplined on average 0.08 times during the year.", "caption": "Results, p. 569 (PDF p. 4), left column: lines from the first paragraph (above the grey rule) and from the second (below it). The marked words give the level of the outcome, 0.20 in control schools and 0.08 at 20% social referents, as the number of times each student was disciplined.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019-b.png" }, { "alt": "The caption of Fig. 1, Overall school climate results: distribution of disciplinary events throughout school networks, comparing treatment, and control schools. A red box that runs over three lines marks the words the average number of times each student in the school was disciplined for peer conflict. A second red box, over two lines, marks the words the number of times they were disciplined during the year.", "caption": "Caption of Fig. 1, p. 568 (PDF p. 3). The marked words describe the colour of a school as the average number of times each student was disciplined for peer conflict, and the size of a student’s node as the number of times the student was disciplined.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "The disciplinary outcome is described as the number of times each student was disciplined for peer conflict, but it includes writeups that name the student as the target of harassment: 903 of the 4,075 units of the outcome on the public data. The outcome is an index of administrative records, and item 28 of the SI list counts writeups, not incidents.", "id": "P037-N019", "location": "SI Appendix, “Codes for administrative disciplinary action (counts summed for each individual and school)”, pp. 10–12 (PDF pp. 10–12 of the SI Appendix), item 28; main text, Results, p. 569 (PDF p. 4), left column, first two paragraphs; caption of Fig. 1, p. 568 (PDF p. 3)", "pdf_page": 10, "printed_page": "10–12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": { "as_printed": "SI Appendix, p.~12: ``28. Number of H.I.B. \\hl{incidents} as target''\n\nResults, p.~569: ``In a control school, we estimated that \\hl{each student in the school was officially disciplined for peer conflict on average 0.20 times per year}.'' Later on the same page: ``we estimate that \\hl{each student was disciplined on average 0.08 times during the year}''.\n\nCaption of Fig.~1, p.~568: ``Color coding reveals \\hl{the average number of times each student in the school was disciplined for peer conflict}, from dark blue (little conflict) to dark orange (many disciplinary events; higher concentration of dark oranges among control schools). Student nodes are colored red when \\hl{the student was disciplined for conflict}, and their node is scaled to \\hl{the number of times they were disciplined during the year}.''", "corrected": "SI Appendix, p.~12: ``28. Number of H.I.B. \\hl{offender writeups naming the student} as target \\hl{(multiple writeups may refer to one incident)}''\n\nThe definition of the outcome is added to the SI Appendix. For student $i$,\n\\[\nP_i=\\sum_{j=4}^{16}c_{ij}+c_{i,46}+c_{i,47}+o_i+t_i,\n\\]\nwhere $c_{ij}$ denotes student $i$'s count in the field \\texttt{de}$j$, $o_i$ is the count of item 27 (H.I.B. incidents as offender), $t_i$ is the number of H.I.B. offender writeups that name the student as target, and a missing count is taken as zero. \\hl{$P_i$ is an index of administrative records of peer conflict; it is not the number of times student $i$ was disciplined.}\n\nResults, p.~569: ``In a control school, \\ldots\\ \\hl{the index of administrative records of peer conflict, which counts disciplinary reports of a student and writeups naming the student as a target, averaged 0.20 per student} per year.'' In the second paragraph: ``\\ldots\\ \\hl{the fitted mean of this index was about 0.095 per student} during the year'' (the adjusted prediction reconstructed on the public data, agreeing with Fig.~2D to plotted precision; \\fid{P037-N010}).\n\nCaption of Fig.~1, p.~568: ``Color coding reveals \\hl{the average per student of the index of administrative records of peer conflict (disciplinary reports of a student and writeups naming the student as a target)}, \\ldots\\ Student nodes are colored red when \\hl{the student's index is positive}, and their node is scaled to \\hl{the value of the index}.''\n\nThe printed 0.20 stays as the rounded unadjusted mean of the index over all 28 control schools, reproduced on the public data. If the number of a student's own disciplinary actions is wanted, the count as target is removed and the remaining counts are reconciled by incident; on the public data the sum without the count as target has the values in the last column of the table above, and that sum is not yet a count of distinct disciplinary actions.", "evidence": "The evidence is a check of the codebook, an exact count, a reading of the released code and a recalculation on the public data.\n\nFor a student $t$ and an incident $e$, let $m(e)$ denote the number of offender writeups for $e$ that name $t$ as target. The variable equals $W=\\sum_e m(e)$, and I write $I$ for the number of incidents with $m(e)>0$, the incidents represented in these writeups. Subtracting, I obtain\n\\[\nW-I=\\sum_{e:\\,m(e)>0}\\bigl(m(e)-1\\bigr)\\ge 0,\n\\]\nwith equality only if $m(e)=1$ for every incident with $m(e)>0$. In the codebook's example one incident has three offenders, each of whom names the same target, and I find $W=3$ and $I=1$.\n\nThe script \\texttt{2. Cleaning.R}, lines 36--38, forms the outcome from the seventeen variables \\texttt{de4} to \\texttt{de16}, \\texttt{de46}, \\texttt{de47}, \\texttt{hibo} and \\texttt{hibt}, and the school and network scripts use this variable when the disciplinary outcome is selected. I ran these lines on the public data (ICPSR 37070 V2), with an adapter for the release's encodings; the filter of the first script retains 24,191 of the 24,471 records, in 56 schools.\n\n\\begin{audittable}{lrr}{Components of the disciplinary outcome on the public data: sums over the 24,191 retained records of the counts that the released code adds, and their shares of the total. The count as target is the number of offender writeups that name the student as target.}\nComponent & Sum & Share \\\\\nThirteen categories of the list (\\texttt{de4} to \\texttt{de16}) & 2,281 & 56.0\\% \\\\\nTaking or abusing other student property (\\texttt{de46}) & 44 & 1.1\\% \\\\\nOnline harassment (\\texttt{de47}) & 36 & 0.9\\% \\\\\nH.I.B. incidents as offender (\\texttt{hibo}, item 27) & 811 & 19.9\\% \\\\\nH.I.B. writeups as target (\\texttt{hibt}, item 28) & 903 & 22.2\\% \\\\\nOutcome: sum of the seventeen counts & 4,075 & 100.0\\% \\\\\nSum without the count as target & 3,172 & 77.8\\% \\\\\n\\end{audittable}\n\nThe 389 students whose outcome is positive through the count as target alone account for 650 of the 903 writeups, and all of them are in the 49 schools of the conflict analysis (26 control, 23 treatment). I then ran the school-level regressions of the script \\texttt{4. School results-with-correction.R} (lines 162 and 208, with the robust covariance of lines 163 and 209) on the public data, once with the outcome as coded and once without the count as target. The same 49 schools pass the script's screen in both cases.\n\n\\begin{audittable}{lrrr}{The disciplinary outcome with and without the count as target, on the public data. Published: the values printed in the Results and in Tables S6 and S7. As coded: the sum of the seventeen counts. Without target: the sum without the count as target. The two right-hand columns are computed from the public data with the released code; they do not establish the values that the authors' own data file would give. The first mean is the script's reporting line, over all 28 control schools; the second is over the 26 control schools of the analysis. Main effect: the school level main effect of Table S6. Interaction: the interaction with the proportion of referents of Table S7. Robust standard errors in parentheses.}\nQuantity & Published & As coded & Without target \\\\\nMean, 28 control schools & 0.20 & 0.197 & 0.154 \\\\\nMean, 26 control schools & & 0.212 & 0.166 \\\\\nMain effect & $-0.06$ (0.03) & $-0.057$ (0.035) & $-0.053$ (0.029) \\\\\nInteraction & $-0.85$ (0.46) & $-0.841$ (0.463) & $-0.773$ (0.384) \\\\\n\\end{audittable}\n\nOn the public data the main effect and its standard error round to the printed values of Table S6, and the interaction is $-0.84$ where Table S7 prints $-0.85$. The script's reporting line (line 273) averages over all 28 control schools. Its value on the public data is 0.197, which rounds to the printed 0.20; the choice of schools in that average is the subject of \\fid{P037-N009}.", "problem": "The released code builds the outcome as the sum of seventeen administrative counts for each student, with every missing count set to zero: the thirteen categories of the SI list, two categories that the list omits (\\fid{P037-N018}), the H.I.B. count as offender of item 27, and the H.I.B. count as target of item 28. The count of item 28 does not record discipline of the student. The investigators' codebook defines the variable (\\texttt{hibt}) in these words: ``Since HIB incidents were reported by offender, if 3 offenders targeted one other student as part of one event, the target would be listed in 3 writeups, and this value would reflect that number of writeups (3), not the number of incidents (1)''. A student therefore scores on the outcome each time a writeup names the student as the target of harassment by others. On the public data this count contributes 903 of the 4,075 units of the outcome (22.2\\%). There are 526 students with a positive count as target, and 389 of them (74\\%) have zero in the other sixteen counts, so that their outcome is positive through writeups as target alone. The main text describes the level of the outcome as the number of times each student was disciplined (0.20 in control schools, 0.08 at 20\\% social referents), and the caption of Fig.~1 describes the colours and node sizes in the same way. These descriptions are inaccurate. The outcome is an index of administrative records of peer conflict, which counts disciplinary reports of a student and writeups that name the student as target. Item 28 also misstates its own unit: the count is a number of writeups, and it equals the number of incidents represented for that student only if each such incident contributes exactly one writeup naming that student.", "what_survives": "The survey outcomes (perceived norms, conversations with friends, wristbands), the randomization and the network definitions are not affected. The estimates for the disciplinary outcome remain estimates for a defined index of administrative records, and the printed 0.20 is reproduced to rounding on the public data as the mean of that index. Without the count as target, the two coefficients of the school-level regressions keep their sign and a similar size on the public data. This comparison does not establish that the published standard errors or statements of significance are unchanged, and the values come from the public data: the authors' own data file is not public, and ICPSR states that some variables of the release were changed. What changes is the description of the outcome and of its levels as each student's own discipline, in the Results and in the caption of Fig.~1, and the unit of item 28.\n\nThree points are not established. I did not recompute the network estimates of Fig.~3D and Table S5 or the drawing of Fig.~1 without the count as target. The check does not show that one incident was counted both under the code ``Bully/HIB'' (item 3) and under the offender count of item 27; on the public data 58 students have both counts positive, 26 only the first and 608 only the second, and these numbers do not link the counts to incidents. Removing the count as target does not turn the remaining sum into a count of distinct disciplinary actions." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N019", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Five lines of text from page 13 of the SI Appendix. A red box marks the words in percentage points in the phrase the slope of the average causal effect with respect to group composition in percentage points. A second red box marks the end of the parenthesis that follows: c percent is tau-zero plus beta c.", "caption": "SI Appendix, p. 13 (PDF p. 13), text after equation (2). The marked words give the unit of the slope $\\beta$ as percentage points, and the marked formula gives the effect at composition $c\\%$ as $\\tau_0+\\beta c$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N020.png" } ], "formal_release": null, "formal_statements": [], "headline": "The slope $\\beta$ is reported per unit of the proportion of social referents in the seed group, on the scale from 0 to 1, and not per percentage point.", "id": "P037-N020", "location": "SI Appendix, p. 13 (PDF p. 13 of the SI Appendix), text after equations (1) and (2); the same wording after equations (3) and (4), pp. 15–16, and in the note to Table S7 (PDF p. 42)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``$C_{ij}$ is the composition of the seed group (\\hl{percentage} of seed group that is classified as a social referent)''\n\n``$\\beta$ represents the slope of the average causal effect with respect to group composition \\hl{in percentage points} (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\hl{\\beta c}$.)''\n\nNote to Table S7: ``the first derivative of the effect size with respect to the \\hl{percentage} of the group that were referents''", "corrected": "``$C_{ij}$ is the composition of the seed group (\\hl{proportion} of seed group that is classified as a social referent)'', and likewise for $C^{E}_{ij}$.\n\n``$\\beta$ represents the slope of the average causal effect with respect to group composition, \\hl{per unit of this proportion} (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\hl{\\beta c/100}$.)''\n\nNote to Table S7: ``the first derivative of the effect size with respect to the \\hl{proportion} of the group that were referents''", "intended_meaning": "The coefficients of Table S7 and the lines of Fig.~2 use the proportion of social referents in the seed group. The title of Table S7 says ``proportion'', and the horizontal axis of Fig.~2 is labelled ``Proportion of social referent seeds in anti-conflict group'', with ticks from 0.05 to 0.20. With the peer-conflict coefficients $\\tau_0=0.04$ and $\\beta=-0.85$ of Table S7, the effect at 20\\% social referents is $0.04-0.85\\times 0.20=-0.13$. The printed formula with $c=20$ would give $0.04-0.85\\times 20=-16.96$. The slope per percentage point is $\\beta/100=-0.0085$.\n\nThe released code uses the proportion. The first script computes the composition of the seed group and of the seed-eligible group as means of 0/1 indicators of social referents (lines 141--146). The school script enters these proportions in the regression (line 208) and draws the treatment line of Fig.~2 as the adjusted control mean plus $\\tau_0+\\beta p$, where $p$ is the proportion (lines 311--313 and 332--333). With these scripts adapted to the public ICPSR 37070 V2 data, I find $\\tau_0=0.0408$ and $\\beta=-0.841$ per unit of proportion, that is, $-0.00841$ per percentage point, and a fit with the percentage as predictor preserves fitted values within numerical precision. The public data do not reproduce the published fit exactly: the slope rounds to $-0.84$ where Table S7 prints $-0.85$, and the predicted outcome at 20\\% is 0.095 where the main text prints 0.08 (see \\fid{P037-N010}). The authors' own data file is not public, and ICPSR notes that some variables were changed for release; this note does not explain these differences." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N020", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of five lines of the SI paragraph beginning 'component probability, pi_i(d_k), is, by definition, equal to the expected proportion of treatment assignments that induce exposure d_k for student i.' Two red boxes mark the words 'achieve an arbitrarily precise estimate of pi_i via J = 10,000', which run over two lines. The sentence continues 'simulated random assignments of Z according to the experimental design', followed by the formula for the estimated probability.", "caption": "SI Appendix, p. 14 (supplement PDF p. 14), the paragraph that introduces the simulated exposure probabilities. The marked words attach “arbitrarily precise” to the fixed choice $J = 10{,}000$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N021.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22" }, { "id": "P037-T29", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T29" }, { "id": "P037-T30", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T30" } ], "headline": "A fixed number of $10{,}000$ simulated assignments does not give an “arbitrarily precise” estimate of the exposure probabilities; on the public data, simulation noise changes whether 33 students pass the code’s probability rule, about 1% of those who pass.", "id": "P037-N021", "location": "SI Appendix, p. 14 (supplement PDF p. 14), paragraph on the exposure probabilities; the method is described again on SI p. 18. The corresponding passage of the article is on p. 568 (PDF p. 3).", "pdf_page": 14, "printed_page": "14", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T22", "P037-T29", "P037-T30" ], "text": { "as_printed": "We achieve an \\hl{arbitrarily precise} estimate of $\\pi_i$ via $J = 10{,}000$ simulated random assignments of $\\mathbf{Z}$ according to the experimental design: \\ldots", "corrected": "We \\hl{estimate} $\\pi_i$ \\hl{using} $J = 10{,}000$ simulated random assignments of $\\mathbf{Z}$ according to the experimental design\\hl{; increasing $J$ can make the Monte Carlo approximation arbitrarily accurate at any prescribed confidence level.}\n\nThe formula for $\\widehat{\\pi}_i(d_k)$ that follows the colon is corrected separately in \\fid{P037-E003}, and the rule on the estimated probabilities is described in \\fid{P037-N025}.", "evidence": "The evidence is an exact counterexample to the printed sentence and a recalculation on the public data (ICPSR 37070 V2).\n\nFor the counterexample, suppose that the school is treated with probability $1/2$ and has 40 seed-eligible students in four blocks of 10, with five seeds drawn from each block. Let a student who is not eligible have ten eligible peers, five in each of two blocks, one of them a social referent. The student has no seed peer only if the five seeds of each of the two blocks are the five students who are not peers. Since $\\binom{10}{5} = 252$, the four exposure probabilities are\n\\[\n\\pi(d_C) = \\frac{1}{2}, \\qquad \\pi(d_S) = \\frac{1}{2 \\cdot 252^2} = \\frac{1}{127{,}008}, \\qquad \\pi(d_{NI}) = \\frac{31{,}751}{127{,}008}, \\qquad \\pi(d_I) = \\frac{1}{4}.\n\\]\nAll four are positive. An estimate from $10{,}000$ draws is a multiple of $1/10{,}000$. This implies that its error for $\\pi(d_S)$ is at least $1/127{,}008$, with equality only at the estimate zero. With independent draws the estimate is zero with probability $(127{,}007/127{,}008)^{10{,}000} \\approx 0.924$.\n\nIn a reconstruction on the public data, I computed the exposure probabilities from combinatorial formulas and also estimated them from $10{,}000$ draws by adapting the authors' scripts, with their random seed (123) and the sampling method of the R versions of the time (before 3.6). Before the other exclusions, 2,921 students have all four combinatorial probabilities above $0.01$. With the estimated probabilities, 2,908 do: 23 of the 2,921 fail, and 10 other students pass. The two sets differ in 33 students. With the sampling method of current R versions and the same seed, 2,918 students pass and the two sets differ in 25. The reconstruction matches the published totals of 24,191 students in 56 schools. ICPSR reports changes to some variables; the authors' own data file and their saved draws are not public, so these counts describe the public data and not the original run.", "problem": "An estimate from a fixed number of simulated assignments (draws) has a fixed precision. SI p.~18 states the matter correctly: an arbitrarily precise estimate can be achieved by simulated assignments, and the study uses $J = 10{,}000$. The sentence on p.~14 attaches the precision to that fixed $J$. The precision matters in the application, because the authors' code keeps a student in the network analyses only if each of the four estimated exposure probabilities is greater than $0.01$ (\\fid{P037-N025}). For a student whose probability is near $0.01$, simulation noise can change the comparison.", "what_survives": "The statement of SI p.~18 on convergence and the definition of $\\pi_i(d_k)$ stand; the estimator's formula needs the separate correction in \\fid{P037-E003}. No student with probability zero of some exposure passes the rule in either run. I have not established a change to any published estimate; on the public data, using the historical sampler, the 16 exposure means computed with the simulated and combinatorial probabilities differ by at most $0.0281$." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N021", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of panel B of Fig. 2, 'Perceived social norm against conflict', with a red control line, a black treatment line and dashed bands. A red box marks the vertical axis label, 'Perceived Social Norms Against Conflict (First Principal Component, Higher Indicates Norm is More Anti-conflict)'.", "caption": "Fig. 2B, p. 569 (PDF p. 4). The marked axis label calls the index the first principal component and says that higher values are more anticonflict.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N023.png" }, { "alt": "Crop of Table S3, 'Factor loadings for Prescriptive norm items (behaviors perceived to be desirable at school)', with columns Survey Item, Wave 1 and Wave 2 and rows Item 1 to Item 7 and Item 9 to Item 13. A red box marks the two columns of loadings. The loadings are negative for Items 1, 2, 7, 10, 11, 12 and 13 and positive for Items 3, 4, 5, 6 and 9; Item 13 has a loading in Wave 2 only.", "caption": "SI Appendix, Table S3, p. 38 (supplement PDF p. 38). The marked loadings are those of the first principal component before its sign is reversed: they are negative on the items worded against conflict.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N023-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The SI does not state how the prescriptive-norm index is scored: the code uses the items as they were administered and negates the whole first principal component.", "id": "P037-N023", "location": "Figs. 2B and 3B, axis labels, pp. 569–570 (PDF pp. 4–5); SI Appendix, list of prescriptive-norm items, p. 9, and Table S3, p. 38 (supplement PDF pp. 9 and 38)", "pdf_page": 4, "printed_page": "569–570", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Fig.~2B, axis label: ``Perceived Social Norms Against Conflict (First Principal Component, Higher Indicates Norm is More Anti-conflict)''. The axis of Fig.~3B has the same label after the words ``Predicted Mean of''.\n\n\\begin{audittable}{lrr}{Table S3 (SI p.~38), ``Factor loadings for Prescriptive norm items (behaviors perceived to be desirable at school)''.}\nSurvey Item & Wave 1 & Wave 2 \\\\\nItem 1 & $-0.29$ & $-0.27$ \\\\\nItem 2 & $-0.26$ & $-0.26$ \\\\\nItem 3 & $0.32$ & $0.26$ \\\\\nItem 4 & $0.30$ & $0.27$ \\\\\nItem 5 & $0.28$ & $0.23$ \\\\\nItem 6 & $0.27$ & $0.25$ \\\\\nItem 7 & $-0.29$ & $-0.28$ \\\\\nItem 9 & $0.09$ & $0.10$ \\\\\nItem 10 & $-0.36$ & $-0.34$ \\\\\nItem 11 & $-0.34$ & $-0.33$ \\\\\nItem 12 & $-0.40$ & $-0.38$ \\\\\nItem 13 & & $-0.38$ \\\\\n\\end{audittable}\n\nSI p.~9, in the section ``Survey wording and school administrative record description'':\n\nPrescriptive norms scale (\\hl{asked at pretest and posttest}, answered using pictorial scale featured above): ``How many students think it is\\ldots''\n\n\\begin{itemize}\n\\item 3. \\hl{Not funny} to mess with or pick on others\n\\item 4. \\hl{Not OK} to say mean things to girl because of what she does with boys\n\\item 5. \\hl{Not funny} to post online or text mean things\n\\item 6. \\hl{Not OK} to use racial, ethnic jokes\n\\item \\hl{8.} \\hl{OK} to tell adults about conflict\n\\end{itemize}", "corrected": "The SI should identify the list on p.~9 as labels and retain the questionnaire's numbering:\n\n``Prescriptive-norm items collected at pretest and posttest are summarized below by \\hl{labels expressed in the anti-conflict direction, rather than by the administered wording}. \\hl{The administered PN3--PN6 and PN9 have the opposite polarity from these labels.} PN8 concerns academic effort and is omitted here. \\hl{The numbers below retain the PN numbers used in Table S3}; PN13 was collected only in Wave 2.''\n\nThe twelve labels should be numbered 1, 2, 3, 4, 5, 6, 7, \\hl{9, 10, 11, 12, 13}. The same wording and numbering repair is described under \\fid{P037-E006}.\n\nAdd the scoring rule with Table S3 or with that list:\n\n``The prescriptive-norm index uses the item responses as administered; no item is reverse-coded. In each wave, a principal components analysis of the covariance matrix is fitted to the students with complete responses, after entries greater than 10 are set to missing. The index is \\hl{the negative of the first principal component} whose loadings are printed in Table S3,\n\\[\nY_w = \\hl{-}\\,u_w^{\\top}(x_w - \\mu_w),\n\\]\nwhere $x_w$ is the vector of a student's complete responses in wave $w$, $u_w$ is the \\hl{unrounded first principal loading vector, oriented with the signs shown in Table S3}, and $\\mu_w$ is the vector of item means among the complete cases. \\hl{The index's coefficients have the opposite signs to Table S3}, and higher values indicate more anti-conflict norms. Missing baseline scores are replaced by the mean baseline score; missing posttest scores remain missing.''\n\nAnyone who runs the released scripts should fix the orientation explicitly, by requiring a positive coefficient of the index on item 1. The scripts' unconditional change of sign gives the opposite orientation under the R 4.4.2 default of \\texttt{princomp}.", "evidence": "I read the authors' scripts and ran both of their scoring branches, for the school analysis and for the network analysis, on the public data (ICPSR 37070 V2), adapting its variable coding and missing values to the scripts' input format. After the authors' filter the public data have 24,191 students in 56 schools, and the two principal components analyses use 19,630 complete cases in Wave 1 and 19,293 in Wave 2. With \\texttt{fix\\_sign=FALSE} in R 4.4.2, all 23 loadings agree with Table S3 at two decimals. Let $u_w$ be the unrounded first principal loading vector of wave $w$ with the signs of Table S3, let $x_w$ be a student's vector of item answers and let $\\mu_w$ be the vector of item means among the complete cases. For complete responses, both branches return exactly\n\\[\nY_w = -\\,u_w^{\\top}(x_w - \\mu_w) .\n\\]\nSince $u_w$ is negative on the items worded against conflict and positive on the other five, $Y_w$ increases with every move toward an anticonflict answer. Its correlation with item 1 is $0.595$ in Wave 1 and $0.607$ in Wave 2. If $C u = \\lambda u$ for the covariance matrix $C$, then $C(-u) = \\lambda(-u)$ and $(-u)^{\\top} C (-u) = u^{\\top} C u$. This implies that $-u$ is a first principal direction with the same explained variance. A missing baseline score is replaced by the mean baseline score (4,561 students on the public data), and a missing posttest score stays missing.\n\nWith this orientation, the school-level effect on the public data is $-0.014$, where Table S6 prints $-0.01$. The four exposure means from a run of the network analysis on the public data are $0.127$, $-0.823$, $-0.456$ and $0.030$ for control schools, no seed exposure, nonreferent exposure and referent exposure, on 2,273 students. The contrasts of referent exposure with the other two exposures in treatment schools are positive, as in Fig.~3B. These means are not those of Table S5, which prints $0.141$, $-0.825$, $-0.468$ and $0.007$; the authors' own files are not public, and ICPSR states that some variables in the release were changed. Under the R 4.4.2 default of \\texttt{princomp}, the rounded loadings are the negatives of Table S3 and every one of these estimates changes sign.", "problem": "The axes of Figs.~2B and 3B call the index the first principal component of the prescriptive-norm items and say that higher values are more anticonflict. Table S3 prints loadings for the same items. The SI does not say how the index is computed from the items, and the loadings cannot be read against the axes without that rule. The authors' code supplies it. The item columns enter as they were administered, and no item is reverse-coded. In each wave, a principal components analysis of the covariance matrix is fitted to the students with complete answers. The first component is then multiplied by $-1$.\n\nTo obtain the figures' stated anticonflict orientation from the component in Table S3, the index takes its negative. A component's global sign is arbitrary, so its negative is still a first principal component. The printed loadings are negative on the items that are worded against conflict and positive on items 3 to 6 and 9, which the questionnaire words the other way. With the printed signs the component decreases when a student moves toward the anticonflict answer on any item, and the index increases. The list of these items on SI p.~9 presents the five anticonflict labels as administered wording and numbers the items differently from the table (\\fid{P037-E006}). It omits item 8, on academic effort, and numbers items 9 to 13 as 8 to 12. Against that list, the loadings have mixed signs on items that all point the same way.\n\nThe sign also depends on the version of R when the code is run today. Since version 3.5.0, R's \\texttt{princomp} chooses the sign that makes the first loading non-negative, and the first loading of Table S3 is negative. The scripts multiply by $-1$ whatever sign the component has. Under the R 4.4.2 default, the same code returns a pro-conflict index and estimates of the opposite sign. This is established for a run on the public data. It is not shown for the published analysis.", "what_survives": "No published loading, estimate or figure is shown to be wrong. On the public data, the index scored as the code scores it, with the sign convention that reproduces Table S3, has the anticonflict orientation that the axes state, and the school and network results for norms have the published direction. The authors' own files and their original run are not available; my public-data recalculations do not establish their original standard errors or $P$ values for norms. The outcomes for disciplinary reports, wristbands and talking with friends do not use the index." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N023", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of eight lines of the Methods and Materials, beginning 'We considered four conditions of exposure to the seed students in each school'. A red box marks the end of that paragraph, over two lines: 'restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure.' The last line of the crop begins the next paragraph, 'Twenty-four percent of seed students did not accept our invitation to join'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), right column. The marked sentence restricts the network analyses to 2,451 students with a positive probability of all four levels of exposure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N025.png" }, { "alt": "Crop of three lines of the SI: 'We restrict ourselves to the subpopulation of students (n=2451) for which there exists some probability that students could receive all four exposures: 0 < pi_i(d_k) < 1 for k in {C, S, NI, I}.' Red boxes mark '(n=2451)' and the inequality '0 < pi_i(d_k) < 1'.", "caption": "SI Appendix, p. 18 (supplement PDF p. 18). The marked terms give the size of the population, $n = 2451$, and its definition by $0 < \\pi_i(d_k) < 1$ for the four exposures.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N025-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T21", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T21" }, { "id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22" }, { "id": "P037-T39", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T39" } ], "headline": "The network results are not computed on one population of 2,451 students with positive probability of all four exposures: for each outcome the authors' code keeps the students whose four estimated exposure probabilities exceed $0.01$, who were not seeds in the realised assignment and whose outcome and preprocessed pretreatment score are available, with a further school-mean cutoff for disciplinary reports, and it averages over that set.", "id": "P037-N025", "location": "Methods and Materials, p. 568 (PDF p. 3), right column; SI Appendix, pp. 14, 18 and 19, and the note to Table S5, p. 40", "pdf_page": 3, "printed_page": "568", "reaches": [ { "effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T21", "P037-T22", "P037-T39" ], "text": { "as_printed": "Article, p.~568: ``In this particular analysis, we restricted our network-based analyses to the subpopulation of \\hl{2,451 students who had a positive probability} of falling into all four levels of exposure.''\n\nSI Appendix, p.~18: ``We restrict ourselves to the subpopulation of students \\hl{(n=2451)} for which there exists some probability that students could receive all four exposures: $\\hl{0 < \\pi_i(d_k) < 1}$ for $k \\in \\{C,S,NI,I\\}$.''\n\nSI Appendix, note to Table S5, p.~40: ``\\ldots\\ computed on the subpopulation of \\hl{2,451 students who had a positive probability} of falling into all four levels of exposure.''", "corrected": "Article, p.~568: ``In this particular analysis, \\hl{for each outcome} we restricted our network-based analyses to the \\hl{students whose estimated probability of falling into each of the four levels of exposure exceeded 0.01, who were not seeds in the realised assignment and whose outcome and preprocessed pretreatment score were available; for disciplinary reports, only schools whose mean count exceeded 0.02 were used. The regression was fitted and its predictions averaged over the same students}.''\n\nSI Appendix, p.~18: ``\\hl{For each outcome,} we restrict ourselves to the students \\hl{with estimated probabilities} $\\hl{\\widehat{\\pi}_i(d_k) > 0.01}$ for $k \\in \\{C,S,NI,I\\}$\\hl{, who were not directly treated seeds in the realised assignment and whose outcome and preprocessed pretreatment score are available; for disciplinary reports, only schools whose mean count exceeds 0.02 are used. The regression is fitted and its predictions averaged over the same students.}''\n\nThe same restriction replaces the population descriptions on SI pp.~14 and 19 and in the note to Table S5. The single number 2,451 on p.~568, on SI pp.~18 and 19 and in the note to Table S5 should be replaced by the number of students of the original run for each outcome. On the public data these numbers are 2,457 (talk), 2,273 (norms), 2,847 (wristband) and 2,495 (disciplinary reports); they are not the original numbers. In the estimator on SI p.~19, sum over the outcome's analysis set and set $N_s$ equal to its size (\\fid{P037-N033}). If the stated population is to be kept instead, the analysis has to be carried out on it, with a treatment of the students whose outcome is missing; I have not done that analysis.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data (ICPSR 37070 V2). In the network script (\\texttt{3. Network results.R}), line 190 sets the value $0.01$, line 195 defines the analysis set by conditions (i)--(iv) and pretreatment-score availability, line 216 fits the regression on it, and lines 223--226 average the fitted values over it. For norms the code imputes missing pretreatment scores; for the other outcomes it uses a constant. The availability condition removes no student on the public data.\n\nAn exact example shows that condition (i) differs from positive probability. Suppose that a student who is not eligible has six eligible peers, one of them a social referent, in a block of 12 eligible students from which six seeds are drawn. There are $\\binom{12}{6} = 924$ equally likely draws, and the student has no seed peer in exactly one of them. With a treatment probability of $1/2$ for the school, the probabilities of the control school, of no seed peer, of a non-referent seed peer only and of a referent seed peer are $1/2$, $1/1848$, $461/1848$ and $1/4$. All are positive, and the second is below $0.01$.\n\nI recalculated these counts on the public data using the scripts, after converting its variable names and codes to the form that the scripts read. I used the random seed of the code and the sampling method of the R versions of the time (before 3.6).\n\n\\begin{audittable}{lrrrr}{Students of the public data at each step from the stated population to the analysis set, for the four outcomes. The first row uses probabilities computed exactly from the design; the article prints 2,451 for every outcome.}\n & Talk & Norms & Wristband & Conflict \\\\\nPositive probability of all four exposures & 3,120 & 3,120 & 3,120 & 3,120 \\\\\n(i) All four estimated probabilities above $0.01$ & 2,908 & 2,908 & 2,908 & 2,908 \\\\\n(ii) and not a seed in the realised assignment & 2,847 & 2,847 & 2,847 & 2,847 \\\\\n(iii) and outcome observed & 2,457 & 2,273 & 2,847 & 2,847 \\\\\n(iv) and school kept for disciplinary reports: the analysis set & 2,457 & 2,273 & 2,847 & 2,495 \\\\\nSchools represented in the analysis set & 56 & 55 & 56 & 49 \\\\\n\\end{audittable}\n\nWith the exact probabilities in condition (i), 2,921 of the 3,120 students pass. Using exact probabilities, the cutoff removes 199 students of the positive-probability population on the public data; the further membership changes from simulation noise are the subject of \\fid{P037-N021}. Condition (ii) removes 61 students. Seed-eligible students who were not drawn as seeds stay in the analysis set: 207 of the 2,847 students of the wristband analysis are seed-eligible. The code holds the analysis set fixed in the $1{,}000$ simulated assignments from which the standard errors are computed. Since no student of the analysis set has the directly treated exposure as observed exposure, the imputation of potential outcomes on SI pp.~19--20 never needs a mean for that exposure.\n\nThe public data reproduce the published 24,191 students in 56 schools. None of the four analysis sets has 2,451 students. The authors' own data file and their saved draws are not public, and ICPSR states that some variables of the public data were changed. The four numbers of the original run are not recovered, and the counts in the table are counts for the public data.", "problem": "The article, the SI Appendix (pp.~14, 18 and 19) and the note to Table S5 describe one population for every network result: the students with positive probability of each of the four exposures, 2,451 in number. This population is fixed before the assignment. The authors' code computes the results on another set, which I call the analysis set of the outcome. For each outcome the code keeps a student when the preprocessed pretreatment score is available and four conditions hold: (i) each of the four exposure probabilities, as estimated from $10{,}000$ simulated assignments, is greater than $0.01$; (ii) the student was not a seed in the realised assignment; (iii) the outcome is observed; and (iv) for disciplinary reports, the school's mean count exceeds $0.02$ (\\fid{P037-N007}). The regression is fitted on the analysis set, and the fitted values are averaged over the same set.\n\nCondition (i) is stricter than positive probability. Condition (ii) depends on the realised assignment, and conditions (iii) and (iv) depend on the outcome. The SI Appendix sets the directly treated exposure aside (p.~17), which accounts for leaving the outcomes of the realised seeds out of the fit. It does not say that these students are also left out of the set over which the means are averaged. The reported means are averages over a set that depends on the assignment and on the outcome, so the number of students must be reported for each outcome.", "what_survives": "Positive probability as a requirement, the four conditions of exposure and the reported network estimates are untouched. No estimate, $P$ value or conclusion of the article is shown to change. The school-level regressions (Tables S6 and S7) do not use the analysis sets. What changes is the population to which Fig.~3 and Table S5 refer, and its size for each outcome: describing the rule correctly replaces the fixed population of the article by a set that depends on the assignment and on the outcome." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N025", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Fig. 2, four panels with its caption at the right. Each panel plots a horizontal red line for control schools and a sloped black line for treatment schools against the proportion of social referent seeds in the anti-conflict group, with labelled ticks from 0.05 to 0.20, each between two dashed curves. Red boxes mark the key of each panel, 'Control schools' predicted mean (and 95 percent CI)' and 'Treatment schools' predicted mean (and 95 percent CI)', and the words of the caption 'and are surrounded by 95 percent confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit'.", "caption": "Fig. 2 and its caption, p. 569 (PDF p. 4). The marked sentence of the caption and the marked keys of the four panels call the dashed bands 95% confidence intervals for the predicted means of control and treatment schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026.png" }, { "alt": "Crop of the first page of Fig. S6, 'Re-analysis of Fig. 2 and 3 using betweenness in place of indegree': four panels, (a) Friends talked to me about conflict, (b) Perceived social norm against conflict, (c) Wearing anti-conflict wristband and (d) Peer conflict (disciplinary records), each with a red control line, a black treatment line and dashed bands. Red boxes mark the key of each panel, 'Control schools' predicted mean (and 95 percent CI)' and 'Treatment schools' predicted mean (and 95 percent CI)', and the words of the caption 'and are surrounded by 95 percent confidence intervals estimated using heteroskedasticity-robust standard errors for the regression fit'.", "caption": "SI Appendix, Fig. S6, school-level panels (a)–(d), p. 33 (supplement PDF p. 33). The same keys and the corresponding sentence of the caption are marked; a separate plotting loop uses the same band formulas.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026-b.png" }, { "alt": "Four panels, A to D, one for each outcome of Fig. 2: friends talked to me about conflict (score from 0 to 3), perceived social norm against conflict (first principal component), wearing anti-conflict wristband (proportion reporting wearing) and peer conflict (reports per student). In each panel there are four positions: control schools, and treatment schools at 5, 10 and 20 percent social referent seeds. At each position an orange interval with a square marker, labelled as drawn, stands beside a blue interval with a round marker, labelled as captioned; both are centred on the same predicted mean. The orange interval for control schools is shorter than the blue one in all four panels. For treatment schools the orange interval is longer than the blue one at ten of the twelve positions, and slightly shorter in panel B at 20 percent and in panel D at 5 percent. A note under each panel title says that the line is fitted on 56 schools, or 49 in panel D, and the drawn treatment band on 56.", "caption": "The bands of Fig. 2 on the public data (ICPSR 37070 V2), at control schools and at treatment schools with 5%, 10% and 20% social referent seeds. Orange squares: the interval as the plotting code draws it. Blue circles: the interval for the predicted mean, computed with the robust covariance matrix of the fit of the line, as the caption describes. The half-widths are $1.96$ times the unrounded standard errors summarised in the table above.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026-bands.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T23", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T23" }, { "id": "P037-T24", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T24" }, { "id": "P037-T40", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T40" } ], "headline": "The dashed bands of Fig. 2, and of the school-level panels of Fig. S6, are not the 95% confidence intervals for the two predicted means that the caption and the keys describe: the band around the treatment line uses the standard error of the treatment–control contrast, and the band around the control line uses the variance of the control schools' residuals divided by 56.", "id": "P037-N026", "location": "Fig. 2, panels A–D, keys and caption, p. 569 (PDF p. 4); SI Appendix, Fig. S6, school-level panels (a)–(d) and caption, p. 33 (supplement PDF p. 33)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T23", "P037-T24", "P037-T40" ], "text": { "as_printed": "Fig.~2, caption: ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of social referent seeds, and are \\hl{surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit}.''\n\nFig.~2, key in each panel: ``Control schools' predicted mean \\hl{(and 95\\% CI)}'' and ``Treatment schools' predicted mean \\hl{(and 95\\% CI)}''.\n\nFig.~S6, caption (SI p.~33): ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of high betweenness seeds, and are \\hl{surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust standard errors for the regression fit}.''", "corrected": "The caption and the keys can stand if the bands are redrawn. For each panel, let $\\widehat b$ and\n\\[\n\\widehat V = (X^{\\top}X)^{-1} X^{\\top} \\operatorname{diag}(e_1^2, \\ldots, e_n^2)\\, X\\, (X^{\\top}X)^{-1}\n\\]\nbe the coefficients and the heteroskedasticity-robust covariance matrix of the regression that gives the lines, where $X$ contains its \\hl{linearly independent design columns} and $e_i$ are its residuals, fitted on the schools of that panel's analysis (49 for peer conflict, 56 for the other outcomes). With $\\bar z$ the average of the block indicators and covariates over those schools, $\\ell_0 = (1, 0, 0, \\bar z^{\\top})$ and $\\ell_1(c) = (1, 1, c, \\bar z^{\\top})$, the bands are\n\\[\n\\widehat m_0 \\pm 1.96\\sqrt{\\hl{\\ell_0 \\widehat V \\ell_0^{\\top}}}, \\qquad \\widehat m_1(c) \\pm 1.96\\sqrt{\\hl{\\ell_1(c)\\, \\widehat V\\, \\ell_1(c)^{\\top}}} ,\n\\]\nin place of the residual-variance formula for control and the contrast standard error from the plotting refit for treatment. For Fig.~S6, $\\widehat b$, $\\widehat V$, $\\bar z$ and the control mean $\\widehat m_0$ all come from the betweenness regression. These are approximate pointwise intervals under the normal approximation that the paper uses, and no new assumption is needed.\n\nOn the public data the corrected standard errors are those in the last column of the table above. In panel D, for example, the half-width of the control band becomes $0.0445$ in place of $0.0287$, and the half-width of the treatment band at $c = 0.10$ becomes $1.96 \\times 0.0217$ in place of $1.96 \\times 0.0296$. These are values on the public data; the values from the authors' own files are not available.", "evidence": "The evidence is a derivation with an exact counterexample, a recalculation on the public data, and measurements of the printed figures from rendered pages.\n\n\\emph{Derivation.} For a fixed fitting sample and covariates, since\n\\[\n\\tau_0 T_i + \\beta\\, T_i C_i = (\\tau_0 + \\beta c)\\, T_i + \\beta\\, T_i (C_i - c),\n\\]\nthe refit with $C_i - c$ is the same regression in other coordinates. It has the same fitted values and residuals, its coefficient on $T_i$ is $\\widehat\\tau_0 + c\\widehat\\beta$, and the robust variance of that coefficient is $d(c)\\widehat V d(c)^{\\top}$. The variance of the treatment mean is\n\\[\n\\ell_1 \\widehat V \\ell_1^{\\top} = \\ell_0 \\widehat V \\ell_0^{\\top} + d \\widehat V d^{\\top} + 2\\,\\ell_0 \\widehat V d^{\\top},\n\\]\nwhich differs from $d \\widehat V d^{\\top}$ by the variance of the control mean and twice a covariance. Since the covariance may be negative, the drawn band may be wider or narrower than the captioned one.\n\n\\emph{Counterexample.} The synthetic example has 56 schools and no covariates: 14 control schools with $Y = 0$ and 14 with $Y = 1/5$; 7 treatment schools with $(C, Y) = (0, 1/10)$, 7 with $(0, 3/10)$, 7 with $(1/5, 2/5)$ and 7 with $(1/5, 3/5)$. The least squares fit of $Y = \\alpha + \\tau_0 T + \\beta\\, T C$ is $(\\widehat\\alpha, \\widehat\\tau_0, \\widehat\\beta) = (1/10, 1/10, 3/2)$, every residual is $\\pm 1/10$, and the robust covariance matrix is\n\\[\n\\widehat V = \\begin{pmatrix} 1/2800 & -1/2800 & 0 \\\\ -1/2800 & 3/2800 & -1/280 \\\\ 0 & -1/280 & 1/28 \\end{pmatrix} .\n\\]\nAt $c = 1/10$ the two predicted means are $1/10$ and $7/20$. Each has variance $1/2800$, and the contrast has variance $1/1400$, with covariance $-1/2800$ between the control mean and the contrast. The code's treatment band uses $1/1400$, twice the variance of the treatment mean. Its control band uses $s_0^2/56 = (28/100)/(27 \\cdot 56) = 1/5400$, which is $14/27$ of the variance of the control mean. All of these values are exact rational numbers.\n\n\\emph{Recalculation.} I recalculated the authors' school regressions on the public data (ICPSR 37070 V2; 24,191 students in 56 schools after the authors' filter), using an adapter for the public release and current R packages, and compared the standard error that the plotting code uses with the robust standard error of the predicted mean from the fit of each line. The table reports both for the four panels of Fig.~2, and the figure below draws the corresponding intervals. In panel D at $c = 0.05$, the drawn standard error is $0.0374$, where the contrast from the 49-school fit of the line has standard error $0.0482$. For Fig.~S6(d) at $c = 0.20$, the drawn standard error is $0.0476$ and that of the predicted mean from the betweenness fit is $0.0656$.\n\n\\begin{audittable}{lrr}{Standard errors behind the bands of Fig.~2, from the authors' school script on the public data: as the plotting code computes them, and for the predicted mean from the fit of the line. Multiply by $1.96$ for the half-width of a band.}\nPanel and line & As drawn & For the predicted mean \\\\\nA, control & $0.0082$ & $0.0122$ \\\\\nA, treatment at $c = 0.05$ & $0.0229$ & $0.0215$ \\\\\nA, treatment at $c = 0.10$ & $0.0174$ & $0.0129$ \\\\\nA, treatment at $c = 0.20$ & $0.0338$ & $0.0293$ \\\\\nB, control & $0.0343$ & $0.0534$ \\\\\nB, treatment at $c = 0.05$ & $0.1076$ & $0.0772$ \\\\\nB, treatment at $c = 0.10$ & $0.0807$ & $0.0545$ \\\\\nB, treatment at $c = 0.20$ & $0.1164$ & $0.1189$ \\\\\nC, control & $0.0040$ & $0.0055$ \\\\\nC, treatment at $c = 0.05$ & $0.0144$ & $0.0132$ \\\\\nC, treatment at $c = 0.10$ & $0.0110$ & $0.0094$ \\\\\nC, treatment at $c = 0.20$ & $0.0205$ & $0.0199$ \\\\\nD, control & $0.0146$ & $0.0227$ \\\\\nD, treatment at $c = 0.05$ & $0.0374$ & $0.0382$ \\\\\nD, treatment at $c = 0.10$ & $0.0296$ & $0.0217$ \\\\\nD, treatment at $c = 0.20$ & $0.0464$ & $0.0413$ \\\\\n\\end{audittable}\n\n\\emph{Measurement of the printed figure.} The authors' own analysis files are not public, and ICPSR states that some variables in the release were changed. To connect the code to the printed figure, I compared the bands that the code draws on the public data with the bands as printed in the two peer-conflict panels. In the published panel D, the half-width of the control band measures $0.0289$, and the half-widths of the treatment band at $c = 0.05$, $0.10$ and $0.20$ measure $0.0744$, $0.0572$ and $0.0917$. From the code on the public data, I obtain $0.0287$ and $0.0733$, $0.0579$ and $0.0910$. In the published Fig.~S6(d), the measured half-widths are $0.0302$ and $0.0758$, $0.0559$ and $0.0938$, against $0.0302$ and $0.0755$, $0.0576$ and $0.0933$ from the code. Each pair agrees within $0.002$, the tolerance of the measurement. These measurements support the code's construction in the two peer-conflict panels. The captioned interval for the control mean of panel D would have half-width $0.0445$ on the public data.", "problem": "Each panel of Fig.~2 comes from a school-level regression of the outcome $Y_i$ on the treatment indicator $T_i$, its product with the proportion $C_i$ of social referents among the seeds, block indicators and school covariates $z_i$ (SI equation (2), with the adjustments of SI p.~14):\n\\[\nY_i = \\alpha + \\tau_0 T_i + \\beta\\, T_i C_i + z_i^{\\top}\\eta + u_i .\n\\]\nAfter omitting redundant design columns, let $\\bar z$ be the average of $z_i$ over the schools in the fit, and write $\\ell_0 = (1, 0, 0, \\bar z^{\\top})$, $\\ell_1(c) = (1, 1, c, \\bar z^{\\top})$ and $d(c) = \\ell_1(c) - \\ell_0 = (0, 1, c, 0^{\\top})$. The red line is the predicted control mean $\\widehat m_0 = \\ell_0 \\widehat b$, where $\\widehat b$ is the vector of estimated coefficients, and the black line is the predicted treatment mean $\\widehat m_1(c) = \\ell_1(c)\\widehat b$ at proportion $c$. The caption and the keys say that the dashed curves are 95\\% confidence intervals for these two means, computed from the robust covariance matrix $\\widehat V$ of the fit. Such intervals are\n\\[\n\\widehat m_0 \\pm 1.96\\sqrt{\\ell_0 \\widehat V \\ell_0^{\\top}}, \\qquad \\widehat m_1(c) \\pm 1.96\\sqrt{\\ell_1(c)\\, \\widehat V\\, \\ell_1(c)^{\\top}} .\n\\]\nThe plotting code draws something else, in four respects.\n\n\\begin{enumerate}\n\\item The treatment band. For each $c$ the code refits the regression with $C_i$ replaced by $C_i - c$ and takes the robust standard error of the coefficient on $T_i$. When the refit uses the same schools and covariates as the line, that coefficient is $\\widehat\\tau_0 + c\\widehat\\beta$, the difference between the two lines, and its variance is $d(c)\\widehat V d(c)^{\\top}$. The band uses a standard error of the contrast, drawn around the treatment line. It treats the control mean as known.\n\\item The control band. The code uses $1.96\\sqrt{s_0^2/56}$, where $s_0^2$ is the sample variance of the residuals of the control schools. This is not $1.96\\sqrt{\\ell_0 \\widehat V \\ell_0^{\\top}}$, and the divisor is 56 although the fit has 28 control schools, or 26 for peer conflict on the public data.\n\\item Panel D. The line is fitted on the 49 schools of the conflict analyses (\\fid{P037-N007}), but the refit that gives its treatment band uses all 56 schools.\n\\item Fig.~S6. The refit behind the treatment bands controls for the share of social referents by indegree in the seed-eligible group, where the fit of the lines controls for the share by betweenness. The bands of Fig.~S6 share the two variance errors, panel (d) also uses the mismatched sample, and its lines are placed on the control mean of the indegree regression.\n\\end{enumerate}\n\nThe bands are not wrong in one direction. On the public data, the control band is narrower than the captioned interval in every panel. At the three proportions checked (5\\%, 10\\% and 20\\%), the treatment band is wider than the captioned interval in ten of the twelve cases and narrower in two. The Results cite Fig.~2 for the school-level effects, and a judgement of uncertainty from the overlap or the width of these bands does not have the meaning that the caption gives it. The coefficients, their standard errors and the tests are computed elsewhere in the code and do not use these formulas.", "what_survives": "The lines of Fig.~2 do not change. The coefficients for treatment and for its interaction with the proportion of social referents, their robust standard errors, the Wald tests and the bootstrap calculations are computed before the plotting, and none of them uses the band formulas. The peer-exposure analyses of Fig.~3 use other calculations. In Fig.~S6 the slopes and the treatment coefficients stay as they are, but the plotted levels move slightly when the lines are placed on the control mean of the betweenness regression: for peer conflict on the public data that mean is $0.2171$, where the plotting code uses $0.2225$ from the indegree regression. No reading of a panel is shown to reverse. This check does not establish the coverage of any of these intervals; the behaviour of the robust standard errors with 49 schools is the subject of \\fid{P037-N006}. The cells of Table S7 that differ from the fit on the public data are recorded under \\fid{P037-N011}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N026", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of seven lines of the left column of the Results, ending 'twice the average effect of the anticonflict groups.' A red box marks the parenthesis '(proportions ranged from 0 to 37 percent)' in the sentence 'When 20 percent of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37 percent), we estimate that each student was disciplined on average 0.08 times during the year'.", "caption": "Results, second paragraph, p. 569 (PDF p. 4), left column. The marked parenthesis gives the range of the proportion of social referent seeds.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N027.png" } ], "formal_release": null, "formal_statements": [], "headline": "Among the anticonflict groups of the 28 treatment schools, the proportion of social referent seeds ranges from 0 to 25% on the public data, not from 0 to 37%; only three of the 28 groups reach the 20% at which the sentence evaluates its projection.", "id": "P037-N027", "location": "Results, second paragraph, p. 569 (PDF p. 4), left column, parenthesis “(proportions ranged from 0 to 37%)”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to \\hl{37\\%}), we estimate that each student was disciplined on average 0.08 times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.", "corrected": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (\\hl{in the public replication data,} proportions \\hl{among assigned seeds in the 28 treatment schools} ranged from 0 to \\hl{25\\%}), we estimate \\ldots\n\nThe qualifier ``in the public replication data'' can be dropped if the authors' own file gives the same range. A range over the lottery groups of all 56 schools would have to be labelled as such; on the public data it is 0 to $35.7\\%$.", "evidence": "The evidence is a recalculation on the public data (ICPSR 37070 V2). I executed the authors' code for the nomination network with an adapter for the public file and applied their rule for social referents: indegree at or above the 90th percentile of the student's school, ties included. On the public data, I find 28 treatment schools with 728 assigned seeds, the published count. The proportions were then compared as exact fractions. Among the treatment schools the smallest is $0$ and the largest is $5/20$. Among all 56 schools the largest is $10/28$.\n\nThe source of the printed 37\\% is not established. The authors' own analysis file is not public, and ICPSR states that some variables in the release were changed; its notes list no change to the variables used here. The range of 0 to 25\\% is the range on the public data.", "problem": "The sentence concerns the anticonflict groups of treatment schools, and the parenthesis reads as the range of their composition: the proportion of social referents among the seeds assigned in each of the 28 treatment schools. On the public data that range is 0 to 25\\%. Three of the 28 groups reach 20\\%, with 5 social referents among 20 seeds in one school and 5 among 24 in two others. The projection at 20\\% is therefore near the top of the observed compositions, which the printed range does not show.\n\nThe code also computes a composition for the students drawn in the seed lottery of each control school, where no anticonflict group met. Over all 56 schools the largest value on the public data is 10 of 28, or $35.7\\%$, in a control school. It rounds to 36\\%, not 37\\%.", "what_survives": "Nothing is computed from the parenthesis. In the regression the seed-group composition enters only multiplied by the treatment indicator, which is zero in every control school, and the projection at 20\\% lies inside the observed range of treatment-group compositions. The fitted models, their tests and the peer-exposure analyses are untouched by this correction. The level of $0.08$ in the same sentence is the subject of \\fid{P037-N010}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N027", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of a paragraph of ten lines beginning 'Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks.' Red boxes mark the words 'the percentage' in three places: before 'of students identified as white, black, and hispanic', before 'of students identified as having limited English proficiency', and before 'of students receiving free or reduced lunch'.", "caption": "SI Appendix, “Study design”, p. 5 (supplement PDF p. 5). The three marked phrases describe the demographic blocking variables as percentages.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N030.png" } ], "formal_release": null, "formal_statements": [], "headline": "The blocks of schools were formed on the numbers of students in each demographic group, not on the percentages that the SI lists, and one of the 14 blocks was not formed by the matching procedure.", "id": "P037-N030", "location": "SI Appendix, “Study design”, p. 5 (supplement PDF p. 5), paragraph beginning “Schools were assigned to blocks of four”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks. Blocks were composed to maximize balance on the following variables: the latitude and longitude location of each school, the average school population as measured by the number of students who took the pre-randomization student surveys, the 5th, 6th, and 7th grade population during the year prior to the study (2011), \\hl{the percentage} of students identified as white, black, and hispanic, \\hl{the percentage} of students identified as having limited English proficiency, and \\hl{the percentage} of students receiving free or reduced lunch as identified by the New Jersey State Department of Education, and finally the average network clustering coefficient and network density calculated from student network data gathered in the pre-randomization student surveys.", "corrected": "Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks. Blocks were composed to maximize balance on the following variables: the latitude and longitude location of each school, the average school population as measured by the number of students who took the pre-randomization student surveys, the 5th, 6th, and 7th grade population during the year prior to the study (2011), \\hl{the number} of students identified as white, black, and hispanic, \\hl{the number} of students identified as having limited English proficiency, and \\hl{the number} of students receiving free or reduced lunch as identified by the New Jersey State Department of Education, and finally the average network clustering coefficient and network density calculated from student network data gathered in the pre-randomization student surveys. \\hl{Thirteen blocks were formed using the matching procedure on the listed variables; the four schools that already had a block label were assigned to a fourteenth block.}\n\nThe percentages and proportions of the other demographic columns of Table S1 need no change for this correction.", "evidence": "The evidence is a reading of the code, with three checks. First, I executed the statements of the script that build the blocking variables on invented school records and confirmed that the five demographic columns reach the blocking routine as counts. The codebook of the public release describes their components as numbers of students. Second, Table S1 independently rules out percentages for limited English proficiency: for one block it prints a mean of $51.25$ and a standard deviation of $84.74$. Four percentages $x_i$ in $[0, 100]$ with mean $\\mu$ have variance\n\\[\n\\frac{1}{4}\\sum_i x_i^2 - \\mu^2 \\le 100\\mu - \\mu^2 \\le 2500 ,\n\\]\nand this implies that their standard deviation is at most $50$, or about $57.7$ with the divisor 3.\n\nThird, counts and shares are not equivalent under the covariance scaling of the Mahalanobis distance, which is the default of the blocking routine in blockTools version 0.6.6, even when enrollment is among the variables. Suppose that four invented schools have enrollments $100$, $200$, $300$ and $400$ and group counts $10$, $20$, $120$ and $200$, so that the shares are $1/10$, $1/10$, $2/5$ and $1/2$. With the pair (enrollment, count), the squared distance between the second and third schools is $291/122$ and that between the second and fourth is $358/61$. The third school is the nearer match. With the pair (enrollment, share), the two distances are $51/10$ and $22/5$, and the fourth school is nearer.\n\nThe check does not reconstruct the original matching. The school file that the script reads and the original software environment are unavailable; the public school variables are given in ranges.", "problem": "The authors' blocking script does not divide by enrollment. It defines the variables for white, black and Hispanic students and for free or reduced lunch as sums of numbers of students, and it passes them, with the number of students of limited English proficiency, to the routine that forms the blocks. Division by the school total appears only later in the script, for the summaries of Table S1. When schools differ in size, a count is not a rescaled share, and counts also follow school size. Matching on counts is therefore not matching on percentages.\n\nThe script also keeps four schools out of the matching. It forms 13 blocks with the matching routine and carries those four schools, which already had a block label, in as the fourteenth block. The code does not say how that group was chosen.", "what_survives": "The randomization within blocks does not depend on how the blocks were formed. The script assigns two of the four schools of each block to treatment by a random permutation, so the six allocations are equally likely and each school has probability $1/2$. The public data show 56 schools in 14 blocks of four, each with two treatment and two control schools. The analyses use the recorded blocks, so correcting this description changes no estimate or standard error. Nothing here shows that the fourteenth block was poorly balanced, or which blocks would have differed had shares been used." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N030", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of four lines of the SI: 'Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects. Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) to estimate the standard errors under the maintained hypothesis that, for all i,'. A red box marks the words '1,000 simulated randomizations (both randomization across schools and within seed-eligibles)', which run over two lines.", "caption": "SI Appendix, p. 19 (supplement PDF p. 19), the paragraph on randomization-based confidence intervals. The marked words say that the $1{,}000$ simulated randomizations include the randomization across schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N031.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T31", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T31" }, { "id": "P037-T32", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T32" } ], "headline": "The authors' code draws the school assignments simulated for the network analyses as 28 of the 56 schools without the blocks, and not as two of the four schools of each block as in the experiment; with blocked draws on the public data, the 16 standard errors corresponding to Table S5 change by at most 7% relative to unblocked draws, and none of the 12 referent contrasts crosses $0.05$.", "id": "P037-N031", "location": "SI Appendix, pp. 14 and 18 (supplement PDF pp. 14 and 18), “according to the experimental design”; the paragraph on randomization-based confidence intervals on SI pp. 19–20. The design is described on SI p. 5.", "pdf_page": 14, "printed_page": "14", "reaches": [ { "effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": "P013-N016", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [ { "id": "P013-N016", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N016", "work_id": "P013" } ], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T31", "P037-T32" ], "text": { "as_printed": "SI Appendix, p.~14: ``We achieve an arbitrarily precise estimate of $\\pi_i$ via $J = 10,000$ simulated random assignments of $\\mathbf{Z}$ \\hl{according to the experimental design}:''\n\nSI Appendix, pp.~19--20: ``Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) to estimate the standard errors under the maintained hypothesis that, for all $i$, $y_i(d_k) = y_i(D_i) + \\hat{\\mu}_R(d_k) - \\hat{\\mu}_R(D_i)$.''\n\nSI Appendix, p.~5: ``Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks.''", "corrected": "The school assignments should be drawn within the blocks, by \\texttt{sch.perms <- genperms(schtreat, \\hl{blockvar = schrb,} clustvar = schid, maxiter = niter)}, where \\texttt{schrb} is the block of the school. Under this call each simulated assignment treats two of the four schools of each of the 14 blocks, each of the six choices in a block having probability $1/6$, independently over blocks. The exposure matrices, and the standard errors, confidence intervals and $P$ values of Fig.~3, Table S5, Fig.~S4 and the network panels on SI p.~34, should then be recomputed. With this change the simulations follow the experimental design as stated on SI pp.~14 and 18.\n\nIf the unblocked procedure is retained, replace ``according to the experimental design'' on SI pp.~14 and 18 with ``\\hl{with 28 of the 56 schools drawn uniformly without regard to the school blocks, and seeds drawn among seed-eligibles within the student blocks of each school}''.", "evidence": "I checked the authors' code, derived the assignment laws and recalculated the network inference on the public data (ICPSR 37070 V2), using an adapter for the authors' scripts. The first script (\\texttt{1. Sanitize and generate perms, probs.R}, line 58) generates the school assignments by \\texttt{sch.perms <- genperms(schtreat, clustvar = schid, maxiter = niter)}. The call names the schools as clusters and names no blocks. Applied to eight schools in two blocks, it generates all 70 assignments with four treated schools, of which 34 do not treat two schools in each block.\n\nFor two schools $a$ and $c$ of one block, with treatment indicators $S_a$ and $S_c$, under the law of the experiment I have\n\\[\n\\Pr(S_a = S_c = 1) = \\frac{2}{4}\\cdot\\frac{1}{3} = \\frac{1}{6},\n\\]\nand under the law of the code I have\n\\[\n\\Pr(S_a = S_c = 1) = \\frac{28}{56}\\cdot\\frac{27}{55} = \\frac{27}{110}.\n\\]\nThe exact marginal exposure probabilities agree. Let $U_a$ be the seeds drawn in school $a$, which are drawn independently of the school assignment. Since the network of the public data has no tie between schools, the exposure of a student of school $a$ is a function of $(S_a, U_a)$. Under both laws $\\Pr(S_a = 1) = 1/2$ and $U_a$ is independent of $S_a$. This implies that the pair $(S_a, U_a)$ has the same distribution under both, and that each $\\pi_i(d_k)$ is unchanged.\n\nOn the public data I ran the network script twice, once with the code's school draws and once with school draws made within the blocks, each with $10{,}000$ draws for the exposure probabilities and $1{,}000$ for the variances. The largest change among the 16 standard errors corresponding to Table S5 is $0.0105$, or 7.0\\%. Of the 12 contrasts with referent peer exposure, none crosses $0.05$, and the largest change in a $P$ value is $0.150$, from $0.823$ to $0.673$, for a contrast in disciplinary reports. With the estimated probabilities and the screened sample held fixed, so that only the school draws change, the largest change in a standard error is 5.8\\% and the largest change in a $P$ value is $0.009$. For prescriptive norms, the contrast of referent with non-referent peer exposure, which the article reports with $P < 0.01$, has $P = 0.0101$ with the code's draws, $0.0144$ with blocked draws, and $0.0113$ with blocked draws and the probabilities held fixed. These finite simulations have Monte Carlo uncertainty. The standard errors from the code's draws on the public data are close to those of Table S5 and do not all round to them. Since the authors' own data file and their saved draws are not public, these values are values for the public data, and they do not show whether the original $P$ value was below $0.01$. I did not recompute the placebo panels (Fig.~S4) or the betweenness panels (SI p.~34), which use the same draws.", "problem": "The experiment assigned two of the four schools of each of 14 blocks to treatment. In the authors' code, the simulated school assignments are generated without the block variable, and each one treats 28 schools drawn uniformly from the 56. Under both laws each school is treated with probability $1/2$, but the joint laws differ. The exact exposure probabilities depend only on the assignment of the student's own school and on the seeds drawn in it, and they are the same under both laws. The standard errors, confidence intervals and $P$ values of the network analyses come from the same simulated assignments, and they depend on the joint law. The SI Appendix describes these simulations as following the experimental design.", "what_survives": "The experiment was blocked as described. The exact exposure probabilities, the population with positive probability of all four exposures and the point estimates for fixed probability estimates and screening are unchanged; fresh finite simulations of the probabilities can change the screened sample and point estimates. The school-level regressions (Figs.~1 and 2, Tables S6 and S7) and the permutation check within treatment schools (Fig.~S5) do not use the simulated school assignments. The statement of the SI Appendix that the schools are independent is a separate matter (\\fid{P037-E004}), which blocked draws do not repair." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N031", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of nine lines of text. A red box marks 'equals student i's row in the network adjacency matrix' in the sentence 'We define the exposure mapping as a function, f(z, theta-i), such that the parameter, theta-i, equals student i's row in the network adjacency matrix.' The following sentences define gamma as an N-length vector whose jth element is 1 if student j has an indegree in the top decile of student j's school, and say that z-prime theta-i is the number of student i's peers that are assigned to be seeds.", "caption": "SI Appendix, p. 17. The marked words define the parameter $\\theta_i$ of the exposure mapping. The next sentence defines social referents by indegree, the number of nominations received.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N032.png" }, { "alt": "Crop of seven lines of text. Red boxes mark the sentence 'Our network question (i.e., whom did you choose to spend time with, face to face or online) directly measures who is observing whose behavior.'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence says that the network question measures who observes whose behavior, which is a directed relation.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N032-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T25", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T25" }, { "id": "P037-T26", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T26" }, { "id": "P037-T41", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T41" } ], "headline": "In the authors' primary social influence code, two students are peers when either nominated the other; the supplement defines a student’s exposure by “student $i$'s row in the network adjacency matrix” and does not say that the matrix of nominations was made symmetric.", "id": "P037-N032", "location": "SI Appendix, p. 17 (PDF p. 17 of the supplement), definition of the exposure mapping; Methods and Materials, p. 567 (PDF p. 2), right column, sentence on the network question; exposure conditions (ii)–(iv), p. 568 (PDF p. 3)", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T25", "P037-T26", "P037-T41" ], "text": { "as_printed": "SI p.~17: We define the exposure mapping as a function, $f(\\mathbf{z}, \\theta_i)$, such that the parameter, $\\theta_i$, equals \\hl{student $i$'s row in the network adjacency matrix}.\n\nMain text, p.~567: Our network question (i.e., ``whom did you choose to spend time with, face to face or online'') directly measures who is observing whose behavior.", "corrected": "SI p.~17: \\hl{For the primary social influence analyses,} we define the exposure mapping as a function, $f(\\mathbf{z}, \\theta_i)$, such that the parameter, $\\theta_i$, equals student $i$'s row in \\hl{the symmetrized network adjacency matrix. Let $B_{ij}=1$ if $i\\ne j$ and student $i$ nominated student $j$ before treatment, and $B_{ij}=0$ otherwise, and set $S_{ij}=\\mathbf{1}(B_{ij}+B_{ji}>0)$. Then $\\theta_i$ is the $i$th row of $S$, for the observed exposure and for the exposure probabilities: two students are peers when either nominated the other.} Further let $\\gamma$ be an $N$-length vector denoting referent vs.\\ nonreferent status, so that the $j$th element of $\\gamma$ takes on the value 1 if student $j$ has an indegree in the top decile of student $j$'s school, and 0 otherwise. \\hl{Compute indegree from nominations received from other students, counting repeated nominations separately.}\n\nTo be added on SI p.~17: \\hl{In the analysis by betweenness (Fig.~S6), $\\gamma$ is the indicator of high betweenness in the directed nominations, and $\\theta_i$ is unchanged. The alternative analysis of Fig.~S5 does not use $S$. There the peers of a seed, or of a seed-eligible student who was not selected, are the students whom that student nominated, each counted as often as listed.}\n\nMain text, p.~568, conditions (ii)--(iv): ``peers'' are \\hl{students joined by a nomination in either direction}.", "evidence": "The evidence is a check of the pages, a reading of the authors' scripts, a derivation, and a count on the public release of the data (ICPSR 37070, version 2).\n\n\\emph{The printed identities hold for $S$.} With $\\theta_i$ the $i$th row of $S$, I have\n\\[\n\\mathbf{z}'\\theta_i=\\sum_j z_jS_{ij},\\qquad (\\mathbf{z}\\circ\\gamma)'\\theta_i=\\sum_j z_j\\gamma_jS_{ij}.\n\\]\nThe first sum counts the seeds who nominated student $i$ or were nominated by her, and the second counts the social referents among them. The sentences of SI p.~17 on these two counts are true for $S$, and they are true for $B$ with a different meaning of ``peers''.\n\n\\emph{The exposure depends on the matrix.} A nomination in either direction can only add seed peers. For a student in a treated school who is not a seed, passing from $B$ to $S$ can move her from the condition with no seed peer to one of the two conditions with a seed peer, and from the condition with a seed peer and no social referent among her seed peers to the condition with a seed peer who is a social referent. It can move her in no other way. Under $S$ the probability of the condition with no seed peer is therefore at most its probability under $B$, and the probability of the condition with a seed peer who is a social referent is at least its probability under $B$.\n\n\\emph{The count.} I computed the observed exposure condition of each of the 24,191 students kept by the authors' rule on the public data, once with $B$ and once with $S$. The graph has 181,044 directed ties and 276,652 ordered pairs joined in either direction.\n\n\\begin{audittable}{lrr}{Observed exposure conditions of the 24,191 students on the public data (ICPSR 37070, version 2), under the two definitions of a peer. The third column is the definition in the authors' code.}\nExposure condition & Students whom $i$ nominated ($B$) & Either direction ($S$) \\\\\n\\hline\nControl school & 12,253 & 12,253 \\\\\nSeed & 728 & 728 \\\\\nTreated school, no seed peer & 7,330 & 5,754 \\\\\nSeed peer, none a social referent & 2,879 & 4,306 \\\\\nSeed peer who is a social referent & 1,001 & 1,150 \\\\\n\\end{audittable}\n\nThe condition differs for 1,609 students: 1,460 move from no seed peer to a seed peer who is not a social referent, 116 from no seed peer to a seed peer who is a social referent, and 33 from a seed peer who is not a social referent to one who is. In all, 1,576 students have a seed peer only because a seed nominated them.\n\nThese counts cover all students kept on the public data. They do not identify the sample of 2,451 students reported in the article, which has further restrictions (see \\fid{P037-N025}). I did not estimate any contrast with $B$ in place of $S$, and no published estimate is shown to change. The authors' own input files are not public, and ICPSR reports changes to some released variables. The check does not establish which matrix produced the printed estimates; the released code uses $S$.", "problem": "The network question is directed. Each student nominates the students with whom she chose to spend time, and the main text argues that the question measures who observes whose behavior. On the same page of the supplement, social referents are defined by indegree, the number of nominations received. In this context it is natural to take ``student $i$'s row'' to be the students whom student $i$ nominated, although the words ``network adjacency matrix'' also allow a matrix derived from the nominations.\n\nThe authors' code uses a derived matrix. Let $B_{ij}=1$ if $i\\ne j$ and student $i$ nominated student $j$, and $B_{ij}=0$ otherwise, and set\n\\[\nS_{ij}=\\mathbf{1}\\,(B_{ij}+B_{ji}>0).\n\\]\nThe script \\texttt{1. Sanitize and generate perms, probs.R} forms $S$ (lines 125--126) and takes $\\theta_i$ to be the $i$th row of $S$, for the observed exposure and for the simulated assignments from which the exposure probabilities are estimated (lines 170--181). Two students are peers when either nominated the other. Referent status is still computed from indegree in the directed nominations. Neither the article nor the supplement states this step.\n\nThe exposure conditions, the exposure probabilities and the analysis sample depend on which matrix is used. In the released code, the contrasts corresponding to Fig.~3 and Table S5 compare students by the seeds among those who nominated them or whom they nominated. They cannot be read as effects of exposure only to seeds whom the student chose to spend time with, and the analysis cannot be rebuilt from the printed description.", "what_survives": "The printed identities are correct for $S$. The released code uses $S$ in the same way for the observed exposure and for the simulated assignments, and it computes referent status from directed indegree. In that code, the same definition of a peer serves the placebo outcomes of Fig.~S4 and the analysis by betweenness of Fig.~S6. The school-level results of Figs.~1 and 2 and Tables S6 and S7 do not use the exposure conditions. The word ``peers'' in the statements of the main results is compatible with $S$. What changes is the reading of the within-school contrasts: they cannot be read as effects of seeds whom the student nominated. An analysis with $B$ would be a different analysis, with its own exposure probabilities and its own sample, and I did not carry it out." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N032", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the top of SI page 19. After the words 'define an estimated parameter vector associated with exposure condition d_k', a displayed formula sets xi-hat(d_k) equal to the arg min over xi(d_k) of the sum over i with D_i = d_k of 1 over pi_i(d_k) times the squared difference between y_i(d_k) and g(x_i, xi(d_k)). A red box marks the summation sign with its index 'i: D_i = d_k'. A second red box marks the words 'among students in a given level of exposure.' in the sentence below the formula.", "caption": "SI Appendix, p. 19 (supplement PDF p. 19). The marked sum runs over the students observed in exposure $d_k$ only, and the marked words say that the regression is fitted among the students in a given level of exposure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N033.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T27", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T27" }, { "id": "P037-T28", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T28" }, { "id": "P037-T42", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T42" } ], "headline": "The authors' code computes the numbers of Fig. 3 and Table S5 from one weighted regression over the four exposures with common covariate slopes, and not from the separate weighted regression within each exposure that the SI Appendix displays; the two procedures can give different estimates.", "id": "P037-N033", "location": "SI Appendix, p. 19 (supplement PDF p. 19), the two displays that define the estimator; the corresponding sentence of the article is on p. 568 (PDF p. 3), right column", "pdf_page": 19, "printed_page": "19", "reaches": [ { "effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P037-T27", "P037-T28", "P037-T42" ], "text": { "as_printed": "For covariate adjustment, define an estimated parameter vector associated with exposure condition $d_k$\n\\[\n\\widehat{\\xi}(d_k) = \\arg\\min_{\\xi(d_k)} \\hl{\\sum_{i:D_i=d_k}} \\frac{1}{\\pi_i(d_k)} \\bigl[y_i(d_k) - g\\left(\\mathbf{x}_i, \\xi(d_k)\\right)\\bigr]^2 ,\n\\]\nwhere $g(.)$ is the specification for the regression of $y_i(d_k)$ on $\\mathbf{x}_i$. $\\xi(d_k)$ is the parameter vector arising from an inverse probability weighted regression of outcomes on covariates \\hl{among students in a given level of exposure}. Then the regression estimator for the mean is\n\\[\n\\widehat{\\mu}_R(d_k) = \\hl{\\frac{1}{N_s} \\sum_{i:\\pi_i(d_k)>0,\\forall k\\in\\{C,S,NI,I\\}}} \\left[ \\mathrm{I}(D_i = d_k)\\, \\frac{y_i(d_k) - g\\bigl(\\mathbf{x}_i, \\widehat{\\xi}(d_k)\\bigr)}{\\hat{\\pi}_i(d_k)} + g\\bigl(\\mathbf{x}_i, \\widehat{\\xi}(d_k)\\bigr) \\right],\n\\]\nwhere $N_s$ denotes the number of students such that $0 < \\pi_i(d_k) < 1$ for all $i$ and for $k \\in \\{C,S,NI,I\\}$.", "corrected": "For covariate adjustment, define the estimated intercepts and the estimated common slope vector\n\\[\n\\bigl(\\widehat{\\alpha}_C, \\widehat{\\alpha}_S, \\widehat{\\alpha}_{NI}, \\widehat{\\alpha}_I, \\widehat{\\beta}\\bigr) = \\arg\\min_{\\alpha, \\beta} \\hl{\\sum_{i \\in A}} \\frac{1}{\\widehat{\\pi}_i(D_i)} \\bigl[y_i(D_i) - \\hl{\\alpha_{k(i)} - \\mathbf{x}_i^{\\top}\\beta}\\bigr]^2 ,\n\\]\nwhere $A$ is the outcome-specific set of students with \\hl{all four estimated exposure probabilities greater than $0.01$, no observed direct seed exposure, and a nonmissing outcome and processed pretreatment score; for peer conflict, the observed school mean must also exceed $0.02$} (\\fid{P037-N025}). Here $k(i)$ is the exposure observed for student $i$, so that $D_i = d_{k(i)}$, and $\\widehat{\\pi}_i(D_i)$ is the estimated probability of that exposure. The vector $\\mathbf{x}_i$ contains the estimated probabilities of referent peer, non-referent peer, school and both direct seed exposures; the pretreatment score for norm outcomes (a constant for other outcomes); and three background covariates: self-identification as white, whether friends say the student's house is nice, and whether a language other than English is spoken at home. Missing background values and pretreatment norm scores are replaced by their respective means before selection and fitting. This is one inverse probability weighted regression of outcomes on \\hl{indicators of the four exposures and on covariates, with the same covariate slopes in every exposure}. Then the regression estimator for the mean is\n\\[\n\\widehat{\\mu}_R(d_k) = \\hl{\\frac{1}{|A|} \\sum_{i \\in A}} \\bigl(\\hl{\\widehat{\\alpha}_k + \\mathbf{x}_i^{\\top}\\widehat{\\beta}}\\bigr), \\qquad k \\in \\{C, S, NI, I\\}.\n\\]\nSince the weighted residuals sum to zero within each exposure, adding the term $\\mathrm{I}(D_i = d_k)\\,\\bigl[y_i(D_i) - \\widehat{\\alpha}_k - \\mathbf{x}_i^{\\top}\\widehat{\\beta}\\bigr]/\\widehat{\\pi}_i(d_k)$ inside the sum does not change $\\widehat{\\mu}_R(d_k)$.\n\nThis describes an average over the realized, outcome-specific set $A$, rather than the fixed positive-probability population stated in the SI Appendix. Retaining that fixed population requires revising the averaging set and addressing missing outcomes and selection. If the separate regressions of the printed display are intended instead, the estimates of Fig.~3 and Table S5 and their standard errors have to be recomputed with them.", "evidence": "The evidence is a reading of the authors' code, an exact example and a recalculation on the public data (ICPSR 37070 V2). In the network script (\\texttt{3. Network results.R}), line 216 regresses the outcome on indicators of the exposures, the student's estimated exposure probabilities, the pretreatment outcome and three background covariates. The weights are the inverses of the estimated probabilities of the observed exposures, and the regression has no interaction of an exposure with a covariate. Lines 223--226 average the fitted values over the analysis set.\n\nThe residual term of the display does not account for the difference. A regression with an intercept for each exposure has weighted residuals that sum to zero within each exposure. When the fitting and averaging sets and weights agree, the augmented form of the display equals the average of the fitted values. The difference lies in the slopes.\n\nThe synthetic example has twelve units, each with probability $1/4$ of each exposure, one covariate $x$, and potential outcomes $y(d_C) = y(d_S) = y(d_{NI}) = x$ and $y(d_I) = 2x$. Let the units observed in $d_C$, $d_S$, $d_{NI}$ and $d_I$ have $x$ equal to $0,1,2$; $1,2,3$; $2,3,4$; and $3,4,5$. The mean of $x$ over the twelve units is $5/2$. The separate regressions fit $y = x$ in the first three exposures and $y = 2x$ in the fourth without error, and I obtain the means $5/2$, $5/2$, $5/2$ and $5$. The pooled regression has common slope\n\\[\n\\widehat{\\beta} = \\frac{2 + 2 + 2 + 4}{2 + 2 + 2 + 2} = \\frac{5}{4},\n\\]\nintercepts $-1/4$, $-1/2$, $-3/4$ and $3$, and means $23/8$, $21/8$, $19/8$ and $49/8$. In the display, the numerator sums over the four exposures the cross-products of $x$ and $y$ about their means within the exposure, and the denominator sums the corresponding sums of squares of $x$. The contrast of $d_I$ with $d_{NI}$ is $5/2$ under the separate regressions and $15/4$ under the pooled one, and the residual terms vanish in both. In a second synthetic example, with the probabilities $1/2$, $1/8$, $1/8$ and $1/4$ of a simplified school and seed design, I find the contrasts $5/2$ and $42/11$.\n\nIn a reconstruction on the public data I fitted both procedures with the analysis set and covariates of the code, using inverse probability weights from analytically computed exposure probabilities in both fits, so that only the constraint of common slopes changes.\n\n\\begin{audittable}{lrr}{Estimated mean under referent peer exposure minus estimated mean under non-referent peer exposure on the public data, from the pooled regression of the code and from separate regressions within each exposure.}\nOutcome & Pooled regression & Separate regressions \\\\\nFriends talk about conflict & $0.036$ & $0.027$ \\\\\nPrescriptive norms & $0.468$ & $0.487$ \\\\\nWristband & $0.108$ & $0.099$ \\\\\nPeer conflict & $-0.005$ & $-0.019$ \\\\\n\\end{audittable}\n\nAmong the 16 means, the largest difference is for prescriptive norms under school exposure $d_S$ (a treatment school and no seed peer), $-0.795$ pooled against $-0.905$ separate. This comparison uses the analysis set and not the population stated in the SI Appendix, and the public data do not reproduce Table S5 exactly; ICPSR reports that some released variables were changed. The authors' own data file is not public, and I recomputed no standard error or $P$ value for the separate regressions.", "problem": "The SI Appendix defines, for each exposure $d_k$, a weighted regression fitted to the students observed in that exposure, with its own parameter vector $\\xi(d_k)$. With a linear specification, each exposure then has its own covariate slopes. The authors' code fits one weighted regression to all students of the analysis set of the outcome (the students that the code keeps, described in \\fid{P037-N025}), with an intercept for each exposure and one vector of covariate slopes common to the four exposures. Its estimate of the mean under $d_k$ is the average, over the same set, of the fitted values with the intercept of $d_k$. The two procedures are different estimators. When the fitted slopes differ between exposures they ordinarily return different means, and the SI Appendix does not assume equal slopes. The displayed estimator therefore does not describe how Fig.~3 and Table S5 were computed. The sentence of the article (p.~568) speaks of ``a weighted least-squares regression'' and is compatible with either procedure.", "what_survives": "The displayed estimator is a legitimate estimator, and this discrepancy does not refute the randomization or exposure definitions. Pooling is not shown to bias the estimates, and no $P$ value or conclusion of the article is shown to change. The school-level regressions and the corrected 25\\% school-level conflict reduction use another estimator. What changes is the description of the method behind Fig.~3 and Table S5. The analysis set, over which the means are averaged, is the subject of \\fid{P037-N025}, and the regression used in the simulated assignments is the subject of \\fid{P037-N034}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N033", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Two excerpts separated by a grey horizontal rule. Above the rule, four lines beginning Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects; two red boxes mark the words to estimate the standard errors under the maintained hypothesis that, for all i. Below the rule, three lines; a red box marks the equation y sub i of d sub k equals y sub i of D sub i plus mu-hat sub R of d sub k minus mu-hat sub R of D sub i, which is followed by Aronow [52] derives the consistency of a variance estimator under this maintained hypothesis.", "caption": "SI Appendix, end of p. 19 (above the grey rule) and beginning of p. 20 (below it; PDF pp. 19–20). The marked words and the marked equation state how the standard errors are estimated: under the hypothesis that each potential outcome is the observed outcome shifted by the difference of the estimated means.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N034.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T33", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T33" }, { "id": "P037-T34", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T34" } ], "headline": "The randomization standard errors of the social-influence estimates do not follow the stated constant-effect procedure for the estimator used in the released code: the refits replace two probability controls of the point regression by their sum. Keeping the two controls in every refit recovers the variances of constant-effect imputation followed by that same regression; on the public data the standard errors move by less than 1%.", "id": "P037-N034", "location": "SI Appendix, pp. 19–20 (PDF pp. 19–20 of the SI Appendix), from “Randomization-based confidence intervals” to “using a normal approximation”; the same description on pp. 14–15; it underlies the standard errors of Table S5 (PDF p. 40) and the intervals of Fig. 3 (main text, p. 570)", "pdf_page": 19, "printed_page": "19–20", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T33", "P037-T34" ], "text": { "as_printed": "``Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects. Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) \\hl{to estimate the standard errors under the maintained hypothesis that, for all $i$, $y_i(d_k)=y_i(D_i)+\\hat\\mu_R(d_k)-\\hat\\mu_R(D_i)$}. Aronow [52] derives the consistency of a variance estimator under this maintained hypothesis. Wald-type confidence intervals and $p$-values are then constructed using a normal approximation.''", "corrected": "For the released pooled regression, keep $A$ and $x_i$ fixed across assignments and compute estimable averaged predictions as follows. For each of $B=1{,}000$ simulated assignments $\\mathbf{z}^1,\\ldots,\\mathbf{z}^B$, let $e_i^b$ denote the exposure indicators of student $i$ under $\\mathbf{z}^b$ and $D_i^b$ the exposure, and compute\n\\[\n(\\hat\\tau^b,\\hat\\gamma^b)=\\arg\\min_{\\tau,\\gamma}\\sum_{i\\in A}\\frac{1}{\\hat\\pi_i(D_i^b)}\\bigl(r_i-(e_i^b)^\\top\\tau-x_i^\\top\\gamma\\bigr)^2,\\qquad \\hat m^b(d_k)=\\hat\\tau^b_k+\\bar x^\\top\\hat\\gamma^b ,\n\\]\n\\hl{with the covariate vector $x_i$ of the point regression, in which $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$ enter as separate controls}. The standard error of $\\hat\\mu(d_k)$ is the standard deviation of $\\hat m^b(d_k)$ over $b$, and the standard error of $\\hat\\mu(d_k)-\\hat\\mu(d_l)$ is the standard deviation of $\\hat m^b(d_k)-\\hat m^b(d_l)$. This procedure has the same variances as imputing $y_i^b=y_i+\\hat\\mu(D_i^b)-\\hat\\mu(D_i)$ and refitting the same pooled point regression; it does not repair the separate discrepancy in the estimator displayed on p.~19. In the released code, replace \\texttt{pi.01} at line 257 of \\texttt{3. Network results.R} by \\hl{\\texttt{pi.01.inf + pi.01.ninf}}. The standard errors, intervals and Wald tests of Fig.~3 and Table S5 are then recomputed.", "evidence": "The evidence is a derivation, an exact counterexample and a recalculation on the public data.\n\nLet $A$ denote the fixed analysis set. For $i\\in A$, let $e_i$ denote the vector of indicators of the observed exposure $D_i$, and let $x_i$ denote the fixed covariates of the point regression, in which $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$ are separate entries. The point regression minimizes $\\sum_{i\\in A}\\hat\\pi_i(D_i)^{-1}(y_i-e_i^\\top\\tau-x_i^\\top\\gamma)^2$. Choose fixed coefficients $(\\hat\\tau,\\hat\\gamma)$ from this fit, giving estimates $\\hat\\mu(d_k)=\\hat\\tau_k+\\bar x^\\top\\hat\\gamma$ with $\\bar x=|A|^{-1}\\sum_{i\\in A}x_i$, and residuals $r_i=y_i-e_i^\\top\\hat\\tau-x_i^\\top\\hat\\gamma$. For a simulated assignment $\\mathbf{z}$ with exposure indicators $e_i^{\\mathbf{z}}$, the imputed outcome under the hypothesis of constant effects is\n\\[\ny_i^{\\mathbf{z}}=y_i+(e_i^{\\mathbf{z}}-e_i)^\\top\\hat\\tau=r_i+(e_i^{\\mathbf{z}})^\\top\\hat\\tau+x_i^\\top\\hat\\gamma .\n\\]\nLet $H_{\\mathbf{z}}$ denote the matrix with rows $((e_i^{\\mathbf{z}})^\\top,x_i^\\top)$ and $W_{\\mathbf{z}}$ the diagonal matrix of the positive weights of the simulated exposures. When $H_{\\mathbf{z}}$ has full column rank, set $L_{\\mathbf{z}}=(H_{\\mathbf{z}}^\\top W_{\\mathbf{z}}H_{\\mathbf{z}})^{-1}H_{\\mathbf{z}}^\\top W_{\\mathbf{z}}$. Since $L_{\\mathbf{z}}H_{\\mathbf{z}}$ is then the identity, I have\n\\[\nL_{\\mathbf{z}}\\,y^{\\mathbf{z}}=L_{\\mathbf{z}}\\,r+(\\hat\\tau,\\hat\\gamma).\n\\]\nI conclude that the estimated means from the refit of the imputed outcomes and from the refit of the residuals differ by the constant $\\hat\\mu(d_k)$ for every assignment, and their variances over the assignments are equal. When columns of $H_{\\mathbf{z}}$ are aliased, the same holds for averaged predictions whose rows lie in its row space (estimable predictions); these are independent of the choice of coefficients. The identity uses that the columns $x_i$ are in the refit. With the two probability controls replaced by their sum, the component $(\\hat\\gamma_I-\\hat\\gamma_{NI})\\hat\\pi_i(d_I)$ of the fitted values need not lie in the span of the refit, and the identity need not hold.\n\nThe following example shows that the variance can change. Eight units form four pairs. Two of the four pairs are drawn with equal probabilities to be exposed to seeds. In such a pair $j$, the first unit has exposure $d_I$ with probability $q_j$ and $d_{NI}$ otherwise, where $(q_1,q_2,q_3,q_4)=(1/10,1/5,3/10,2/5)$, and the second unit has the other of the two exposures. In each remaining pair the two units have $d_C$ and $d_S$, in either order with probability $1/2$. There are 96 assignments. For every unit the probabilities of $d_I$ and $d_{NI}$ sum to $1/2$, so that the sum control is constant and the separate controls add the direction $\\hat\\pi_i(d_I)$. The observed exposures are $(d_I,d_{NI},d_I,d_{NI},d_C,d_S,d_C,d_S)$ and the observed outcomes are $(0,2,1,0,3,1,2,4)$. For the point regression, I find the estimates $46/13$, $19/13$, $-47/39$ and $112/39$ for $d_C$, $d_S$, $d_{NI}$ and $d_I$. Over the 96 assignments, in exact rational arithmetic, the contrast between $d_I$ and $d_{NI}$ has the same variance for the refit of the imputed outcomes and for the refit of the residuals with the separate controls, with standard deviation $3.027$. For the refit of the residuals with the sum control the variance is $1453114769225810453/621302156766720000$, and the standard deviation is $1.529$. The example is an abstract design: its probability of control exposure is $1/4$, where the paper's is $1/2$, and it is not a model of the school experiment.\n\nUsing the authors' scripts with an adapter for the public data (ICPSR 37070 V2), I compared the two versions of the refit on the same 1,000 simulated assignments, for the four outcomes. ICPSR says some variables in this release were changed; the point estimates and analysis sets of this comparison differ from the published ones. The sixteen standard errors of the estimated means change by at most 0.80\\%, and the twelve standard errors of the contrasts between the referent exposure and each other exposure by at most 0.79\\%. None of the twelve Wald tests changes its decision at the level 0.05. The changes are not all in one direction. One estimated $P$ value, for the norms contrast between the referent and the nonreferent exposure, moves from 0.0101 to 0.0099; the change is smaller than its Monte Carlo standard error, and it does not establish a change in a published statement of significance.", "problem": "The passage describes the following computation: impute the potential outcomes of each student by the displayed equation, draw assignments, recompute the reported estimator for each draw, and take the standard deviation of the results. The released code differs from this in two respects, and one of them is a defect. The code refits the residuals of the point regression instead of imputed outcomes. For the released pooled regression, the two computations have the same variances when each refit keeps the analysis set and covariates and determines the averaged predictions uniquely. The refits change the covariates. The point regression (\\texttt{3. Network results.R}, line 216) controls separately for the estimated probabilities of the referent and the nonreferent exposure, $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$, and the refits (line 257) control only for their sum. The standard errors, intervals and Wald tests of Fig.~3 and Table S5 therefore come from the randomization distribution of a regression other than the one that produced the estimates. The estimator in question is the pooled regression of the released code; its difference from the estimator displayed on p.~19 is the subject of \\fid{P037-N033}.", "what_survives": "The point estimates of Fig.~3 and Table S5 do not depend on the refits and are unchanged. The school-level analysis, Fig.~2, and Tables S6 and S7 use a separate computation. The displayed equation is a coherent hypothesis, and refitting residuals is equivalent for variances to its imputation for the pooled regression under the conditions above. The check establishes no systematic direction of the change, no failure of coverage and no inconsistency of the variance estimator, and no conclusion of the paper is shown to change. The figures of less than 1\\% are an observation in one simulation on the public data, not a bound; the published standard errors can be recomputed only from the authors' own data file and saved simulations, which are not public. The correction concerns the controls of the refits alone. The simulated school draws are the subject of \\fid{P037-N031} and the description of the analysis set is the subject of \\fid{P037-N025}." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N034", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Fig. S2, Treatment-control balance across pretreatment covariates: a dot plot with fifteen rows, one for each covariate, and a horizontal axis for the p-value from 0 to 1, with a dashed vertical line at p = 0.05. A red box marks the row for the percentage receiving free or reduced-price lunch, the seventh from the top, whose dot is near 0.34. Only the dot of the last row, Network cluster coefficient, lies to the left of the dashed line.", "caption": "SI Appendix, Fig. S2 (PDF p. 25): the $p$-values of the fifteen balance tests. The marked row is labelled as free and reduced-price lunch together; the released code computes it from free lunch alone.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N035.png" } ], "formal_release": null, "formal_statements": [], "headline": "The row of Fig. S2 labelled “%Free reduced lunch” is computed in the released code from free lunch alone; the label should name the free lunch ratio, or the test should be recomputed with free and reduced-price lunch together.", "id": "P037-N035", "location": "SI Appendix, Fig. S2 (PDF p. 25 of the SI Appendix), row “%Free reduced lunch”, and the text below the figure", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Row label of Fig.~S2: ``\\%Free \\hl{reduced} lunch''\n\nText below the figure: ``To compute these $p$-values, we use the school-level treatment assignment to predict the 15 pretreatment school-level covariates as outcomes in our regression specification.''", "corrected": "If the released computation is kept, the row label reads ``\\hl{Free lunch count / population included in the study}'', and the text below the figure states that the row is the number of students with free lunch divided by the number of students in the study and that this computation does not test free and reduced-price lunch together.\n\nTo test free and reduced-price lunch together with the same denominator, replace line 133 with \\texttt{prFr <- (free\\_lunch + \\hl{reduced\\_lunch})/incsize}, document that denominator, and recompute the $p$-value and plotted point on the authors' own data. To match Table S1, use \\texttt{row\\_total} in place of \\texttt{incsize}, after checking the relevant population.", "evidence": "I checked the released code and an exact example. With $F$ and $R$ the numbers of students with free and with reduced-price lunch in a school and $D$ the denominator, the two measures differ by $(F+R)/D-F/D=R/D$, which is positive whenever $R>0$. The two tests are not interchangeable. In an invented example of eight schools in two blocks, with two treated schools in each block and $D=100$ in every school, the figure's regression specification has treatment coefficient $0$ for $F/D$ and $1/10$ for $(F+R)/D$, with the same robust variance $1/20000$; the squared test statistics are $0$ and $200$.\n\nI cannot reconstruct the published point from the public data or establish that this released version generated it. ICPSR 37070 V2 reports the lunch counts as interval codes, not as counts, and the authors' own data file is not public. With those codes in place of counts, an adapted calculation of the affected regression on the public data gives $p=0.40$ for free lunch alone and $p=0.45$ for free and reduced-price lunch together, keeping the same denominator of students in the study; these values illustrate the code and do not reconstruct the figure. The two scripts also use different denominators (students in the study in the figure code, total enrolment in Table S1).", "problem": "The released figure script sets the variable of this row to the number of students with free lunch divided by the number of students in the study (\\texttt{8. Balance test and plot.R}, line 133), and tests and plots it under the label ``\\%Free reduced lunch'' (lines 127--128 and 141--155). The codebook of the released data lists the recipients of free lunch and of reduced-price lunch as separate counts. The column of Table S1 with the same name is computed from the sum of the two counts (\\texttt{7. Blocking, randomization, and summary statistics.R}, line 152), and the study design names free or reduced lunch among the blocking variables (SI Appendix, p.~5). The released code therefore tests a different covariate from the one this row's label names.", "what_survives": "Changing this numerator alone leaves the other fourteen balance outcomes unchanged. The released scripts that compute the school-level and the social-influence estimates do not use this variable, and the finding does not call for revising an effect estimate or the randomization. The size of the correction to the plotted point is not established. If the row is recomputed, the statement below the figure that only one of the 15 covariates ``has a significant imbalance at the 0.05 level'' has to be checked again on the authors' own data." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N035", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Five lines of text from page 29 of the SI Appendix, beginning As an additional robustness check of our estimated network effects. A red box marks the words to assess within-school social influence effects. The following sentence reads: We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected.", "caption": "SI Appendix, p. 29 (PDF p. 29), first paragraph. The marked words give the purpose of the alternative analysis as assessing social influence effects; the next sentence defines its statistic as a difference between two means of outcomes of peers.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N036.png" }, { "alt": "Seven lines of text from page 32 of the SI Appendix, beginning Our results comport with the estimates found in our primary analysis. A red box marks the words restricted to social influence effects in the last sentence, which reads: Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology are restricted to social influence effects and do not incorporate any differences attributable to the climate of the school.", "caption": "SI Appendix, p. 32 (PDF p. 32), the paragraph after Fig. S5. The marked words say that the comparisons of this analysis are restricted to social influence effects.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N036-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P037-T35", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T35" }, { "id": "P037-T36", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T36" } ], "headline": "The alternative analysis of Fig. S5 is said to be restricted to social influence effects, but its statistic counts the outcomes of seeds among the peers and can respond to the effect of a student’s own treatment; its results are evidence against the null hypothesis of no effect of any kind.", "id": "P037-N036", "location": "SI Appendix, p. 29 (PDF p. 29 of the SI Appendix), first paragraph; Fig. S5 (PDF pp. 30–32) and the paragraph after it on p. 32", "pdf_page": 29, "printed_page": "29", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P037-T35", "P037-T36" ], "text": { "as_printed": "p.~29: ``As an additional robustness check of our estimated network effects, we employed a variant of the method used by (15) \\hl{to assess within-school social influence effects}. We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected. This difference-in-means serves as our test statistic.''\n\np.~32: ``Overall, we found a statistically significant \\hl{influence effect} on wristband usage ($p < 0.05$). Among the peers of referents, we found statistically significant \\hl{influence effects} on wristband usage ($p < 0.01$) and prescriptive and descriptive norms ($p < 0.05$). Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology are \\hl{restricted to social influence effects} and do not incorporate any differences attributable to the climate of the school.''", "corrected": "p.~29: ``\\ldots\\ a variant of the method used by (15), \\hl{a test of the sharp null hypothesis of no treatment effect within treated schools}. \\hl{The peers of a seed-eligible student are the students whom that student named, counted once for each nomination; seeds may be among them.} We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected.''\n\np.~32: ``Overall, we found a statistically significant \\hl{difference in} wristband usage ($p < 0.05$). Among the peers of referents, we found statistically significant \\hl{differences in} wristband usage ($p < 0.01$) and prescriptive and descriptive norms ($p < 0.05$). Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology \\hl{are made within treated schools} and do not incorporate any differences attributable to the climate of the school. \\hl{They do not by themselves separate the effect of a student's own assignment as a seed from the influence of seeds on other students. The $p$-value of 0.009 for wristbands among the peers of referents is an estimate from 1,000 permutations and lies within Monte Carlo error of 0.01.}''", "evidence": "I checked the released code, recomputed counts on the public data and enumerated an exact counterexample. Ten seed-eligible students of one treated school form five pairs, and each student names the other member of the pair. Five of the ten are drawn as seeds, with all 252 assignments equally likely, and the outcome of each student is $Y_i(\\mathbf{z})=z_i$. The outcome of a student depends on that student's own assignment alone, and seeds have no influence on other students. Suppose that the seeds are one member of each pair. The students named by seeds have outcome $0$ and the students named by the others have outcome $1$, so that the statistic equals $-1$. With the outcomes held fixed, the statistic has absolute value $1$ for 2 of the 252 assignments, and the two-tailed $p$-value of the permutation test is $2/252=1/126<0.01$. Over the 252 assignments of the experiment, the test reports $p<0.01$ with probability $32/252=8/63$.", "problem": "In the released code (\\texttt{6. Bond et al analysis.R}), the peers of a seed-eligible student are the students whom that student named. The statistic is the mean outcome over the nominations made by seeds minus the mean outcome over the nominations made by seed-eligible students who were not selected. A student enters a mean once for each nomination received, and students who are themselves seeds are kept. In a reconstruction on the public data (ICPSR 37070 V2), before removing missing outcomes, seeds receive 296 of the 5,377 nominations made by seeds (5.5\\%) and 359 of the 5,235 nominations made by seed-eligible students who were not selected (6.9\\%). The statistic therefore contains outcomes of treated students, and an effect of a student's own treatment can move it. The test of p.~29 is a test of the sharp null hypothesis of no effect, and it can reject that hypothesis when seeds have no influence on other students. The statements that the analysis assesses social influence effects, and is restricted to them, claim more than the test supports.", "what_survives": "The statistic and its permutation test are a coherent test of the sharp null hypothesis of no effect, and the statistically significant results of Fig.~S5 are evidence against that hypothesis. The primary analyses of social influence (Fig.~3 and Table S5) and the school-level analysis do not use this statistic, and this finding does not refute their estimates. It does not show that the differences of Fig.~S5 come from the direct effect of treatment. The printed $p=0.009$ for wristbands among peers of social referent seeds is an estimate from 1,000 permutations, with a Monte Carlo standard error of 0.003, and it is not shown to be wrong. In a reconstruction on the public data the five overall statistics of Fig.~S5 round to the printed values and the five statistics for social referents do not, so the public data cannot settle the second column's $p$-values from the authors' own files." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N036", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "The last two rows of Table S2, Middle East and Native American, each with four entries of the form mean (standard error) and a paired t-test p, followed by the note. Two red boxes mark the sentence Mean estimates include adjustment for block fixed effects, split across two lines.", "caption": "SI Appendix, Table S2 (PDF p. 37): the last two rows of the table and its note. The marked sentence says that the means of the table are adjusted for block fixed effects; the released code computes unadjusted means.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N037.png" } ], "formal_release": null, "formal_statements": [], "headline": "The means of Table S2 are unadjusted subgroup means; the sentence of the note that says they include adjustment for block fixed effects should be replaced.", "id": "P037-N037", "location": "SI Appendix, Table S2 (PDF p. 37 of the SI Appendix; the page carries no number), note, second sentence", "pdf_page": 37, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "``Note: Heteroskedasticity-robust standard errors clustered by school in parentheses. \\hl{Mean estimates include adjustment for block fixed effects.} Paired $t$-test $p$ refers to the $p$-value from a $t$-test comparing each variable's school-level means for social referents against the school-level means for non-social referents, pooled across eligible and non-eligible students.''", "corrected": "``Note: Heteroskedasticity-robust standard errors clustered by school in parentheses. \\hl{Mean estimates are unadjusted subgroup means.} Paired $t$-test $p$ refers to the $p$-value from a $t$-test comparing each variable's school-level means for social referents against the school-level means for non-social referents, pooled across eligible and non-eligible students.''", "evidence": "I checked the released code and recalculated the entries on the public data (ICPSR 37070 V2) using adapted portions of the authors' scripts. For a group $g$ with nonmissing values $y_1,\\ldots,y_n$, the script \\texttt{9. Elig and Ref descriptives.R} (lines 73--80) computes\n\\[\n\\widehat m_g=\\frac{1}{n}\\sum_{i=1}^{n}y_i,\\qquad \\widehat V(\\widehat m_g)=\\frac{M}{M-1}\\,\\frac{\\sum_{s}U_s^2}{n^2},\\qquad U_s=\\sum_{i\\in s}(y_i-\\widehat m_g),\n\\]\nwhere $s$ runs over the $M$ schools with nonmissing values in the group; the standard error comes from a regression on a constant alone. The script fits no block effect and applies no weights. On the public data, all 124 means and all 124 standard errors that can be compared (31 rows and four groups) agree with the printed entries at two decimals. The rows Age, Other, Middle East and Native American cannot be compared, because the public release masks or recodes these variables. For illustration, standardizing each subgroup to equal weights for the 14 school blocks changes 62 of the 124 means at two decimals on the public data. In the row White, for example, the printed means are 0.70, 0.63, 0.71 and 0.66, and the means with equal block weights are 0.70, 0.61, 0.68 and 0.62 on the public data.", "problem": "Table S2 reports a mean and a standard error for each of four groups of students (eligible or ineligible, social referent or not) and each of 35 variables. The note says that the means are adjusted for block fixed effects. The released code computes each mean as the average of the variable over the students of the group with a nonmissing value, and no block enters the computation. The standard errors are clustered by school, as the note says.", "what_survives": "The 31 rows reproduced on the public data retain their unadjusted subgroup means and school-clustered standard errors. The main text (p.~567) cites the table for differences between social referents and other students in dating, in compliments on their house and in having older siblings; on the public data the unadjusted differences are 0.23, 0.11 and 0.08, and their direction is as described. The released scripts contain no computation of the column ``Paired $t$-test $p$'', and I have not settled those values. The school-level and social-influence estimates do not use this table." }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N037", "work": { "arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/" }, "work_id": "P037" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P037.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_witness_design_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1036", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "The covered component is the population coefficient and equality criterion in (iv). The explicit potential-outcome slope and average causal-curve identities are assigned to witness_mixture_causal_curve in the other split; no limiting OLS or finite-sample OLS expectation is asserted here.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_witness_mixture", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1159", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.stratum_moment_certificates", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L534", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/P037_FiniteDesign.lean:1103 gives both the schoolwise difference b1*(c-c′) and the average causal curve tau+b1*(c-1/5), exactly the causal-curve component of T1(iv). The potential-outcome definition at line 1088 is the specified affine witness schedule, with arbitrary b1 and b2. Extending that constructed schedule to all real compositions preserves every feasible composition and adds no hypothesis.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mixture_causal_curve", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1103", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_normal_equation_certificate", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1072", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "The matrix-inverse limit on a total coefficient map is excluded by the finite restriction.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_ols_probability_limit", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3294", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_zero_composition_effect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1013", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_count_pushforward", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L555", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.stratum_count_nonneg", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L582", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_count_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L601", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_count_nonneg", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L650", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_count_common_support", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L662", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_factorization", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L691", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_const", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L705", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_first", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L713", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_second", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L722", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_first_sq", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L731", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_second_sq", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L744", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L758", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_first_self", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L768", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_second_self", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L774", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pair_count_mean_reverse_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L781", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_split", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L845", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_state_probability_law", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L854", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_add", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L867", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_sub", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L872", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_mul", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L877", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_mul_right", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L882", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_const", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L887", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_sum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L891", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_coordinate_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L900", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_block_cross", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L940", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_composition_treatment", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L958", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_div", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L971", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_observed_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L977", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Proves the matrix/vector component of the validator witness, not the expectation of finite-sample OLS or a full refutation by itself.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_expected_normal_equations", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L990", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_mixture_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1120", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mixture_rhs", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1141", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "conditional_on": [], "credit_reason": "This is the exact finite witness to insufficiency of the three expressly listed conditions, not a refutation of the qualified response model, whose sufficient restriction the schedule violates. The row does not claim the determinant formula, mixture formula or OLS limit. Target metadata asymptotic=false conflicts with its part (v), but no limit is claimed by this row. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_witness_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1189", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6b135f3f4f96bd54f68e6d0cbb7b1a2034de9d90344907ae1a66c37526438fb3", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T1" }, "html": { "body": "

SI Appendix pp. 13–14, repeated pp. 15–16 and Table S7 (p. 42); main p. 568. The exact four-school hierarchical design has the stated treatment and conditional seed moments, feasible supports and 454 positive-mass collapsed states. Composition has zero causal slope, yet the expected regression has the displayed matrices and unique slope -16/31; the general response schedule gives (15 b1 + 16 b2)/31. Under independent block repetition, every selection of actual least-squares minimizers has composition coefficient converging in probability to -16/31. Minimizers exist, and singular pooled Gram matrices have probability tending to zero. The usual measure-theoretic conditions are assumed.

", "headline": "SI Appendix, Eq. (2) with the sentence ‘As E[Cij]∝CijE\\mathrm{E}[C_{ij}] \\propto C^E_{ij}E[Cij​]∝CijE​, controlling for CijEC^E_{ij}CijE​ ensures that any random deviations in CijC_{ij}Cij​ from E[Cij]\\mathrm{E}[C_{ij}]E[Cij​] may be interpreted causally’ (repeated with Eqs. (4) and (6)), and main Methods, ‘As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group’. The witness is the validators' construction and is not in the paper", "location": "SI Appendix pp. 13–14 (supplement PDF pp. 13–14), repeated at SI pp. 15–16 and in the note to Table S7 (SI p. 42); main p. 568 (PDF p. 3)" }, "id": "P037-T1", "text": { "body": "SI Appendix pp. 13–14, repeated pp. 15–16 and Table S7 (p. 42); main p. 568. The exact four-school hierarchical design has the stated treatment and conditional seed moments, feasible supports and 454 positive-mass collapsed states. Composition has zero causal slope, yet the expected regression has the displayed matrices and unique slope -16/31; the general response schedule gives (15 b1 + 16 b2)/31. Under independent block repetition, every selection of actual least-squares minimizers has composition coefficient converging in probability to -16/31. Minimizers exist, and singular pooled Gram matrices have probability tending to zero. The usual measure-theoretic conditions are assumed.", "headline": "SI Appendix, Eq. (2) with the sentence ‘As $\\mathrm{E}[C_{ij}] \\propto C^E_{ij}$, controlling for $C^E_{ij}$ ensures that any random deviations in $C_{ij}$ from $\\mathrm{E}[C_{ij}]$ may be interpreted causally’ (repeated with Eqs. (4) and (6)), and main Methods, ‘As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group’. The witness is the validators' construction and is not in the paper", "location": "SI Appendix pp. 13–14 (supplement PDF pp. 13–14), repeated at SI pp. 15–16 and in the note to Table S7 (SI p. 42); main p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T1" }, { "declarations": [ { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "The imposed response moment is exactly the sufficient assumption in current target A, so it is not an illicit assumption of an independently claimed causal result.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.correctly_specified_composition_model", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1300", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Generalization preserves the B4 specialization; the historical eligible-interaction name imposes no q-centering premise.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.adding_eligible_interaction_equivalent", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1427", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.augmented_normal_iff_residual", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1378", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.centered_composition_identifies_common_slope", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1403", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "This is the pointwise B1 algebraic component, not a separate stochastic identification claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_innovation_decomposition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1330", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "The row covers the centered part of B2; RandomizedComposition is the target conditional mean-zero premise, not a disguised assumed slope result.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_innovation_orthogonality", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1336", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "The fixed four-school kernel matches standing assumption S1; the B=0 case adds a trivial case without restricting positive B.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_add", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1239", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_block_constant", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1275", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_left_add", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1372", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Auxiliary fact used in the expected normal-equation expansions in (A) and (B); no causal identification is asserted by this row alone.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_mul", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1245", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_sub", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1367", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.withinBlocksCross_sum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1250", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "The baseline printed-column solution for Y0 is not a separate conjunct. With X=(D,q,Z) and Y1=Y0+Z/5, its normal equations at theta are equivalent to those for Y1 at theta+(0,0,1/5); the displayed solution therefore gives (-3/31,-15/62,-16/31) for Y0, exactly the missing comparison in (C), with existence and uniqueness preserved. This is an algebraically equivalent presentation, not an excluded case; the claim-stage verdict is unchanged.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_repaired_regression_comparisons", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1608", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_left_scaled_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1487", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_right_scaled_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1497", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_mean_scalar_right_factor", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1507", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_repaired_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1519", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_repaired_rhs", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1570", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.witness_equations_matrix_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1597", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P037_FiniteDesign.lean:3044 exports normalization, the marginal composition mean, and its centered second moment under the uniform product seed law. T2(DM) instead also states the treated-school conditional moments and concludes (R), E[D_i(C_i−μ_i) | D] = 0; the row expressly credits these consequences. No school-assignment variable, joint law, independence hypothesis, conditional moment, or RandomizedComposition conclusion occurs in this declaration. The marginal formulas, including the singleton convention, match, but the missing independent-coupling assertion leaves the credited conditional conclusions unformalized.', 'suggested_fix': 'Extend the exported statement to a joint law with the specified per-school stratified seed law independent of the school-assignment vector D, and assert the conditional mean and variance on D_i = 1 and E[D_i(C_i−μ_i) | D] = 0. Do not add independence of seed draws across schools, which this component does not require.', 'second_review': 'P037/P037_FiniteDesign.lean:3044 exports normalization, the marginal composition mean and the centered second moment over StratifiedSeedState, defined at P037/P037_FiniteDesign.lean:2865. T2(DM) additionally requires the mean and variance conditional on D_i = 1 and concludes (R), E[D_i(C_i−μ_i) | D] = 0. The declaration contains no school-assignment vector, joint law or independence hypothesis connecting seed draws to that vector, so the required conditional conclusions cannot be extracted from its marginal model. Extend the statement with the independent coupling and its conditional conclusions; the missing credited component is substantive.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P037.stratified_composition_design_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3008", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "“Stratum” is read as a nonempty eligible group, as required by the target ratios K_h/N_h; total seeds are already strictly positive in DM.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.design_mean_equal_sampling_fraction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3025", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "This is the exact synthetic count example in DM, not a refutation row or a verified observation about a particular school.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.unequal_strata_design_mean_certificate", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3043", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "“Nonzero in general” in the description means the shift need not vanish; the theorem does not claim it is nonzero for every input.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_centering_special_case", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3051", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "conditional_on": [], "credit_reason": "Positive variation is a target hypothesis; no claim about finite-sample unbiasedness, estimator convergence or application-specific causal identification is credited here.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.design_mean_centered_common_slope", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L3062", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:9df278272aa42895e7c6db4a6e75e1fcb0f7d68b1fdbcce1a62a56c870bbce0f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T2", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T2" }, "html": { "body": "

SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); main p. 568 (PDF p. 3). Finite seed-law moments and explicitly imposed response and randomization conditions yield the stated population regression identification and witness comparisons.

", "headline": "Expected-equations identification under a response moment condition, and optional centered or augmented regressions", "location": "SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); main p. 568 (PDF p. 3)" }, "id": "P037-T2", "text": { "body": "SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); main p. 568 (PDF p. 3). Finite seed-law moments and explicitly imposed response and randomization conditions yield the stated population regression identification and witness comparisons.", "headline": "Expected-equations identification under a response moment condition, and optional centered or augmented regressions", "location": "SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); main p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T2" }, { "declarations": [ { "claim_revision_id": "catalog:4eff6dd71ea3b2afff114053f7b1799a90fb82ca95960ae35e1bac51c176cff2", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P037_ReportedNumbers.lean:38 proves the rounding witnesses, their rational Wald ratios, and comparisons with an arbitrary real c satisfying 19/10 ≤ c ≤ 2. T3(iv) additionally asserts that the actual normal 0.975 quantile lies in this interval and concludes 2(1−Φ(z_A)) < 1/20 < 2(1−Φ(z_B)). The declaration neither defines the Gaussian CDF or its critical value nor asserts those bounds or P-value inequalities. The record’s permission to present a conditional version, and the description’s disclosure of this restriction, do not change the unconditional Gaussian assertions in the supplied claim text for the credited component (ii)–(iv).', 'suggested_fix': 'Keep the current result as the rational-comparison helper, and extend the credited statement with the standard Gaussian CDF, a proved bracket for its 0.975 critical value, and the two resulting normal P-value inequalities.', 'second_review': 'P037/P037_ReportedNumbers.lean:38 proves the rounding witnesses and opposite comparisons only for an arbitrary c satisfying 19/10 ≤ c ≤ 2; waldRatio and printsTableS6 at lines 19 and 23 contain no Gaussian distribution. T3(iv) also asserts the actual normal 0.975 quantile lies in this bracket and concludes the two normal P-value inequalities. Disclosure of the bracket premise in the description does not establish those unconditional assertions in the supplied claim text. Keep the rational comparison helper and extend the credited statement with a proved Gaussian bracket and the resulting P-value inequalities.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P037.rounded_wald_opposite_decisions", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L38", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:4eff6dd71ea3b2afff114053f7b1799a90fb82ca95960ae35e1bac51c176cff2", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.significance_claims_incompatible", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L29", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4eff6dd71ea3b2afff114053f7b1799a90fb82ca95960ae35e1bac51c176cff2", "conditional_on": [], "credit_reason": "This row is the attainability direction of T3(v); R224-04 supplies the outer-bound direction. It does not claim the separate Gaussian quantile calculation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.wald_envelope_attainable", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L67", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4eff6dd71ea3b2afff114053f7b1799a90fb82ca95960ae35e1bac51c176cff2", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.wald_rounding_envelope", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L51", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:4eff6dd71ea3b2afff114053f7b1799a90fb82ca95960ae35e1bac51c176cff2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T3" }, "html": { "body": "

main p. 569; 2018 Correction p. E3598; SI Table S6 p. 41. The two significance descriptions are incompatible. The strict rounding bins give the same displayed estimate and standard error for two witnesses with opposite actual two-sided normal Wald decisions. The exact ratio envelope is (11/7,13/5), containing the unique normal 0.975 quantile, which lies strictly between 1.95996 and 1.95997. The normal-tail comparisons at 1.9 and 2 and the six stated five-decimal approximations are certified; rounding alone therefore does not determine significance. Decimal tail approximations mean error less than 0.000005. The usual measure-theoretic conditions are assumed.

", "headline": "Results, first paragraph, ‘(P<0.05P < 0.05P<0.05, heteroskedasticity-robust Wald test)’, against the last sentence of the 2018 Correction; corrected claim: the two descriptions cannot both hold for one test, and the published estimate and standard error (Table S6) do not decide between them", "location": "main p. 569 (PDF p. 4); 2018 Correction, p. E3598 (correction PDF p. 1); SI Table S6, p. 41 (supplement PDF p. 41)" }, "id": "P037-T3", "text": { "body": "main p. 569; 2018 Correction p. E3598; SI Table S6 p. 41. The two significance descriptions are incompatible. The strict rounding bins give the same displayed estimate and standard error for two witnesses with opposite actual two-sided normal Wald decisions. The exact ratio envelope is (11/7,13/5), containing the unique normal 0.975 quantile, which lies strictly between 1.95996 and 1.95997. The normal-tail comparisons at 1.9 and 2 and the six stated five-decimal approximations are certified; rounding alone therefore does not determine significance. Decimal tail approximations mean error less than 0.000005. The usual measure-theoretic conditions are assumed.", "headline": "Results, first paragraph, ‘($P < 0.05$, heteroskedasticity-robust Wald test)’, against the last sentence of the 2018 Correction; corrected claim: the two descriptions cannot both hold for one test, and the published estimate and standard error (Table S6) do not decide between them", "location": "main p. 569 (PDF p. 4); 2018 Correction, p. E3598 (correction PDF p. 1); SI Table S6, p. 41 (supplement PDF p. 41)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T3" }, { "declarations": [ { "claim_revision_id": "catalog:3f02e58d28ae04d707432b1d918a16b0d195a15270f31809e84111cbacd9e5fd", "conditional_on": [], "credit_reason": "This is a complete witness against the target’s explicitly specified direct-independent-uniform rule, evaluated against the genuine Dirichlet law. It does not identify the authors’ implementation or any coefficient-standard-error discrepancy.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.uniform_weights_not_bayesian_bootstrap_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B4b.lean#L403", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:3f02e58d28ae04d707432b1d918a16b0d195a15270f31809e84111cbacd9e5fd", "conditional_on": [], "credit_reason": "This is an auxiliary algebraic fact in T4(i),(ii), genuinely used in the witness argument; its deterministic generalization preserves the intended specialization.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.bootstrap_ratio_cancellation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L64", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:3f02e58d28ae04d707432b1d918a16b0d195a15270f31809e84111cbacd9e5fd", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.exponential_witness_integrals", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L85", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:3f02e58d28ae04d707432b1d918a16b0d195a15270f31809e84111cbacd9e5fd", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.uniform_witness_integral", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L76", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:3f02e58d28ae04d707432b1d918a16b0d195a15270f31809e84111cbacd9e5fd", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T4" }, "html": { "body": "

SI Appendix Table S6 note p. 41 and Table S7 note p. 42. For every number of schools at least two, normalized independent uniforms give ratio-event probability 1/6. Normalized independent standard exponentials have the constant-density Dirichlet(1,...,1) simplex law and give probability 1/4, also the probability E1 < E2/3 and the integral of exp(-v)(1-exp(-v/3)) on the positive half-line. These are different weight laws, with probability difference -1/12. All normalization and ratio denominators are positive almost surely. Positive common rescaling, including normalization, preserves the unique WLS coefficients for positive weights and full column rank. The usual measure-theoretic conditions are assumed.

", "headline": "Notes to Tables S6 and S7, ‘Bayesian bootstrap standard errors generated using 5,000 replicates, applying weights to each school drawn from the standard uniform distribution’: an event on which normalized independent uniform weights and Bayesian-bootstrap weights have different probabilities (the validator's witness; not in the paper)", "location": "SI Appendix, Table S6 note, p. 41, and Table S7 note, p. 42 (supplement PDF pp. 41–42)" }, "id": "P037-T4", "text": { "body": "SI Appendix Table S6 note p. 41 and Table S7 note p. 42. For every number of schools at least two, normalized independent uniforms give ratio-event probability 1/6. Normalized independent standard exponentials have the constant-density Dirichlet(1,...,1) simplex law and give probability 1/4, also the probability E1 < E2/3 and the integral of exp(-v)(1-exp(-v/3)) on the positive half-line. These are different weight laws, with probability difference -1/12. All normalization and ratio denominators are positive almost surely. Positive common rescaling, including normalization, preserves the unique WLS coefficients for positive weights and full column rank. The usual measure-theoretic conditions are assumed.", "headline": "Notes to Tables S6 and S7, ‘Bayesian bootstrap standard errors generated using 5,000 replicates, applying weights to each school drawn from the standard uniform distribution’: an event on which normalized independent uniform weights and Bayesian-bootstrap weights have different probabilities (the validator's witness; not in the paper)", "location": "SI Appendix, Table S6 note, p. 41, and Table S7 note, p. 42 (supplement PDF pp. 41–42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T4" }, { "declarations": [ { "claim_revision_id": "source-claim-import-2026-11-08:P037-T5:1", "conditional_on": [ "L. Devroye, Non-Uniform Random Variate Generation, Springer, 1986, Chapter XI, Theorem 4.1, p. 594" ], "credit_reason": "Its distributional equality requires a published theorem as a premise, which is not added to this Challenge.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.exponentials_give_bayesian_bootstrap_conditional", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Cond/CondP037P041Proofs.lean#L57", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T5:1", "conditional_on": [], "credit_reason": "This is only the scalar gamma-integral step of T5(b); the statement does not itself supply the change of variables, Jacobian or joint density.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.simplex_gamma_integral", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L134", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T5:1", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.uniform_log_gives_exponential", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L96", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T5:1", "conditional_on": [], "credit_reason": "The full-rank and positivity conditions appear in T4(iv), explicitly cross-referenced by T5(d), so they are target assumptions rather than added restrictions. The singular-matrix zero-inverse convention does not affect the target domain.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.weighted_least_squares_scale", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Bootstrap.lean#L169", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "source-claim-import-2026-11-08:P037-T5:1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T5" }, "html": { "body": "

SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42). The Bayesian bootstrap uses the probability law with constant density (n-1)! in the first n-1 open-simplex coordinates, supported on positive unit-sum weights. For every n≥2, independent Exp(1) multipliers normalized by their sum have this law; independent U(0,1) draws transformed by minus-log give independent Exp(1) inputs and the same construction. The forward normalization map in coordinate order (p₁,…,pₙ₋₁,t) and its inverse have absolute Jacobian determinants t^{-(n-1)} and t^{n-1}, respectively. The joint density is t^{n-1} exp(-t) on the open simplex times t>0 and zero elsewhere; its t-integral is (n-1)!. The n spacings of n-1 ordered independent uniforms, including both endpoint gaps, have the same simplex law. Direct independent uniform multipliers, even after normalization, have a different law (ratio-event probabilities 1/6 versus 1/4). Positive common scaling preserves the unique full-rank WLS coefficient, so raw and normalized uniform replicates agree. The usual measure-theoretic conditions are assumed. These facts justify the conditional relabel/recompute correction, without identifying the rule actually used or supplying replacement standard errors.

", "headline": "Notes to Tables S6 and S7 and the three rows ‘Bayesian Bootstrap SE’, corrected: the weight law of the Bayesian bootstrap, and what the described uniform weights are", "location": "SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42)" }, "id": "P037-T5", "text": { "body": "SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42). The Bayesian bootstrap uses the probability law with constant density (n-1)! in the first n-1 open-simplex coordinates, supported on positive unit-sum weights. For every n≥2, independent Exp(1) multipliers normalized by their sum have this law; independent U(0,1) draws transformed by minus-log give independent Exp(1) inputs and the same construction. The forward normalization map in coordinate order (p₁,…,pₙ₋₁,t) and its inverse have absolute Jacobian determinants t^{-(n-1)} and t^{n-1}, respectively. The joint density is t^{n-1} exp(-t) on the open simplex times t>0 and zero elsewhere; its t-integral is (n-1)!. The n spacings of n-1 ordered independent uniforms, including both endpoint gaps, have the same simplex law. Direct independent uniform multipliers, even after normalization, have a different law (ratio-event probabilities 1/6 versus 1/4). Positive common scaling preserves the unique full-rank WLS coefficient, so raw and normalized uniform replicates agree. The usual measure-theoretic conditions are assumed. These facts justify the conditional relabel/recompute correction, without identifying the rule actually used or supplying replacement standard errors.", "headline": "Notes to Tables S6 and S7 and the three rows ‘Bayesian Bootstrap SE’, corrected: the weight law of the Bayesian bootstrap, and what the described uniform weights are", "location": "SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T5" }, { "declarations": [ { "claim_revision_id": "catalog:79615c10e88175f0aec79749fbe3276c27d9dc0cf3f66f4526e777392d1018df", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.corrected_reduction_arithmetic", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L103", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:79615c10e88175f0aec79749fbe3276c27d9dc0cf3f66f4526e777392d1018df", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.mixed_basis_cannot_round_to_25", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L135", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:79615c10e88175f0aec79749fbe3276c27d9dc0cf3f66f4526e777392d1018df", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.rounded_totals_reduction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L116", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:79615c10e88175f0aec79749fbe3276c27d9dc0cf3f66f4526e777392d1018df", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T6" }, "html": { "body": "

main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6); 2018 Correction, p. E3598 (correction PDF p. 1). The printed totals and rounding bins give the exact corrected percentage arithmetic and exclude deriving 25 percent from the two rounded mixed-basis numbers.

", "headline": "Abstract, Significance statement, Results (first paragraph) and Discussion: the ‘30%’ reduction in disciplinary reports, corrected to 25% (2018 Correction)", "location": "main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6); 2018 Correction, p. E3598 (correction PDF p. 1)" }, "id": "P037-T6", "text": { "body": "main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6); 2018 Correction, p. E3598 (correction PDF p. 1). The printed totals and rounding bins give the exact corrected percentage arithmetic and exclude deriving 25 percent from the two rounded mixed-basis numbers.", "headline": "Abstract, Significance statement, Results (first paragraph) and Discussion: the ‘30%’ reduction in disciplinary reports, corrected to 25% (2018 Correction)", "location": "main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6); 2018 Correction, p. E3598 (correction PDF p. 1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T6" }, { "declarations": [ { "claim_revision_id": "catalog:c0a293e2197087da7c5962cd488e6314caed1f5a9612a1bac6c7580e133cd755", "conditional_on": [], "credit_reason": "A witness row for the literal printed simultaneous exposure cases, not proof of a positive exposure mapping and not a claim about the implemented classifier.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.printed_exposure_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L215", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0a293e2197087da7c5962cd488e6314caed1f5a9612a1bac6c7580e133cd755", "conditional_on": [], "credit_reason": "This is the full failure of the printed simultaneous-case mapping, not a claim that no tie-breaking classifier can be defined.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.printed_exposure_not_single_valued", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L242", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0a293e2197087da7c5962cd488e6314caed1f5a9612a1bac6c7580e133cd755", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.printed_exposure_overlaps", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L229", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c0a293e2197087da7c5962cd488e6314caed1f5a9612a1bac6c7580e133cd755", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T7" }, "html": { "body": "

SI Appendix p. 17 (supplement PDF p. 17). Both feasible witness configurations satisfy two distinct printed exposure cases, and the overlap rules preclude a single-valued mapping.

", "headline": "SI Appendix, the display ‘The exposure mapping is:’, third line (dNId_{NI}dNI​), as printed: two feasible configurations that satisfy two cases at once (the validator's witnesses; not in the paper)", "location": "SI Appendix p. 17 (supplement PDF p. 17)" }, "id": "P037-T7", "text": { "body": "SI Appendix p. 17 (supplement PDF p. 17). Both feasible witness configurations satisfy two distinct printed exposure cases, and the overlap rules preclude a single-valued mapping.", "headline": "SI Appendix, the display ‘The exposure mapping is:’, third line ($d_{NI}$), as printed: two feasible configurations that satisfy two cases at once (the validator's witnesses; not in the paper)", "location": "SI Appendix p. 17 (supplement PDF p. 17)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T7" }, { "declarations": [ { "claim_revision_id": "catalog:619600845baef9913c5cceef60528b89bc074d847af3d4a0a053284b24e9feeb", "conditional_on": [], "credit_reason": "The corrected target itself quantifies over all nonnegative counts r<=t, so omitting the survey’s peer cap is not a narrowing. Unique existence exports both exhaustiveness and disjointness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.corrected_exposure_partition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L255", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:619600845baef9913c5cceef60528b89bc074d847af3d4a0a053284b24e9feeb", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T8" }, "html": { "body": "

SI Appendix p. 17 (supplement PDF p. 17). The five corrected exposure predicates partition every feasible configuration and give school and direct exposure on the two witnesses.

", "headline": "SI Appendix, the exposure mapping, corrected: with the repaired dNId_{NI}dNI​ condition the five cases partition the assignments generated by the design", "location": "SI Appendix p. 17 (supplement PDF p. 17)" }, "id": "P037-T8", "text": { "body": "SI Appendix p. 17 (supplement PDF p. 17). The five corrected exposure predicates partition every feasible configuration and give school and direct exposure on the two witnesses.", "headline": "SI Appendix, the exposure mapping, corrected: with the repaired $d_{NI}$ condition the five cases partition the assignments generated by the design", "location": "SI Appendix p. 17 (supplement PDF p. 17)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T8" }, { "declarations": [ { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "conditional_on": [], "credit_reason": "This is an acceptable generalization. Positive J excludes total-division artifacts; the binary common-mean hypotheses force 0≤π≤1.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.monte_carlo_frequency_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1764", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.exposure_frequency_printed_example", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L284", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "conditional_on": [], "credit_reason": "The expectation/probability identification in T9(i) is definitional, not an extra assumed statistical result; no target cases are lost.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.exposure_indicator_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L306", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "conditional_on": [], "credit_reason": "This is an acceptable generalization, and finite laws cover the paper's finite assignment design and every finite vector of simulated indicators. Metadata inconsistency: T9.asymptotic is false although this component and the target expressly include J→∞ convergence.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.monte_carlo_frequency_consistent", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1837", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.printed_monte_carlo_last_draw", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L294", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c3efef2d3c5fdc7daad4e8e5aaf32da4993598ba50dfdb61753181c2286c045c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T9", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T9" }, "html": { "body": "

SI Appendix p. 18 and p. 14 (supplement PDF pp. 18 and 14). Independent finite-design exposure draws give unbiased frequency estimates with exact variance, a Chebyshev error bound and consistency, whereas the printed formula uses the last draw.

", "headline": "SI Appendix, the simulated exposure probability π^i(dk)\\hat\\pi_i(d_k)π^i​(dk​) (printed twice), corrected: the Monte Carlo frequency of the draws that induce exposure dkd_kdk​", "location": "SI Appendix p. 18 and p. 14 (supplement PDF pp. 18 and 14)" }, "id": "P037-T9", "text": { "body": "SI Appendix p. 18 and p. 14 (supplement PDF pp. 18 and 14). Independent finite-design exposure draws give unbiased frequency estimates with exact variance, a Chebyshev error bound and consistency, whereas the printed formula uses the last draw.", "headline": "SI Appendix, the simulated exposure probability $\\hat\\pi_i(d_k)$ (printed twice), corrected: the Monte Carlo frequency of the draws that induce exposure $d_k$", "location": "SI Appendix p. 18 and p. 14 (supplement PDF pp. 18 and 14)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T9" }, { "declarations": [ { "claim_revision_id": "catalog:416a5e2fa238f32d665545cdb334f108df5716bb184594217a92c0bbf67bfe13", "conditional_on": [], "credit_reason": "The witness refutes the assertion that this block randomization makes schools independent. It does not refute an LLN or CLT whose own independence and other hypotheses hold, or establish any estimator's inconsistency.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.within_block_dependence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L133", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:416a5e2fa238f32d665545cdb334f108df5716bb184594217a92c0bbf67bfe13", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.between_block_independence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L163", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:416a5e2fa238f32d665545cdb334f108df5716bb184594217a92c0bbf67bfe13", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.block_design_constraints", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L121", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:416a5e2fa238f32d665545cdb334f108df5716bb184594217a92c0bbf67bfe13", "conditional_on": [], "credit_reason": "An arbitrary ambient probability space is unnecessary for a claim depending only on the finite joint law of four binary indicators; the finite representation excludes no joint law in T10(iv).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.fixed_count_forces_dependence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1870", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:416a5e2fa238f32d665545cdb334f108df5716bb184594217a92c0bbf67bfe13", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T10", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T10" }, "html": { "body": "

SI Appendix p. 19 (supplement PDF p. 19); the design is on SI p. 5 and in Fig. S1, SI p. 24. Two-of-four school assignment creates negative within-block treatment and control-exposure covariance, while independently randomized blocks have the stated product law.

", "headline": "SI Appendix, ‘Consistency and asymptotic normality directly follow ... given a growing sequence of independent schools of finite size, as holds by construction given our between-school randomization’: the treatment indicators of two schools in one block are dependent (the validator's computation; not in the paper)", "location": "SI Appendix p. 19 (supplement PDF p. 19); the design is on SI p. 5 and in Fig. S1, SI p. 24" }, "id": "P037-T10", "text": { "body": "SI Appendix p. 19 (supplement PDF p. 19); the design is on SI p. 5 and in Fig. S1, SI p. 24. Two-of-four school assignment creates negative within-block treatment and control-exposure covariance, while independently randomized blocks have the stated product law.", "headline": "SI Appendix, ‘Consistency and asymptotic normality directly follow ... given a growing sequence of independent schools of finite size, as holds by construction given our between-school randomization’: the treatment indicators of two schools in one block are dependent (the validator's computation; not in the paper)", "location": "SI Appendix p. 19 (supplement PDF p. 19); the design is on SI p. 5 and in Fig. S1, SI p. 24" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T10" }, { "declarations": [ { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/Wave1/Gap/GLP037B1.lean:885 matches the fitted-limit component of corrected T11(8): coefficient consistency, a remainder negligible after multiplication by sqrt(B), estimator consistency, and the Gaussian limit under the stated variance condition. Its uniform bounds, positive Gram limit, positive estimated probabilities, and simulation rate are hypotheses of the corrected target itself. B+1 indexing and coordinatewise formulations preserve the target’s range and fixed-dimensional convergence. The zero inverse on singular finite samples is an extension on asymptotically negligible samples, not an additional full-rank assumption.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.fitted_block_estimator_limit_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GLP037B1.lean#L877", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "A harmless generalization of T11(3), not reduced coverage. The original row description is accurate; the disagreement with the record is only the verdict label.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.augmented_contribution_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L337", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "A harmless generalization of individual fixed-prediction unbiasedness in T11(1). Neither this lemma nor its row description asserts unbiasedness of a prediction fitted on the same random sample; the disagreement with the record is only the verdict label.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.augmented_contribution_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L324", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.augmented_subtraction_certificates", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2711", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Auxiliary fact for the centered-total bound, not a standalone proof of T11(6). Design normalization forces a nonempty state type, so the absolute bound already implies A>=0.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.bounded_design_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2166", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.bounded_design_variance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2035", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.coordinateFactor_other", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L419", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.coordinateFactor_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L427", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.coordinateFactor_self", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L412", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designMean_add", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L60", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designMean_const", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L72", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designMean_mul", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L77", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designMean_sub", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L66", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designMean_sum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L83", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designVariance_centered", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L91", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designVariance_mul", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1925", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designVariance_nonneg", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2026", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.designVariance_sum_covariances", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L361", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Auxiliary fact for the corrected finite-design consistency argument; it is not a statement about the fitted estimator by itself.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.finite_chebyshev", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1740", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.finite_product_factorization", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L375", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "The component is exactly the two identities, not fitted-regression consistency. Coordinates replace the dot product, q may be zero, and the statement actually covers unbounded covariate arrays and N=0 as well; the positive-p hypothesis remains stated even though the algebra also works with total inverses at zero.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.fitted_mean_subtraction_identities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2750", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P037_FiniteDesign.lean:2688 assumes UniformBlockBounds, whose definition at P037_FiniteDesign.lean:1991 requires 0 < c, whereas T11 states only N_B ≥ cB without a sign condition. The unconditional contrast-consistency conclusion therefore covers fewer cases than the stated sufficient argument: a fixed nonempty retained set with a nonconstant exposure estimator, and no retained students in newly added blocks, satisfies N_B ≥ −B but need not become consistent. Taking the second exposure’s outcomes and predictions to be zero makes the same example a contrast. The Gaussian branch has its own positive linear variance lower bound, which already supplies population growth; the material discrepancy concerns the unconditional consistency branch and an omitted assumption in the corrected target.', 'suggested_fix': 'In T11’s mathematical sufficient conditions, replace “constants m, c, η, K” with “constants m, c, η, K with c > 0”; keep the Lean statement. This is a condition on the corrected sufficient argument, not a condition asserted by the original paper.', 'second_review': \"At P037/P037_FiniteDesign.lean:2688, contrast consistency uses UniformBlockBounds, which explicitly requires 0 < c at P037/P037_FiniteDesign.lean:1992. T11's corrected sufficient conditions instead say only 'N_B ≥ cB', and (7) transfers consistency to contrasts. With c = -1, one retained student in the first block with π = 1/2, y = 1 and g = 0, no retained students in later blocks, and zero outcomes and predictions for the second exposure, the contrast estimate remains 2I around target 1 and is inconsistent. Add c > 0 to the corrected claim text and retain the Lean statement.\", 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.independent_blocks_exposure_contrasts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2690", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P037_FiniteDesign.lean:2137 obtains consistency under UniformBlockBounds, including 0 < c at P037_FiniteDesign.lean:1992. T11(5) asserts consistency assuming only N_B ≥ cB, which admits nonpositive c and a fixed retained population. For example, retain one student in the first block and none in later blocks, with exposure probability 1/2, outcome 1 and prediction 0: the oracle mean stays 2I, its target is 1, and its error exceeds 1/2 with probability one for every B. All the stated uniform bounds can hold with c = −1, so the target’s sufficient conditions omit an essential positive-growth assumption that Lean correctly supplies.', 'suggested_fix': 'Add c > 0 to T11’s mathematical sufficient conditions before (1)–(8); the Lean consistency theorem then matches (5).', 'second_review': \"P037/P037_FiniteDesign.lean:2137 proves consistency using UniformBlockBounds, including 0 < c at P037/P037_FiniteDesign.lean:1992, while T11(5)'s corrected conditions specify only 'N_B ≥ cB'. A single retained student in the first block, none in subsequent blocks, π = 1/2, y = 1 and g = 0 satisfies that written growth inequality with c = -1, but the estimate 2I differs from its target 1 by exactly 1 for every B. Thus the correction's sufficient conditions are false as written: add c > 0 to the target and preserve the Lean theorem.\", 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.independent_blocks_oracle_consistent", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2136", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'Although P037_FiniteDesign.lean:2457 inherits 0 < c from UniformBlockBounds at P037_FiniteDesign.lean:1991, T11(6)’s own positive variance lower bound already implies the existence of such a growth constant. With H = K + 2K/η, bounded contributions give s_B² ≤ Σ_b |S_b|²H² ≤ mH²N_B, so s_B² ≥ σ₀²B > 0 implies N_B ≥ [σ₀²/(mH²)]B with mH² > 0. Neither conclusion of this declaration contains c, so using this derived positive witness preserves every claimed design even when an originally supplied lower-bound constant was nonpositive. The Gaussian CDF limit and the zero Lindeberg numerator with positive total variance express T11(6) without a material loss of cases.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.independent_blocks_oracle_normal", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2459", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_block_bounds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2050", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "This is the centered-block component of (6), not the full CLT. The bound is everywhere, including zero-mass assignments, which loses no supported assignment.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_block_centered_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2181", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "The pi_def field is an exposure-probability definition, not an assumed unbiasedness result. Arbitrary within-block dependence and arbitrary deterministic predictions remain allowed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_block_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L1998", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/P037_FiniteDesign.lean:2521 supplies the unbiasedness, exact block-variance sum, doubled block bound, and variance bound credited to the finite-contrast component of T11(7). SecondRowExposure uses the same block assignments and retained students for both exposures, so no independence between exposure-specific estimators is added. The abstract finite laws include the target design, and the extra empty-population division convention excludes no nonempty target case.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_contrast_finite_facts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2523", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_lindeberg_vanish", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2199", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P037_FiniteDesign.lean:2108 assumes 0 < c through UniformBlockBounds at P037_FiniteDesign.lean:1991, while T11(3)–(5)’s stated conditions require only N_B ≥ cB. This matters for the exported variance and Chebyshev bounds, although the individual block bounds match. Take one retained student per block with independent I of probability 1/2, y = 1 and g = 0, and choose m = K = 1, η = 1/2 and c = −100: all the written target inequalities hold, but Var(μ̃_B) = 1/B exceeds its claimed bound 25/(10000B). At target index B = 1 and ε = 1/2 the error probability is one, whereas the claimed Chebyshev upper bound is 1/100; the corrected target therefore needs the missing sign condition.', 'suggested_fix': 'Add c > 0 to T11’s mathematical sufficient conditions; retain the valid Lean finite bounds and their existing constants.', 'second_review': \"P037/P037_FiniteDesign.lean:2108 exports the finite bounds under UniformBlockBounds, whose definition at P037/P037_FiniteDesign.lean:1992 requires 0 < c. T11(3)–(5) assume only 'N_B ≥ cB' before claiming the c-dependent variance and Chebyshev bounds. With one student per independent block, π = 1/2, y = 1, g = 0, m = K = 1, η = 1/2 and c = -100, the variance is 1/B rather than at most 25/(10000B), and at B = 1 and ε = 1/2 the claimed probability bound is 1/100 although the error probability is 1. The target's corrected sufficient argument needs c > 0; the existing Lean bounds should remain.\", 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_row_finite_bounds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2107", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/P037_FiniteDesign.lean:2010 states the two credited finite identities from T11(1),(4): unbiasedness and the exact variance sum divided by N squared. Arbitrary finite block-local exposure laws generalize the target without changing its retained-population estimand, and allowing an empty population adds only a trivial boundary case. The inherited modelling paragraph is stale in calling the fitted-limit handling of singular inverses an unproved obligation: P037/Wave1/Gap/GLP037B1.lean:885 supplies that fitted-limit result.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_row_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2009", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.oracle_variance_rate_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L2079", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.product_design_coordinate_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L438", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P037.product_design_coordinate_pair", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L457", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Taking other coordinate functions to be 1 yields every finite-subfamily factorization, including exposure-event indicators, so this auxiliary fact supplies the joint-independence content needed by (2). The stale fitted-limit modelling sentence does not affect it.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.product_design_mean_product", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L393", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P037-T11:accepted", "conditional_on": [], "credit_reason": "Supporting component or algebra is already consumed by the assembled declarations; it is not invoked separately.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.product_design_sum_variance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_FiniteDesign.lean#L496", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "CT-02:P037-T11:accepted", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T11", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T11" }, "html": { "body": "

SI Appendix p. 19 (supplement PDF p. 19). Independent retained-block totals give the finite unbiasedness and variance identities, bounds, consistency, Gaussian limits, contrast results, and fitted-estimator limits under the stated regularity conditions.

", "headline": "SI Appendix, the asymptotic justification of the social-influence estimator, corrected: consistency and asymptotic normality along a growing sequence of independently randomized four-school blocks", "location": "SI Appendix p. 19 (supplement PDF p. 19)" }, "id": "P037-T11", "text": { "body": "SI Appendix p. 19 (supplement PDF p. 19). Independent retained-block totals give the finite unbiasedness and variance identities, bounds, consistency, Gaussian limits, contrast results, and fitted-estimator limits under the stated regularity conditions.", "headline": "SI Appendix, the asymptotic justification of the social-influence estimator, corrected: consistency and asymptotic normality along a growing sequence of independently randomized four-school blocks", "location": "SI Appendix p. 19 (supplement PDF p. 19)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T11" }, { "declarations": [ { "claim_revision_id": "catalog:9ec2ca7b2c7be7d195dce095142d99748286895207570f257843252a9ab187e9", "conditional_on": [], "credit_reason": "The extension at zero is a representation convention, and the target itself defines the abstract rule at every positive integer even where a roster of that size could not contain 40 eligible pupils.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.seed_count_bounds_monotone", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L160", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9ec2ca7b2c7be7d195dce095142d99748286895207570f257843252a9ab187e9", "conditional_on": [], "credit_reason": "The mathematical component is exact. Target-text inconsistency: “no more than 64” is an upper bound, not equality to 64; these below-550 examples satisfy that printed bound and do not themselves refute it. The row is credited for the explicitly corrected rule and examples, not a full refutation of the printed upper-bound sentence.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.seed_count_cap_and_examples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L182", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9ec2ca7b2c7be7d195dce095142d99748286895207570f257843252a9ab187e9", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.seed_count_shares", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L199", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9ec2ca7b2c7be7d195dce095142d99748286895207570f257843252a9ab187e9", "conditional_on": [], "credit_reason": "The equation is the condition 10+3(n−200)/175=k+1/2 after multiplication by 350. Its all-integer extension loses no school-size case.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.seed_interpolation_no_tie", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L155", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:9ec2ca7b2c7be7d195dce095142d99748286895207570f257843252a9ab187e9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T12" }, "html": { "body": "

SI Appendix p. 6 (supplement PDF p. 6); the related sentence is on main p. 567 (PDF p. 2). The corrected integer interpolation rule has no ties, is monotone with the stated bounds and cap threshold, and gives the listed counts and eligible shares. The listed examples refute the reading that the upper-cap parenthesis assigns exactly 64 seed-eligible students to every school with fewer than 550 students. The literal upper bound of no more than 64 for those schools remains true under this rule.

", "headline": "SI Appendix, Study design, the size of the seed-eligible group: ‘no more than 64 seed-eligible students (schools with less than 550 students)’, corrected", "location": "SI Appendix p. 6 (supplement PDF p. 6); the related sentence is on main p. 567 (PDF p. 2)" }, "id": "P037-T12", "text": { "body": "SI Appendix p. 6 (supplement PDF p. 6); the related sentence is on main p. 567 (PDF p. 2). The corrected integer interpolation rule has no ties, is monotone with the stated bounds and cap threshold, and gives the listed counts and eligible shares. The listed examples refute the reading that the upper-cap parenthesis assigns exactly 64 seed-eligible students to every school with fewer than 550 students. The literal upper bound of no more than 64 for those schools remains true under this rule.", "headline": "SI Appendix, Study design, the size of the seed-eligible group: ‘no more than 64 seed-eligible students (schools with less than 550 students)’, corrected", "location": "SI Appendix p. 6 (supplement PDF p. 6); the related sentence is on main p. 567 (PDF p. 2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T12" }, { "declarations": [ { "claim_revision_id": "catalog:f558cb85be6b26c3aad4acf74313cb690c8224ed78e2de36dbcea18f6d921a82", "conditional_on": [], "credit_reason": "The baseline interval is an explicit hypothesis in the target itself. Interval membership of 0.095, 0.57 and 0.60 is all the target's mathematical clauses assert; no actual fit or attainment claim is credited.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_prediction_intervals", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L224", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f558cb85be6b26c3aad4acf74313cb690c8224ed78e2de36dbcea18f6d921a82", "conditional_on": [], "credit_reason": "The target explicitly asks for arithmetic on the printed constants, not recovery of unrounded regression estimates.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.composition_prediction_printed_arithmetic", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_ReportedNumbers.lean#L215", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f558cb85be6b26c3aad4acf74313cb690c8224ed78e2de36dbcea18f6d921a82", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T13" }, "html": { "body": "

main p. 569 (PDF p. 4), with Fig. 2D on the same page; SI Table S7, p. 42 (supplement PDF p. 42). The printed coefficients and explicit coefficient and control-level intervals imply the stated treatment prediction and relative-reduction intervals.

", "headline": "Results, second paragraph: ‘each student was disciplined on average 0.08 times during the year’ when 20% of the group are social referent seeds, corrected to about 0.095", "location": "main p. 569 (PDF p. 4), with Fig. 2D on the same page; SI Table S7, p. 42 (supplement PDF p. 42)" }, "id": "P037-T13", "text": { "body": "main p. 569 (PDF p. 4), with Fig. 2D on the same page; SI Table S7, p. 42 (supplement PDF p. 42). The printed coefficients and explicit coefficient and control-level intervals imply the stated treatment prediction and relative-reduction intervals.", "headline": "Results, second paragraph: ‘each student was disciplined on average 0.08 times during the year’ when 20% of the group are social referent seeds, corrected to about 0.095", "location": "main p. 569 (PDF p. 4), with Fig. 2D on the same page; SI Table S7, p. 42 (supplement PDF p. 42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T13" }, { "declarations": [ { "claim_revision_id": "catalog:01921c2f5d6ffecb3e344a86f0ab3351f83700703207fd184fb909d38cad239b", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:01921c2f5d6ffecb3e344a86f0ab3351f83700703207fd184fb909d38cad239b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T14", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T14" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Main Methods, ‘As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group’, with SI Appendix Eq. (2) (repeated as Eq. (4)), its interpretation, and Eq. (6), as printed", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); Table S7 note, SI p. 42" }, "id": "P037-T14", "text": { "body": "No current theorem is supplied for this target.", "headline": "Main Methods, ‘As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group’, with SI Appendix Eq. (2) (repeated as Eq. (4)), its interpretation, and Eq. (6), as printed", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 13–14 and 15–16 (supplement PDF pp. 13–16); Table S7 note, SI p. 42" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T14" }, { "declarations": [ { "claim_revision_id": "catalog:3492e5016524949a6689340fa9fc637af02a924ef8b861f460b4fdeba654a7f0", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:3492e5016524949a6689340fa9fc637af02a924ef8b861f460b4fdeba654a7f0", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T15", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T15" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Results, first paragraph (‘the treatment significantly reduced ... (P<0.05P < 0.05P<0.05, heteroskedasticity-robust Wald test)’), and the last sentence of the 2018 Correction (‘remain statistically insignificant at the α\\alphaα=0.05 level’), as printed", "location": "main p. 569 (PDF p. 4); 2018 Correction, p. E3598 (correction PDF p. 1)" }, "id": "P037-T15", "text": { "body": "No current theorem is supplied for this target.", "headline": "Results, first paragraph (‘the treatment significantly reduced ... ($P < 0.05$, heteroskedasticity-robust Wald test)’), and the last sentence of the 2018 Correction (‘remain statistically insignificant at the $\\alpha$=0.05 level’), as printed", "location": "main p. 569 (PDF p. 4); 2018 Correction, p. E3598 (correction PDF p. 1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T15" }, { "declarations": [ { "claim_revision_id": "catalog:cbad045dfa91ed9e911a1a0d2116493a06d1f3480cd3e89746703328a234ba85", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:cbad045dfa91ed9e911a1a0d2116493a06d1f3480cd3e89746703328a234ba85", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T16", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T16" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Tables S6 and S7, the rows ‘Bayesian Bootstrap SE’ and the sentence of each note that describes them, as printed", "location": "SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42)" }, "id": "P037-T16", "text": { "body": "No current theorem is supplied for this target.", "headline": "Tables S6 and S7, the rows ‘Bayesian Bootstrap SE’ and the sentence of each note that describes them, as printed", "location": "SI Appendix, Table S6, p. 41, and Table S7, p. 42 (supplement PDF pp. 41–42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T16" }, { "declarations": [ { "claim_revision_id": "catalog:5540fb3b653b88a3bc3c1f0347fbb5a665d0a2e3187b62b1495c376d808c0c91", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:5540fb3b653b88a3bc3c1f0347fbb5a665d0a2e3187b62b1495c376d808c0c91", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T17", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T17" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Abstract, Significance statement, Results (first paragraph) and Discussion: the reduction of disciplinary reports ‘by 30%’, as printed", "location": "main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6)" }, "id": "P037-T17", "text": { "body": "No current theorem is supplied for this target.", "headline": "Abstract, Significance statement, Results (first paragraph) and Discussion: the reduction of disciplinary reports ‘by 30%’, as printed", "location": "main p. 566 (PDF p. 1), p. 569 (PDF p. 4) and p. 571 (PDF p. 6)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T17" }, { "declarations": [ { "claim_revision_id": "catalog:1f16a961892f66530500a33ae3d2877fd55b8f0c0c803aa9b21b0cd203e5d536", "conditional_on": [], "credit_reason": "Fixing the data expresses the target's conditional law; it does not impose a restriction on possible observed full-rank designs or responses. The exported conclusion is covariance equality, with no claim of exact test calibration.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.wild_bootstrap_is_hc0", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L387", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1f16a961892f66530500a33ae3d2877fd55b8f0c0c803aa9b21b0cd203e5d536", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_leverage_facts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L405", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1f16a961892f66530500a33ae3d2877fd55b8f0c0c803aa9b21b0cd203e5d536", "conditional_on": [], "credit_reason": "No extra full-rank assumption is introduced: it is in T18's setting. Probability normalization and finite second moments prevent nonintegrable-integral defaults, while σ²=0 and empty dimensions are allowed without making the theorem degenerate-only.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.hc0_homoskedastic_shortfall", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L418", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1f16a961892f66530500a33ae3d2877fd55b8f0c0c803aa9b21b0cd203e5d536", "conditional_on": [], "credit_reason": "This is explicitly an auxiliary finite row lemma. Its positive norm premise is satisfied by every actual OLS coefficient row under full column rank, since AX=I; combining it with the matrix moment identities yields the coefficient bias ratio for σ²>0.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.hc0_row_bias_factor", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L442", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1f16a961892f66530500a33ae3d2877fd55b8f0c0c803aa9b21b0cd203e5d536", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T18", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T18" }, "html": { "body": "

main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42). Residual multiplier refits have HC0 covariance, and homoskedastic HC0 has the stated positive semidefinite leverage shortfall.

", "headline": "Main Methods, ‘We computed confidence intervals and P values using robust SEs under a normal approximation’, with the heteroskedasticity-robust Wald tests of Results (first paragraph) and of the Fig. 2 caption and the rows ‘Wild Bootstrap SE’ of Tables S6 and S7, corrected: what the released procedure computes (the HC0 variance with a normal reference), that the wild bootstrap as coded has exactly the HC0 covariance, and that HC0 is biased downward under homoskedastic errors by an amount fixed by the leverages", "location": "main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42)" }, "id": "P037-T18", "text": { "body": "main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42). Residual multiplier refits have HC0 covariance, and homoskedastic HC0 has the stated positive semidefinite leverage shortfall.", "headline": "Main Methods, ‘We computed confidence intervals and P values using robust SEs under a normal approximation’, with the heteroskedasticity-robust Wald tests of Results (first paragraph) and of the Fig. 2 caption and the rows ‘Wild Bootstrap SE’ of Tables S6 and S7, corrected: what the released procedure computes (the HC0 variance with a normal reference), that the wild bootstrap as coded has exactly the HC0 covariance, and that HC0 is biased downward under homoskedastic errors by an amount fixed by the leverages", "location": "main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T18" }, { "declarations": [ { "claim_revision_id": "catalog:20c3afb3e67690d23702a84e3808d81e5455a929baad9869428e85b0a38cb980", "conditional_on": [], "credit_reason": "This is an actual normal-equation/inverse/HC0 certificate, not a definition that stipulates the fitted covariance.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.band_ols_hc0_certificate", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L535", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:20c3afb3e67690d23702a84e3808d81e5455a929baad9869428e85b0a38cb980", "conditional_on": [], "credit_reason": "Evaluating the control residual fraction from these explicit rows loses no case: the data and all residuals are fixed. These are variance comparisons, not assertions of exact confidence coverage.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.band_counterexample_values", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L564", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:20c3afb3e67690d23702a84e3808d81e5455a929baad9869428e85b0a38cb980", "conditional_on": [], "credit_reason": "The centered coefficient is explicitly (1/10,1/4,3/2), and the inverse and normal equations certify it rather than assuming the fit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.band_centered_refit_certificate", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L754", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:20c3afb3e67690d23702a84e3808d81e5455a929baad9869428e85b0a38cb980", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T23", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T23" }, "html": { "body": "

main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6. The exact 56-school fit has level variances 1/2800, contrast variance 1/1400 and the residual-control construction 1/5400.

", "headline": "Fig. 2, caption and panel keys (‘Control schools' predicted mean (and 95% CI)’, ‘Treatment schools' predicted mean (and 95% CI)’; ‘95% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit’), and the school-level panels of Fig. S6; the released code draws the treatment band from the standard error of a contrast and the control band from a residual variance divided by 56. An exact data set in which both differ from the robust variance of the predicted mean (the validator's certificate; not in the paper)", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "id": "P037-T23", "text": { "body": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6. The exact 56-school fit has level variances 1/2800, contrast variance 1/1400 and the residual-control construction 1/5400.", "headline": "Fig. 2, caption and panel keys (‘Control schools' predicted mean (and 95% CI)’, ‘Treatment schools' predicted mean (and 95% CI)’; ‘95% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit’), and the school-level panels of Fig. S6; the released code draws the treatment band from the standard error of a contrast and the control band from a residual variance divided by 56. An exact data set in which both differ from the robust variance of the predicted mean (the validator's certificate; not in the paper)", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T23" }, { "declarations": [ { "claim_revision_id": "catalog:8b78b78b542a94db8526bbc9cddcd2f98769cc07119be22a13c30e0a8d976627", "conditional_on": [], "credit_reason": "The generalization preserves the required specialization because every real HC0 covariance is symmetric. No PSD assumption is needed for this identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.prediction_mean_contrast_variance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L490", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8b78b78b542a94db8526bbc9cddcd2f98769cc07119be22a13c30e0a8d976627", "conditional_on": [], "credit_reason": "The full-rank hypothesis is already explicit in T24, so it is not an added restriction. It prevents singular-inverse conventions from substituting for the claimed OLS algebra.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.centered_refit_is_contrast", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L713", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8b78b78b542a94db8526bbc9cddcd2f98769cc07119be22a13c30e0a8d976627", "conditional_on": [], "credit_reason": "The arbitrary zbar and coefficient vector are appropriate for this prediction-row auxiliary identity; the row does not claim to prove interval coverage.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.interaction_prediction_rows", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Inference.lean#L780", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:8b78b78b542a94db8526bbc9cddcd2f98769cc07119be22a13c30e0a8d976627", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T24", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T24" }, "html": { "body": "

main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6. A same-sample centered interaction refit estimates the contrast, whose variance differs algebraically from a treatment predicted-mean variance.

", "headline": "Fig. 2 and the school-level panels of Fig. S6, the 95% bands, corrected: the robust interval of each predicted mean, and what the two constructions of the released code compute", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "id": "P037-T24", "text": { "body": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6. A same-sample centered interaction refit estimates the contrast, whose variance differs algebraically from a treatment predicted-mean variance.", "headline": "Fig. 2 and the school-level panels of Fig. S6, the 95% bands, corrected: the robust interval of each predicted mean, and what the two constructions of the released code compute", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T24" }, { "declarations": [ { "claim_revision_id": "catalog:ba2ac128c5764eff68da030fe1feaa2e27cdbcb7f2718bf6a0ee79913871fb5f", "conditional_on": [], "credit_reason": "A synthetic counterexample component, not an empirical statement about the study and not by itself the asymptotic result.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.outcome_selection_finite_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L68", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ba2ac128c5764eff68da030fe1feaa2e27cdbcb7f2718bf6a0ee79913871fb5f", "conditional_on": [], "credit_reason": "Matches only the credited asymptotic component. Metadata inconsistency: T19 has asymptotic=false although part (v) and this row explicitly concern a probability limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.outcome_selection_regression_limit_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B1.lean#L43", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ba2ac128c5764eff68da030fe1feaa2e27cdbcb7f2718bf6a0ee79913871fb5f", "conditional_on": [], "credit_reason": "A harmless algebraic generalization of the block-centering auxiliary fact; it is not being credited as a complete refutation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.selected_block_crossproducts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L111", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:ba2ac128c5764eff68da030fe1feaa2e27cdbcb7f2718bf6a0ee79913871fb5f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T19", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T19" }, "html": { "body": "

main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10); the regressions are SI Eqs. (1)–(6), pp. 13–16. Outcome-threshold selection changes the kept contrast expectation and the weighted-block probability limit despite constant effects.

", "headline": "Main Methods, ‘(Administrative data were available for 49 of the 56 schools, and we restricted our attention to these schools for analyses of conflict-related events.)’, and the Fig. 1 caption; in the released code a school enters the conflict analyses when its mean count exceeds 0.02. An exact design in which that rule, with complete reporting and one constant effect in every school, does not return the effect (the checker's construction; not in the paper)", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10); the regressions are SI Eqs. (1)–(6), pp. 13–16" }, "id": "P037-T19", "text": { "body": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10); the regressions are SI Eqs. (1)–(6), pp. 13–16. Outcome-threshold selection changes the kept contrast expectation and the weighted-block probability limit despite constant effects.", "headline": "Main Methods, ‘(Administrative data were available for 49 of the 56 schools, and we restricted our attention to these schools for analyses of conflict-related events.)’, and the Fig. 1 caption; in the released code a school enters the conflict analyses when its mean count exceeds 0.02. An exact design in which that rule, with complete reporting and one constant effect in every school, does not return the effect (the checker's construction; not in the paper)", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10); the regressions are SI Eqs. (1)–(6), pp. 13–16" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T19" }, { "declarations": [ { "claim_revision_id": "catalog:197e294e03f6d71ce9c5dfd9635d9a2270eaa9164468217f5fcc69c23126319a", "conditional_on": [], "credit_reason": "No extra positivity for dropped units and no unverified independence of inclusion and exposure is assumed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.selection_ipw_identities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L125", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:197e294e03f6d71ce9c5dfd9635d9a2270eaa9164468217f5fcc69c23126319a", "conditional_on": [], "credit_reason": "Finiteness is unnecessary for this deterministic sufficient condition. The statement does not certify that the empirical outcome screen is assignment-invariant.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.threshold_assignment_invariant", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L193", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:197e294e03f6d71ce9c5dfd9635d9a2270eaa9164468217f5fcc69c23126319a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T20", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T20" }, "html": { "body": "

main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10). Fixed inclusion identifies the kept-population mean by inverse-probability weighting, whereas varying inclusion introduces conditional inclusion factors.

", "headline": "Main Methods, the sentence on the 49 schools, and the Fig. 1 caption, corrected: the rule by which the released code selects the schools of the conflict analyses, and the condition on that rule under which the selected analysis keeps its causal reading", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10)" }, "id": "P037-T20", "text": { "body": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10). Fixed inclusion identifies the kept-population mean by inverse-probability weighting, whereas varying inclusion introduces conditional inclusion factors.", "headline": "Main Methods, the sentence on the 49 schools, and the Fig. 1 caption, corrected: the rule by which the released code selects the schools of the conflict analyses, and the condition on that rule under which the selected analysis keeps its causal reading", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T20" }, { "declarations": [ { "claim_revision_id": "catalog:205b93b14bc24d7e022b002b23bd5fbbc5b1fef420705df1035ce1e72eb4d0d9", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/Wave1/Gap/GP037B1.lean:336 gives the exact exposure probabilities, positivity and cutoff comparisons, and the two supported seed selections with their direct-versus-school labels from T21(W1),(W2). The definitions at P037/P037_Sample.lean:239 and :245 use the target’s disjoint exposure conditions and the fair-school, uniform-six-of-twelve marginal. Adding the three independent peer-free blocks yields the stated 48-eligible school without changing probabilities or labels. The simulation-sequence and full analysis-mask membership assertions are separate components expressly outside this row’s credit, so their absence is not an omitted conclusion of the credited certificate.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.positivity_cutoff_counterexamples_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B1.lean#L332", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:205b93b14bc24d7e022b002b23bd5fbbc5b1fef420705df1035ce1e72eb4d0d9", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P037/Wave1/Gap/GP037B1.lean:160 gives a 10000-draw sequence of valid six-of-twelve selections and the exact frequencies (1/2,1/2000,499/2000,1/4), matching T21(W1)(iv). Each local school Boolean and seed selection has a supported completion to the two-of-four school randomization and three additional peer-free seed blocks. Thus the local representation preserves the target’s existential full-design witness; positivity and failure of the 0.01 cutoff follow immediately from the displayed fractions.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.cutoff_simulated_sequence_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B1.lean#L158", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:205b93b14bc24d7e022b002b23bd5fbbc5b1fef420705df1035ce1e72eb4d0d9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T21", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T21" }, "html": { "body": "

main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40. Two seed designs separate positivity from the 0.01 screen and give distinct seed/nonseed labels, with a supported simulated-frequency witness.

", "headline": "Main Methods, ‘we restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure’, with SI pp. 14, 18–19 and the note to Table S5; the released code keeps the students whose four estimated probabilities exceed 0.01 and who were not seeds in the realized assignment. Two exact designs in which those rules and positivity select different students (the validator's certificates; not in the paper)", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40" }, "id": "P037-T21", "text": { "body": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40. Two seed designs separate positivity from the 0.01 screen and give distinct seed/nonseed labels, with a supported simulated-frequency witness.", "headline": "Main Methods, ‘we restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure’, with SI pp. 14, 18–19 and the note to Table S5; the released code keeps the students whose four estimated probabilities exceed 0.01 and who were not seeds in the realized assignment. Two exact designs in which those rules and positivity select different students (the validator's certificates; not in the paper)", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T21" }, { "declarations": [ { "claim_revision_id": "source-claim-import-2026-11-08:P037-T22:1", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"GP037B1.lean:422 (conclusions at lines 428–433) establishes the population equality, screen inclusions and realized-nonseed inclusion, but does not assert T22(b)'s coordinatewise statement: 'If π_ik = 0 then π̂_ik = 0 for every sequence of assignments from Ω.' An all-four-positive screen inclusion does not supply that assertion for a student whose other exposure coordinates fail the screen; these are precisely relevant cases outside the positivity population. The row expressly credits all of T22(b), so its exported conclusion is materially narrower, even though its stated inclusions are correct. The target's assertion that the four indirect exposure probabilities sum to one is an inaccurate rationale when direct exposure is possible, but disjointness and positivity of the other labels suffice for T22(a), so this is not another material defect in the Lean statement.\", 'suggested_fix': 'Inside the existing π/phat let bindings, add the conjunct ∀ (i : I) (k : Fin 4), π i k = 0 → phat i k = 0; the existing universal draws parameter then covers every supported sequence. This restores the missing T22(b) conclusion without requiring the uncredited parts of T22.', 'second_review': \"P037/Wave1/Gap/GP037B1.lean:422 exports the set inclusions at line 428 but omits T22(b)'s coordinatewise assertion 'If π_ik = 0 then π̂_ik = 0 for every sequence of assignments from Ω'. The all-four-positive inclusions do not themselves assert this for students outside the screen. However, SupportedDesign at P037/P037_Sample.lean:23 assigns every state positive mass, and sampleProb at P037/P037_Sample.lean:30 is a finite nonnegative sum, so π_ik = 0 forces every matching indicator to vanish and hence every empirical count to be zero. Add the missing coordinatewise conjunct; this elementary support argument makes the statement omission minor.\", 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P037.Wave1.analysis_set_inclusions_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B1.lean#L416", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T22:1", "conditional_on": [], "credit_reason": "The strict cutoff is represented correctly: count 100 fails, count 101 passes. No zero-denominator convention supplies the reciprocal conclusion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.cutoff_integer_rule", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L266", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T22:1", "conditional_on": [], "credit_reason": "This row is correctly scoped to one exposure event. It establishes neither the binomial formula nor the randomness of the whole outcome-specific analysis set.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.estimated_cutoff_random_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B2.lean#L72", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "source-claim-import-2026-11-08:P037-T22:1", "conditional_on": [], "credit_reason": "The mask explicitly excludes seeds, so probability monotonicity gives the correct lower bound. Positive seed probability alone would not prove that the entire analysis set has nonconstant membership.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.code_analysis_seed_exclusion", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L360", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "source-claim-import-2026-11-08:P037-T22:1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T22", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T22" }, "html": { "body": "

main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19; Table S5 note, SI p. 40. Corrected claim. The positive-probability population equals the all-four-positive population; the exact and simulated .01 screens have all stated inclusions, with strictness in a legal design. Five-label probabilities sum to one, and the four indirect probabilities sum to at most one. Zero exact probability forces zero supported simulation frequency. Under 10000 independent design draws, exposure counts have the binomial law, the scalar cutoff is count at least 101, and both scalar and joint four-cutoff membership have probability strictly between zero and one for students of the positive-probability population. A balanced supported sequence passes, a constant sequence fails, and retained inverse frequencies are at most 10000/101 < 100. For each fixed supported simulation, the outcome-specific analysis set excludes realized seeds, with exclusion probability at least the direct probability. Positive seed probability alone permits a constant nonempty analysis set; positive seed and positive inclusion probabilities together imply inclusion probability at most one minus the direct probability and strictly between zero and one, so the set is not almost surely constant. Predictions are averaged over this same outcome-specific set with its own cardinality whenever it is nonempty; the printed comparison averages over the fixed all-four-positive population. The usual measure-theoretic conditions are assumed; here all laws are finite. No historical or public-rerun sample size is asserted.

", "headline": "Main Methods, SI pp. 14, 18–19 and the note to Table S5, the population of the social-influence estimates, corrected: the analysis set of the released code, and how it stands to the positive-probability population", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40" }, "id": "P037-T22", "text": { "body": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19; Table S5 note, SI p. 40. Corrected claim. The positive-probability population equals the all-four-positive population; the exact and simulated .01 screens have all stated inclusions, with strictness in a legal design. Five-label probabilities sum to one, and the four indirect probabilities sum to at most one. Zero exact probability forces zero supported simulation frequency. Under 10000 independent design draws, exposure counts have the binomial law, the scalar cutoff is count at least 101, and both scalar and joint four-cutoff membership have probability strictly between zero and one for students of the positive-probability population. A balanced supported sequence passes, a constant sequence fails, and retained inverse frequencies are at most 10000/101 < 100. For each fixed supported simulation, the outcome-specific analysis set excludes realized seeds, with exclusion probability at least the direct probability. Positive seed probability alone permits a constant nonempty analysis set; positive seed and positive inclusion probabilities together imply inclusion probability at most one minus the direct probability and strictly between zero and one, so the set is not almost surely constant. Predictions are averaged over this same outcome-specific set with its own cardinality whenever it is nonempty; the printed comparison averages over the fixed all-four-positive population. The usual measure-theoretic conditions are assumed; here all laws are finite. No historical or public-rerun sample size is asserted.", "headline": "Main Methods, SI pp. 14, 18–19 and the note to Table S5, the population of the social-influence estimates, corrected: the analysis set of the released code, and how it stands to the positive-probability population", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22" }, { "declarations": [ { "claim_revision_id": "catalog:50e30beba597060062d81cee3023b6b5b22df49fcd910a9fa787b2825658b89e", "conditional_on": [], "credit_reason": "Auxiliary arithmetic is assessed as described; it is not accepted by itself as a design counterexample or a variance theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.rare_exposure_frequency_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L324", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:50e30beba597060062d81cee3023b6b5b22df49fcd910a9fa787b2825658b89e", "conditional_on": [], "credit_reason": "Not applied separately: the retained declaration already exports the same supporting bound.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.rare_zero_hit_probability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L338", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:50e30beba597060062d81cee3023b6b5b22df49fcd910a9fa787b2825658b89e", "conditional_on": [], "credit_reason": "The latent-state representation is a pushforward representation of the same design, not a new probability model. The target expressly constructs a fixed synthetic witness, so it does not require existence of such a student in the study.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.rare_probability_design_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B2.lean#L362", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:50e30beba597060062d81cee3023b6b5b22df49fcd910a9fa787b2825658b89e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T29", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T29" }, "html": { "body": "

SI Appendix p. 14; the general simulation statement and J are on SI p. 18. In the stated synthetic four-block, forty-eligible school, exactly one of 63504 uniform first-two-block selections avoids all ten peers; 31752 contain referent 0, and 31751 hit only nonreferent peers. The exposure vector is (1/2, 1/127008, 31751/127008, 1/4), sums to one, and is strictly positive and below one. Every supported 10000-draw frequency misses the school-exposure probability by at least that probability, with the stated sharper zero/positive-count alternatives, regardless of dependence. For independent design draws, the count has the full binomial law, expected count 625/7938, and zero-hit probability strictly between .9 and .95. The school-exposure frequency (count divided by 10000) has variance 127007/161310320640000, relative variance 127007/10000, and relative standard error above 3.5. The control frequency has variance 1/40000 and standard error 1/200. The usual measure-theoretic conditions are assumed; all quantities here are finite sums under explicit finite product laws. This is a synthetic witness and asserts no occurrence in the study.

", "headline": "SI Appendix, ‘We achieve an arbitrarily precise estimate of πi\\pi_iπi​ via J=10,000J = 10,000J=10,000 simulated random assignments of Z\\mathbf{Z}Z according to the experimental design’: a student, in a design of the paper's form, whose probability of school exposure cannot be estimated from 10,000 draws to within half its own size (the validator's witness; not in the paper)", "location": "SI Appendix p. 14 (supplement PDF p. 14); the general statement and the choice of JJJ are on SI p. 18" }, "id": "P037-T29", "text": { "body": "SI Appendix p. 14; the general simulation statement and J are on SI p. 18. In the stated synthetic four-block, forty-eligible school, exactly one of 63504 uniform first-two-block selections avoids all ten peers; 31752 contain referent 0, and 31751 hit only nonreferent peers. The exposure vector is (1/2, 1/127008, 31751/127008, 1/4), sums to one, and is strictly positive and below one. Every supported 10000-draw frequency misses the school-exposure probability by at least that probability, with the stated sharper zero/positive-count alternatives, regardless of dependence. For independent design draws, the count has the full binomial law, expected count 625/7938, and zero-hit probability strictly between .9 and .95. The school-exposure frequency (count divided by 10000) has variance 127007/161310320640000, relative variance 127007/10000, and relative standard error above 3.5. The control frequency has variance 1/40000 and standard error 1/200. The usual measure-theoretic conditions are assumed; all quantities here are finite sums under explicit finite product laws. This is a synthetic witness and asserts no occurrence in the study.", "headline": "SI Appendix, ‘We achieve an arbitrarily precise estimate of $\\pi_i$ via $J = 10,000$ simulated random assignments of $\\mathbf{Z}$ according to the experimental design’: a student, in a design of the paper's form, whose probability of school exposure cannot be estimated from 10,000 draws to within half its own size (the validator's witness; not in the paper)", "location": "SI Appendix p. 14 (supplement PDF p. 14); the general statement and the choice of $J$ are on SI p. 18" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T29" }, { "declarations": [ { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "Not applied separately: the proved blocked-exposure formula already incorporates the avoidance count.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.count_avoiding_seeds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L208", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "The zero example is substantive: six peers in a ten-eligible block leave only four nonpeers for five required seeds.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.blocked_school_probability_examples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L292", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "The extended count domain is harmless and needs no stochastic independence.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.frequency_resolution", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Sample.lean#L307", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "J>0 and ε,δ>0 keep all accuracy-bound denominators nonzero; binomial natural subtraction is used only with r≤J. Strict support positivity makes the p=0 conclusion everywhere on Ω valid.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.exposure_monte_carlo_law_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GLP037B2.lean#L188", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "t_h≤n_h guarantees positive binomial denominators; actual peer subsets guarantee no erroneous natural truncation. Empty-block and zero-seed boundary cases retain the stated probability meaning.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.blocked_school_exposure_formula_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B3.lean#L38", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "conditional_on": [], "credit_reason": "Shared draws need not make the m exposure indicators independent; only the draws themselves are independent. The existential form is the correct union event.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.monte_carlo_simultaneous_bound_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B3.lean#L124", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:fd014b0fd42b495b630b7c7a4c8026a741776a8ba44efac3505297d7fac99d78", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T30", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T30" }, "html": { "body": "

SI Appendix p. 14 and p. 18 (supplement PDF pp. 14 and 18). Independent finite-design simulations have the stated binomial law, moments, precision bounds and blocked school-exposure formula.

", "headline": "SI Appendix, the precision of the simulated exposure probabilities, corrected: the exact law of the estimate for a fixed number of draws, and the sense in which increasing the number of draws gives arbitrary accuracy", "location": "SI Appendix p. 14 and p. 18 (supplement PDF pp. 14 and 18)" }, "id": "P037-T30", "text": { "body": "SI Appendix p. 14 and p. 18 (supplement PDF pp. 14 and 18). Independent finite-design simulations have the stated binomial law, moments, precision bounds and blocked school-exposure formula.", "headline": "SI Appendix, the precision of the simulated exposure probabilities, corrected: the exact law of the estimate for a fixed number of draws, and the sense in which increasing the number of draws gives arbitrary accuracy", "location": "SI Appendix p. 14 and p. 18 (supplement PDF pp. 14 and 18)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T30" }, { "declarations": [ { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "conditional_on": [], "credit_reason": "Arithmetic for additional support ratios, pair coefficients or 56-school variances is outside the retained actual-law certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_support_count_arithmetic", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Design.lean#L39", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "conditional_on": [], "credit_reason": "Arithmetic for additional support ratios, pair coefficients or 56-school variances is outside the retained actual-law certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_pair_probability_arithmetic", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Design.lean#L53", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "conditional_on": [], "credit_reason": "The broadened cases need not be differences of actual treated and control group means, but the n=2m balanced specialization used by T31 is exactly that statistic.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.school_statistic_variance_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B3.lean#L488", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "conditional_on": [], "credit_reason": "Arithmetic for additional support ratios, pair coefficients or 56-school variances is outside the retained actual-law certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_variance_moves_both_ways", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Design.lean#L64", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "conditional_on": [], "credit_reason": "Covers the actual finite sampler witness and the two eight-school variance comparisons, not every clause of the composite T31. The uniform-law interpretation and the use of synthetic sharp-null outcomes are explicit in the target, not additional assumptions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.school_sampler_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B3.lean#L396", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1f336a11d2489638c222e315aec141e8cd4ce9f4c73a8599f14dd6eecb71c0fb", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T31", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T31" }, "html": { "body": "

SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5 and in Fig. S1, SI p. 24. The school samplers have different support counts and pair probabilities, with exact eight-school second-moment changes in both directions.

", "headline": "SI Appendix, ‘simulated random assignments of Z\\mathbf{Z}Z according to the experimental design’ (pp. 14 and 18) and ‘1,000 simulated randomizations (both randomization across schools and within seed-eligibles)’ (pp. 19–20); in the released code the simulated school assignments are 28 of the 56 schools drawn without the blocks. Exact differences between that law and the design of two schools in each block of four (the validators' computations; not in the paper)", "location": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5 and in Fig. S1, SI p. 24" }, "id": "P037-T31", "text": { "body": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5 and in Fig. S1, SI p. 24. The school samplers have different support counts and pair probabilities, with exact eight-school second-moment changes in both directions.", "headline": "SI Appendix, ‘simulated random assignments of $\\mathbf{Z}$ according to the experimental design’ (pp. 14 and 18) and ‘1,000 simulated randomizations (both randomization across schools and within seed-eligibles)’ (pp. 19–20); in the released code the simulated school assignments are 28 of the 56 schools drawn without the blocks. Exact differences between that law and the design of two schools in each block of four (the validators' computations; not in the paper)", "location": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5 and in Fig. S1, SI p. 24" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T31" }, { "declarations": [ { "claim_revision_id": "catalog:c1e540bd2733ba047bef05caf3468d4a32ec5a4c3fbc579fcbb913adf78e7ef5", "conditional_on": [], "credit_reason": "The row explicitly covers only T32(a),(b), not the software repair or the joint-law/variance comparison of (d). The n≥2 domain is natural for the displayed distinct-pair denominator and includes all study cases.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.school_randomization_laws_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B4.lean#L64", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c1e540bd2733ba047bef05caf3468d4a32ec5a4c3fbc579fcbb913adf78e7ef5", "conditional_on": [], "credit_reason": "For normalized nonnegative p,q the product weights are the required independent law. Applying the identity twice yields invariance for any two school laws with marginal 1/2; it does not equate their joint laws across schools.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.school_local_exposure_invariance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Design.lean#L78", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c1e540bd2733ba047bef05caf3468d4a32ec5a4c3fbc579fcbb913adf78e7ef5", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T32", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T32" }, "html": { "body": "

SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5. The blocked school mass normalizes, uniform selection gives the stated marginals and pairs, and local exposure probabilities have a half-treatment mixture form.

", "headline": "SI Appendix, the simulated randomizations ‘according to the experimental design’, corrected: the law of the school assignments that the simulations must draw from, what is unchanged when the blocks are omitted (each student's exposure probabilities) and what is not (the joint law across schools)", "location": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5" }, "id": "P037-T32", "text": { "body": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5. The blocked school mass normalizes, uniform selection gives the stated marginals and pairs, and local exposure probabilities have a half-treatment mixture form.", "headline": "SI Appendix, the simulated randomizations ‘according to the experimental design’, corrected: the law of the school assignments that the simulations must draw from, what is unchanged when the blocks are omitted (each student's exposure probabilities) and what is not (the joint law across schools)", "location": "SI Appendix pp. 14, 18 and 19–20 (supplement PDF pp. 14, 18, 19, 20); the design is on SI p. 5" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T32" }, { "declarations": [ { "claim_revision_id": "catalog:c010382120b10976b96bc4eb5c43818ba5923b83479bc68a44da276c8cc7b3f2", "conditional_on": [], "credit_reason": "The target expressly uses the general finite-design witness, rather than requiring the study’s four seed blocks. It uses the corrected verbal exposure categories and the original directed referents; the finite-count probability definition encodes precisely the stated design.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.nomination_union_counterexamples_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B2.lean#L139", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c010382120b10976b96bc4eb5c43818ba5923b83479bc68a44da276c8cc7b3f2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T25", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T25" }, "html": { "body": "

SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3). The two explicitly specified nomination networks give different exposures and positive exposure probabilities under the nomination and either-direction rows.

", "headline": "SI Appendix, ‘We define the exposure mapping as a function, f(z,θi)f(\\mathbf{z},\\theta_i)f(z,θi​), such that the parameter, θi\\theta_iθi​, equals student iii's row in the network adjacency matrix’, read with main p. 567 (‘directly measures who is observing whose behavior’); the released primary analysis uses the row of the either-direction union of the nominations. Two exact networks in which the nomination row and the union row give different exposures and different exposure probabilities (the checker's certificates; not in the paper)", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "id": "P037-T25", "text": { "body": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3). The two explicitly specified nomination networks give different exposures and positive exposure probabilities under the nomination and either-direction rows.", "headline": "SI Appendix, ‘We define the exposure mapping as a function, $f(\\mathbf{z},\\theta_i)$, such that the parameter, $\\theta_i$, equals student $i$'s row in the network adjacency matrix’, read with main p. 567 (‘directly measures who is observing whose behavior’); the released primary analysis uses the row of the either-direction union of the nominations. Two exact networks in which the nomination row and the union row give different exposures and different exposure probabilities (the checker's certificates; not in the paper)", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T25" }, { "declarations": [ { "claim_revision_id": "catalog:67e21ac5768a1b0633e5529bdcfdf801adf59ed7d78d928b0ceaacfbdcc43f46", "conditional_on": [], "credit_reason": "The statement exactly covers the assigned component, with no extra substantive hypothesis.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.either_direction_properties", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L59", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:67e21ac5768a1b0633e5529bdcfdf801adf59ed7d78d928b0ceaacfbdcc43f46", "conditional_on": [], "credit_reason": "The removed zero-diagonal premise is a harmless generalization, retained as broader_than_claim under the requested categories.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.either_direction_count_identities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L74", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:67e21ac5768a1b0633e5529bdcfdf801adf59ed7d78d928b0ceaacfbdcc43f46", "conditional_on": [], "credit_reason": "The scalar-count formulation loses no graph instance; the reciprocal-eligible-tie sufficient condition follows immediately by setting b=q=0.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.exposure_rows_equal_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L91", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:67e21ac5768a1b0633e5529bdcfdf801adf59ed7d78d928b0ceaacfbdcc43f46", "conditional_on": [], "credit_reason": "Strict mass positivity is exactly appropriate on the support Ω of S5; zero-mass assignments can be removed. The count labels are treated-nonseed S,NI,I=0,1,2, rather than the global C,S,NI,I coding.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.exposure_probability_monotone", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L101", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:67e21ac5768a1b0633e5529bdcfdf801adf59ed7d78d928b0ceaacfbdcc43f46", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T26", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T26" }, "html": { "body": "

SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3). Either-direction peers decompose into outgoing and incoming-only nominations and obey the stated label and probability comparisons.

", "headline": "SI Appendix, the definition of θi\\theta_iθi​ in the exposure mapping, corrected: θi\\theta_iθi​ is student iii's row of the either-direction matrix of nominations; the two counts for that matrix, and when it gives the same exposure as the row of nominations", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "id": "P037-T26", "text": { "body": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3). Either-direction peers decompose into outgoing and incoming-only nominations and obey the stated label and probability comparisons.", "headline": "SI Appendix, the definition of $\\theta_i$ in the exposure mapping, corrected: $\\theta_i$ is student $i$'s row of the either-direction matrix of nominations; the two counts for that matrix, and when it gives the same exposure as the row of nominations", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T26" }, { "declarations": [ { "claim_revision_id": "catalog:c581169941f3d199fa49bb85a873a214720285fe936c63ef12f555ec0120cda6", "conditional_on": [], "credit_reason": "The own-assignment premise is the target's definition of no social influence. The directOnly definition specifies the substantive witness schedule rather than assuming the printed assertion false.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.paired_no_social_influence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L195", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c581169941f3d199fa49bb85a873a214720285fe936c63ef12f555ec0120cda6", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P037/Wave1/Gap/GLP037B4.lean:506, using the definitions at P037/P037_Network.lean:207, implements the reciprocal-pair design, own-assignment outcomes, fixed-outcome histogram, exact p-value, and rejection frequency in T35(ii)-(v). Pair/Boolean indices are bijective with students 0 through 9, and the exact rather than simulated p-value is expressly requested by the target. The four-block clause at P037/Wave1/Gap/GLP037B4.lean:518 uses precisely the support choosing five students per block, whose cardinality is 252^4; the displayed outcomes and statistic give -1 at the observed all-false-side selection, making its extreme-tail count the p-value required by T35(vi). The own-assignment definition already excludes peer effects, and part (i) is separately stated at P037/P037_Network.lean:219; this row's assigned component is (ii)-(vi), so neither that separation nor the implicit elementary four-block evaluations omit a material conclusion.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.direct_effect_permutation_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GLP037B4.lean#L504", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c581169941f3d199fa49bb85a873a214720285fe936c63ef12f555ec0120cda6", "conditional_on": [], "credit_reason": "Assessed as supporting arithmetic, not as a substitute for R228-08's actual-support enumeration statement. The reference to a drafted theorem identifies the original held declaration in the source, whose matching proof is separately credited as R228-08.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.paired_permutation_arithmetic", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L204", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c581169941f3d199fa49bb85a873a214720285fe936c63ef12f555ec0120cda6", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T35", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T35" }, "html": { "body": "

SI Appendix pp. 29 and 32, Fig. S5 (pp. 30–32). Own-seed outcomes alone give the stated one-block distribution, p-values and rejection probability 8/63. With four disjoint copies the uniform support has size 252^4, the copied observation has statistic -1, exactly two assignments have absolute statistic 1, and its exact two-sided p-value is 2/252^4. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).

", "headline": "SI Appendix, the alternative specification of Fig. S5, ‘our comparisons with this methodology are restricted to social influence effects’: a schedule of potential outcomes with a direct effect of being a seed and no effect of any other student's assignment, for which the test rejects (the validator's witness; not in the paper)", "location": "SI Appendix p. 29 and p. 32 (supplement PDF pp. 29 and 32); Fig. S5, SI pp. 30–32" }, "id": "P037-T35", "text": { "body": "SI Appendix pp. 29 and 32, Fig. S5 (pp. 30–32). Own-seed outcomes alone give the stated one-block distribution, p-values and rejection probability 8/63. With four disjoint copies the uniform support has size 252^4, the copied observation has statistic -1, exactly two assignments have absolute statistic 1, and its exact two-sided p-value is 2/252^4. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).", "headline": "SI Appendix, the alternative specification of Fig. S5, ‘our comparisons with this methodology are restricted to social influence effects’: a schedule of potential outcomes with a direct effect of being a seed and no effect of any other student's assignment, for which the test rejects (the validator's witness; not in the paper)", "location": "SI Appendix p. 29 and p. 32 (supplement PDF pp. 29 and 32); Fig. S5, SI pp. 30–32" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T35" }, { "declarations": [ { "claim_revision_id": "catalog:7db15b1401471f0ab3b3201d262ce5c83d89ae9038a5c83ffe45f28f1628f788", "conditional_on": [], "credit_reason": "The binary, support and positive-denominator assumptions are exactly appropriate to the claimed statistic. The row establishes the direct-effect identity, while R228-08 supplies the nonzero-coefficient and over-rejection witness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.nomination_statistic_direct_effect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Network.lean#L167", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7db15b1401471f0ab3b3201d262ce5c83d89ae9038a5c83ffe45f28f1628f788", "conditional_on": [], "credit_reason": "The exact design-weighted tail is the component expressly targeted by T36(b); simulated 1,000-draw p-value accuracy is a different question. Freezing the outcome vector causes no mismatch under the supplied sharp null.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.Wave1.sharp_null_permutation_valid_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GP037B4.lean#L168", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:7db15b1401471f0ab3b3201d262ce5c83d89ae9038a5c83ffe45f28f1628f788", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T36", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T36" }, "html": { "body": "

SI Appendix pp. 29 and 32, Fig. S5 (pp. 30–32). The nomination statistic gives a valid exact test of the global sharp null, admits the stated direct-effect decomposition, and can reject the reciprocal-pair schedule without social influence. Both recipient denominators are positive whenever the general statistic is used as a mean.

", "headline": "SI Appendix, the alternative specification of Fig. S5, corrected: the statistic that the released code computes, the null hypothesis its permutation test is valid for, and the term by which a direct effect of being a seed enters the statistic", "location": "SI Appendix p. 29 and p. 32 (supplement PDF pp. 29 and 32); Fig. S5, SI pp. 30–32" }, "id": "P037-T36", "text": { "body": "SI Appendix pp. 29 and 32, Fig. S5 (pp. 30–32). The nomination statistic gives a valid exact test of the global sharp null, admits the stated direct-effect decomposition, and can reject the reciprocal-pair schedule without social influence. Both recipient denominators are positive whenever the general statistic is used as a mean.", "headline": "SI Appendix, the alternative specification of Fig. S5, corrected: the statistic that the released code computes, the null hypothesis its permutation test is valid for, and the term by which a direct effect of being a seed enters the statistic", "location": "SI Appendix p. 29 and p. 32 (supplement PDF pp. 29 and 32); Fig. S5, SI pp. 30–32" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T36" }, { "declarations": [ { "claim_revision_id": "catalog:e281f67bda96906ee7a3f157704c72d6cb3cc8c6451c28490be809d31ba28beb", "conditional_on": [], "credit_reason": "The credited fit comparison is correct. The independent E1 exposure law is not formalized by the isolated scalar (1/4)^12>0; the separate E2 design is certified in R226-18. The record contains superseded admission/status history followed by completion; no proof audit was performed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.separate_pooled_regression_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L587", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e281f67bda96906ee7a3f157704c72d6cb3cc8c6451c28490be809d31ba28beb", "conditional_on": [], "credit_reason": "The purpose is the normal-equation certificate, not the entire argmin or assignment-design claim; R226-20 supplies the pooled minimizer biconditional.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.regression12_normal_certificates", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L485", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e281f67bda96906ee7a3f157704c72d6cb3cc8c6451c28490be809d31ba28beb", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P037/P037_Regression.lean:1151 states the graph, sole-referent condition, one seed from each eligible pair, focal probabilities, exact four-exposure population, and realization mass specified by T27(E2)(iv). The law defined at P037/P037_Regression.lean:902 assigns equal mass to the 20 treated-school sets and all 4^6 seed-bit vectors; summing unused draws in control schools gives precisely the claimed assignment law. The claim explicitly constructs six schools with four eligibles apiece as a validator's synthetic certificate, so its difference from the actual study protocol in S1-S2 is not a Lean-added restriction or a misstatement of the certificate. No case or conclusion of this component is lost.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.regression12_hierarchical_design", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L1134", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:e281f67bda96906ee7a3f157704c72d6cb3cc8c6451c28490be809d31ba28beb", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T27", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T27" }, "html": { "body": "

SI Appendix pp. 18–19; main p. 568 (PDF p. 3). The specified twelve-unit potential outcomes give different separate and pooled WLS means and contrasts under both weight vectors. E1 consists of independent uniform four-category exposures, with every joint cylinder probability (1/4)^its size, marginals 1/4, weights 4 and observed realization probability (1/4)^12 > 0. E2 is the six-school seed design with focal probabilities (1/2,1/8,1/8,1/4), realization probability 1/640 and weights (2,8,8,4). All stated coefficients, residuals, means and contrast differences hold. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).

", "headline": "SI Appendix, the estimator ξ^(dk)\\hat\\xi(d_k)ξ^​(dk​), μR^(dk)\\widehat{\\mu_R}(d_k)μR​​(dk​) of the average potential outcomes (a separate inverse-probability-weighted regression ‘among students in a given level of exposure’); the released code fits one weighted regression with exposure intercepts and common covariate slopes. An exact data set, with two assignment designs, in which the two procedures give different means and contrasts (the validator's certificate; not in the paper)", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "id": "P037-T27", "text": { "body": "SI Appendix pp. 18–19; main p. 568 (PDF p. 3). The specified twelve-unit potential outcomes give different separate and pooled WLS means and contrasts under both weight vectors. E1 consists of independent uniform four-category exposures, with every joint cylinder probability (1/4)^its size, marginals 1/4, weights 4 and observed realization probability (1/4)^12 > 0. E2 is the six-school seed design with focal probabilities (1/2,1/8,1/8,1/4), realization probability 1/640 and weights (2,8,8,4). All stated coefficients, residuals, means and contrast differences hold. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).", "headline": "SI Appendix, the estimator $\\hat\\xi(d_k)$, $\\widehat{\\mu_R}(d_k)$ of the average potential outcomes (a separate inverse-probability-weighted regression ‘among students in a given level of exposure’); the released code fits one weighted regression with exposure intercepts and common covariate slopes. An exact data set, with two assignment designs, in which the two procedures give different means and contrasts (the validator's certificate; not in the paper)", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T27" }, { "declarations": [ { "claim_revision_id": "catalog:6595de1b644f1dd045255652fed51ac32fb37e054829866bc73ff9530c4a6a2a", "conditional_on": [], "credit_reason": "Actual minimizers are premises because this row proves their normal-equation consequences; it does not assume those consequences in regressionMinimum. Full column rank is present in T28 itself.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.pooled_and_separate_mean_identities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L122", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6595de1b644f1dd045255652fed51ac32fb37e054829866bc73ff9530c4a6a2a", "conditional_on": [], "credit_reason": "Positive Sxx for all four labels is explicitly required by T28(d), so this is not an extra nondegeneracy assumption. It also implies the nonempty-analysis-set hypothesis and nonzero W_k and total Sxx.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.one_covariate_regression_formulas", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L247", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6595de1b644f1dd045255652fed51ac32fb37e054829866bc73ff9530c4a6a2a", "conditional_on": [], "credit_reason": "Auxiliary algebra, rather than a standalone theorem identifying fitted regression outputs.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.separate_pooled_prediction_difference", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L468", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6595de1b644f1dd045255652fed51ac32fb37e054829866bc73ff9530c4a6a2a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T28", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T28" }, "html": { "body": "

SI Appendix pp. 18–19; main p. 568 (PDF p. 3). On the same finite analysis set, joint and separate WLS give the stated zero residual sums, standardized means, one-covariate coefficient formulas, and exact equality characterization.

", "headline": "SI Appendix, the estimator of the average potential outcomes, corrected: the estimator the released code computes (one weighted regression with exposure intercepts and common covariate slopes, averaged over the analysis set), and its relation to the displayed one", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "id": "P037-T28", "text": { "body": "SI Appendix pp. 18–19; main p. 568 (PDF p. 3). On the same finite analysis set, joint and separate WLS give the stated zero residual sums, standardized means, one-covariate coefficient formulas, and exact equality characterization.", "headline": "SI Appendix, the estimator of the average potential outcomes, corrected: the estimator the released code computes (one weighted regression with exposure intercepts and common covariate slopes, averaged over the analysis set), and its relation to the displayed one", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T28" }, { "declarations": [ { "claim_revision_id": "catalog:8be54881d981ea53065fb750e2b1ef99c0d4357d05175ed1e7baf58122cd33e4", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"GLP037B3.lean:386 exports the imputation identity at lines 394–396 only for controlContrast, the I-minus-NI contrast, whereas T33(iii) asserts that the full-control refitted means 'differ by exactly m̂_k for every exposure k and every a.' The individual mean identities, including those for C and S, are missing from the credited certificate; the exact contrast variance discrepancy is nevertheless established. The residual-vector subflag overstates the omission: the coefficients exported at GLP037B3.lean:392, substituted into the fixed definition controlResidual at P037_Regression.lean:779 and the specified outcomes and regressors, give all eight printed residual values by elementary arithmetic. The target itself specifies an abstract 96-assignment exposure design, so its control probability of 1/4 is not a fidelity problem.\", 'suggested_fix': 'Add an all-assignment, all-exposure mean identity: for L_a = regressionMap (controlWeight (controlExposure a)) (controlFullRow (controlExposure a)) and M_a(Y,k) = (L_a.mulVec Y) k.castSucc + (L_a.mulVec Y) 4 / 4, assert ∀ a k, M_a(y*(a),k) = M_a(controlResidual,k) + controlPointMean k, with y*(a) as in the existing contrast conjunct. Retain the existing exact variance conclusions; an explicit residual-vector equality is optional because it follows by substitution from the exported coefficients.', 'second_review': \"P037/Wave1/Gap/GLP037B3.lean:394 exports only the I-minus-NI imputation shift, while T33(iii) requires the means to differ by m̂_k 'for every exposure k and every a'. The declaration already exports full column rank at P037/Wave1/Gap/GLP037B3.lean:391 and the exact variance equalities at line 397. With the positive weights defined at P037/P037_Regression.lean:774 and the inverse-form regressionMap at P037/P037_Regression.lean:30, full rank gives L_a X_a = I. The definitions at P037/P037_Regression.lean:778 and P037/P037_Regression.lean:779 give y*(a) = r + X_a b_0, hence L_a y*(a) = L_a r + b_0 and every mean shifts by m̂_k through elementary algebra; appending those individual identities is a minor statement repair.\", 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P037.Wave1.collapsed_controls_variance_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/Wave1/Gap/GLP037B3.lean#L386", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:8be54881d981ea53065fb750e2b1ef99c0d4357d05175ed1e7baf58122cd33e4", "conditional_on": [], "credit_reason": "The statement is a correctly scoped numerical auxiliary fact, not a standalone variance/refit certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.collapsed_controls_certificate_inequalities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L802", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:8be54881d981ea53065fb750e2b1ef99c0d4357d05175ed1e7baf58122cd33e4", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T33", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T33" }, "html": { "body": "

SI Appendix pp. 19–20, also pp. 14–15. The specified normalized 96-state abstract exposure design has the stated marginals, ranks, point coefficients, standardized means and eight residuals. At every assignment and every exposure the full-control imputed refit mean is the residual refit mean plus the fixed point mean. The two full-control contrast variances equal Vfull; the collapsed residual contrast variance equals Vcol, with 2 < Vcol < 5/2 < 9 < Vfull < 37/4. Their standard deviations round to 1.53 and 3.03. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).

", "headline": "SI Appendix, the randomization standard errors ‘under the maintained hypothesis that, for all iii, yi(dk)=yi(Di)+μ^R(dk)−μ^R(Di)y_i(d_k) = y_i(D_i) + \\hat\\mu_R(d_k) - \\hat\\mu_R(D_i)yi​(dk​)=yi​(Di​)+μ^​R​(dk​)−μ^​R​(Di​)’; in the released code the regressions refitted on the simulated assignments replace two exposure-probability controls of the point regression by their sum. An exact design in which the variance so computed differs from the variance of the point estimator under the stated imputation (the validator's certificate; not in the paper)", "location": "SI Appendix pp. 19–20 (supplement PDF pp. 19–20); the same description is on SI pp. 14–15" }, "id": "P037-T33", "text": { "body": "SI Appendix pp. 19–20, also pp. 14–15. The specified normalized 96-state abstract exposure design has the stated marginals, ranks, point coefficients, standardized means and eight residuals. At every assignment and every exposure the full-control imputed refit mean is the residual refit mean plus the fixed point mean. The two full-control contrast variances equal Vfull; the collapsed residual contrast variance equals Vcol, with 2 < Vcol < 5/2 < 9 < Vfull < 37/4. Their standard deviations round to 1.53 and 3.03. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).", "headline": "SI Appendix, the randomization standard errors ‘under the maintained hypothesis that, for all $i$, $y_i(d_k) = y_i(D_i) + \\hat\\mu_R(d_k) - \\hat\\mu_R(D_i)$’; in the released code the regressions refitted on the simulated assignments replace two exposure-probability controls of the point regression by their sum. An exact design in which the variance so computed differs from the variance of the point estimator under the stated imputation (the validator's certificate; not in the paper)", "location": "SI Appendix pp. 19–20 (supplement PDF pp. 19–20); the same description is on SI pp. 14–15" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T33" }, { "declarations": [ { "claim_revision_id": "catalog:a98d9a966f3aa31ba37673e7886cc5a0cf1bc26313b944fd2c796fa931d221c6", "conditional_on": [], "credit_reason": "Auxiliary left-inverse identity; the credited same_controls_residual_imputation already exports the required WLS coefficient and mean shifts.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.linear_residual_refit_shift", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L621", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:a98d9a966f3aa31ba37673e7886cc5a0cf1bc26313b944fd2c796fa931d221c6", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P037/P037_Regression.lean:694 uses the fixed units and controls, positive weights, and full-rank pooled designs expressly required by T34(a)-(c), with the actual inverse-formula WLS map defined at P037/P037_Regression.lean:30. Its conclusions give the intercept-contrast identity, coefficient refit shift, pointwise mean shift, and contrast-variance equality. The mean shift at P037/P037_Regression.lean:711 is the fixed number m(k), independent of the assignment, so it also establishes equality of each mean's variance under the normalized finite law; a separate variance conjunct adds no mathematical content. The target's remark about aliased representations calls for a full-rank basis of estimable quantities, consistent with the theorem's full-rank formulation.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.same_controls_residual_imputation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Regression.lean#L695", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:a98d9a966f3aa31ba37673e7886cc5a0cf1bc26313b944fd2c796fa931d221c6", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T34", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T34" }, "html": { "body": "

SI Appendix pp. 19–20, also pp. 14–15. For fixed units and controls, positive weights and full column rank, the printed constant-effect imputation equals r + H_z theta, its WLS coefficients shift by theta, and every standardized mean and contrast shifts by its fixed point value and has the same variance as the residual refit. Fewer controls retain only the projection of the omitted control contribution; this equals that contribution exactly when it belongs to the reduced design space. The T33 design has a strictly smaller collapsed control space, an omitted contribution outside the reduced design, and unequal variances. Aliased presentations have the same unique fitted values after passing to a full-rank basis, and the shift and variance identities hold for estimable means and contrasts. The usual measure-theoretic conditions are assumed; finite normalized laws describe distributions on the finitely many exposure assignments.

", "headline": "SI Appendix, the randomization standard errors under constant effects, corrected: refitting the residuals of the point regression reproduces the printed imputation exactly when the refits keep the controls of the point regression, and not when two controls are replaced by their sum", "location": "SI Appendix pp. 19–20 (supplement PDF pp. 19–20); the same description is on SI pp. 14–15" }, "id": "P037-T34", "text": { "body": "SI Appendix pp. 19–20, also pp. 14–15. For fixed units and controls, positive weights and full column rank, the printed constant-effect imputation equals r + H_z theta, its WLS coefficients shift by theta, and every standardized mean and contrast shifts by its fixed point value and has the same variance as the residual refit. Fewer controls retain only the projection of the omitted control contribution; this equals that contribution exactly when it belongs to the reduced design space. The T33 design has a strictly smaller collapsed control space, an omitted contribution outside the reduced design, and unequal variances. Aliased presentations have the same unique fitted values after passing to a full-rank basis, and the shift and variance identities hold for estimable means and contrasts. The usual measure-theoretic conditions are assumed; finite normalized laws describe distributions on the finitely many exposure assignments.", "headline": "SI Appendix, the randomization standard errors under constant effects, corrected: refitting the residuals of the point regression reproduces the printed imputation exactly when the refits keep the controls of the point regression, and not when two controls are replaced by their sum", "location": "SI Appendix pp. 19–20 (supplement PDF pp. 19–20); the same description is on SI pp. 14–15" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T34" }, { "declarations": [ { "claim_revision_id": "catalog:abc293c6f0b35811294a575dc4e0b150cbfec2882aa08cb92f6c8104afd3b14f", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:abc293c6f0b35811294a575dc4e0b150cbfec2882aa08cb92f6c8104afd3b14f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T37", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T37" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Main Methods, ‘We computed confidence intervals and P values using robust SEs under a normal approximation’; the heteroskedasticity-robust Wald tests of Results (first paragraph) and of the Fig. 2 caption; SI p. 13 and the rows ‘Heteroskedasticity Robust SE’ and ‘Wild Bootstrap SE’ of Tables S6 and S7 for peer conflict, as printed", "location": "main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42)" }, "id": "P037-T37", "text": { "body": "No current theorem is supplied for this target.", "headline": "Main Methods, ‘We computed confidence intervals and P values using robust SEs under a normal approximation’; the heteroskedasticity-robust Wald tests of Results (first paragraph) and of the Fig. 2 caption; SI p. 13 and the rows ‘Heteroskedasticity Robust SE’ and ‘Wild Bootstrap SE’ of Tables S6 and S7 for peer conflict, as printed", "location": "main p. 568 (PDF p. 3); main p. 569 (PDF p. 4), Results and Fig. 2 caption; SI p. 13 (supplement PDF p. 13); SI Tables S6 and S7, pp. 41–42 (supplement PDF pp. 41–42)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T37" }, { "declarations": [ { "claim_revision_id": "catalog:b257eab13510336826a4c1021743c3cbc48c74ad8c89c53e538ea044b9281333", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:b257eab13510336826a4c1021743c3cbc48c74ad8c89c53e538ea044b9281333", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T38", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T38" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Main Methods, the sentence on the 49 schools with administrative data; the Fig. 1 caption; SI Appendix, the paragraph before the list of disciplinary codes, as printed", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10)" }, "id": "P037-T38", "text": { "body": "No current theorem is supplied for this target.", "headline": "Main Methods, the sentence on the 49 schools with administrative data; the Fig. 1 caption; SI Appendix, the paragraph before the list of disciplinary codes, as printed", "location": "main p. 567 (PDF p. 2); Fig. 1 caption, main p. 568 (PDF p. 3); SI Appendix p. 10 (supplement PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T38" }, { "declarations": [ { "claim_revision_id": "catalog:c011d879a3880b219fed408ba78718f7716543dbfe7ee482ab7ff8ac01849ae1", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:c011d879a3880b219fed408ba78718f7716543dbfe7ee482ab7ff8ac01849ae1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T39", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T39" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Main Methods, ‘we restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure’; SI Appendix pp. 14, 18 and 19, and the note to Table S5, as printed", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40 (supplement PDF p. 40)" }, "id": "P037-T39", "text": { "body": "No current theorem is supplied for this target.", "headline": "Main Methods, ‘we restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure’; SI Appendix pp. 14, 18 and 19, and the note to Table S5, as printed", "location": "main p. 568 (PDF p. 3); SI Appendix pp. 14, 18 and 19 (supplement PDF pp. 14, 18, 19); Table S5 note, SI p. 40 (supplement PDF p. 40)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T39" }, { "declarations": [ { "claim_revision_id": "catalog:e02cd37fefd9fce17c9afa5895f88538fd1549ed229a4011e565266937abd7d0", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:e02cd37fefd9fce17c9afa5895f88538fd1549ed229a4011e565266937abd7d0", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T40", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T40" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Fig. 2, caption and panel keys, and Fig. S6, caption and panel keys of the school-level panels, as printed", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "id": "P037-T40", "text": { "body": "No current theorem is supplied for this target.", "headline": "Fig. 2, caption and panel keys, and Fig. S6, caption and panel keys of the school-level panels, as printed", "location": "main p. 569 (PDF p. 4), Fig. 2; SI Appendix p. 33 (supplement PDF p. 33), Fig. S6" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T40" }, { "declarations": [ { "claim_revision_id": "catalog:844e0fb7763b88d08aecfd0b7388e736d74391862ceede44cea573917fdb66b0", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:844e0fb7763b88d08aecfd0b7388e736d74391862ceede44cea573917fdb66b0", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T41", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T41" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "SI Appendix, the definition of θi\\theta_iθi​, γ\\gammaγ and the two counts of seed peers in the exposure mapping; main Methods, the network question and the four conditions of exposure, as printed", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "id": "P037-T41", "text": { "body": "No current theorem is supplied for this target.", "headline": "SI Appendix, the definition of $\\theta_i$, $\\gamma$ and the two counts of seed peers in the exposure mapping; main Methods, the network question and the four conditions of exposure, as printed", "location": "SI Appendix p. 17 (supplement PDF p. 17); main p. 567 (PDF p. 2) and p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T41" }, { "declarations": [ { "claim_revision_id": "catalog:976b1dfe628a1204cc1ad86cc0fc54e4d4b1c481d409956d22cfb5b5177354f9", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:976b1dfe628a1204cc1ad86cc0fc54e4d4b1c481d409956d22cfb5b5177354f9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T42", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T42" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "SI Appendix, the covariate-adjusted inverse probability weighted regression estimator ξ^(dk)\\hat\\xi(d_k)ξ^​(dk​), μR^(dk)\\widehat{\\mu_R}(d_k)μR​​(dk​), and the sentence of main Methods that describes it, as printed", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "id": "P037-T42", "text": { "body": "No current theorem is supplied for this target.", "headline": "SI Appendix, the covariate-adjusted inverse probability weighted regression estimator $\\hat\\xi(d_k)$, $\\widehat{\\mu_R}(d_k)$, and the sentence of main Methods that describes it, as printed", "location": "SI Appendix pp. 18–19 (supplement PDF pp. 18–19); main p. 568 (PDF p. 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T42" }, { "declarations": [ { "claim_revision_id": "catalog:d9bf1723cf7901f838b1469e8b547fb12a6483f7e45a48748f440fa74672b5a2", "conditional_on": [], "credit_reason": "Covers the expected-normal-equation bias identity. NormalEquations defines the population projection coefficient and does not assume the bias formula. The ancillary orthogonality results and finite certificate mentioned in the package description are separate statements, not additional conclusions of this credited row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.ExpectedProjection.block_model_bias_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Review.lean#L303", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d9bf1723cf7901f838b1469e8b547fb12a6483f7e45a48748f440fa74672b5a2", "conditional_on": [], "credit_reason": "Covers the common-treatment-intercept expected-projection corollary. The finite design, complete blocks and first-moment assumptions appear in T43 itself. No sample-OLS unbiasedness or asymptotic claim is being credited.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P037.ExpectedProjection.common_intercept_identifies_slope", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P037/P037_Review.lean#L363", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:d9bf1723cf7901f838b1469e8b547fb12a6483f7e45a48748f440fa74672b5a2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P037-T43", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P037-T43" }, "html": { "body": "

accepted 19 October review of E001; SI Eq. (2), p. 13, repeated pp. 15–16; main p. 568 (PDF p. 3). On any finite normalized design and complete four-school blocks, the stated raw first moments and mean-spanning controls imply E[r]=0, E[r't]=0, E[r'V]=0 and E[r'z]=E[r'r]. If E[t't] is nonzero and K=E[r'r]>0, every actual expected-normal-equation solution obeys b-beta=E[sum r_i T_i eta_i]/K; eta=0 implies b=beta for arbitrary fixed untreated levels. A four-state binary complementary two-of-four certificate has compositions 1/4 or 3/4, nonconstant untreated levels, E[t't]=1, K=1/16 and an actual solution. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).

", "headline": "Accepted expected-projection sufficient condition for the original composition slope", "location": "Accepted 19 October review of P037-E001; adjudication.md, Complete expected-normal-equations proof" }, "id": "P037-T43", "text": { "body": "accepted 19 October review of E001; SI Eq. (2), p. 13, repeated pp. 15–16; main p. 568 (PDF p. 3). On any finite normalized design and complete four-school blocks, the stated raw first moments and mean-spanning controls imply E[r]=0, E[r't]=0, E[r'V]=0 and E[r'z]=E[r'r]. If E[t't] is nonzero and K=E[r'r]>0, every actual expected-normal-equation solution obeys b-beta=E[sum r_i T_i eta_i]/K; eta=0 implies b=beta for arbitrary fixed untreated levels. A four-state binary complementary two-of-four certificate has compositions 1/4 or 3/4, nonconstant untreated levels, E[t't]=1, K=1/16 and an actual solution. The usual measure-theoretic conditions are assumed (all spaces and sums here are finite).", "headline": "Accepted expected-projection sufficient condition for the original composition slope", "location": "Accepted 19 October review of P037-E001; adjudication.md, Complete expected-normal-equations proof" }, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T43" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "partial", "targets": [ { "agreement": "to_rounding", "location": "p. 568 (main PDF p. 3), school counts and legend endpoints only", "method": "I recalculated only the school counts (26 control, 23 treatment) and legend endpoints (13 events, school mean 0.54) on the public data using the authors' outcome definition. These agree to the displayed precision; the network drawing was not reproduced.", "status": "recalculated", "target": "Figure 1" }, { "agreement": "to_rounding", "location": "p. 569 (main PDF p. 4), panels A-D", "method": "I ran the authors' code with adaptations for the public data. The plotted lines and treatment bands agree at graph precision (P037-N010, P037-N011); the Results sentence's 0.08 and the description of the bands are separate corrections (P037-N010, P037-N026).", "status": "reproduced", "target": "Figure 2" }, { "agreement": "within_mc_error", "location": "p. 570 (main PDF p. 5); tests on p. 569 (PDF p. 4)", "method": "I ran the authors' code with adaptations for the public data and the historical R sampling convention. Points agree approximately at graph precision; intervals agree within Monte Carlo error conditional on the estimated exposure probabilities. Uncertainty from estimating those probabilities was not assessed.", "status": "reproduced", "target": "Figure 3" }, { "agreement": "disagrees", "location": "SI PDF p. 40", "method": "I ran the authors' code with adaptations for the public data and compared all 32 entries. After rounding to three decimals, 14 of the 16 means and 12 of the 16 standard errors differ from the printed table; the unrounded output agrees with Fig. 3 at the precision of its drawing (P037-E009). Outcome-specific samples also differ from the stated 2,451 students (P037-N025).", "status": "reproduced", "target": "Table S5" }, { "agreement": "within_mc_error", "location": "SI PDF p. 41", "method": "I ran the authors' code with adaptations for the public data, including 5,000 draws for each bootstrap. Effects and robust standard errors agree to two decimals; bootstrap dispersions agree within Monte Carlo error. The uniform-weight row is relabelled in P037-E005.", "status": "reproduced", "target": "Table S6" }, { "agreement": "disagrees", "location": "SI PDF p. 42", "method": "I ran the authors' code with adaptations for the public data and compared all 32 entries. Four coefficients or robust standard errors differ beyond rounding (P037-N011); changing bootstrap seeds cannot change those four entries.", "status": "reproduced", "target": "Table S7" }, { "agreement": "to_rounding", "location": "main PDF pp. 1, 4, 6; correction p. E3598 (PDF p. 1)", "method": "I ran the authors' code with adaptations for the public data. The numerical calculations reproduce 30%, 29%, 25%, 23%, the rounded effect and raw control mean, and the Discussion totals. The 25% uses the 49-school interaction model; the raw 23% and 0.20 use all 56 schools (P037-N001, P037-N009).", "status": "reproduced", "target": "Headline percentage and 2018 correction" }, { "agreement": "exact", "location": "p. 1066 (correction PDF p. 1)", "method": "I inspected the published correction and verified the grant identifier. It adds a funding acknowledgment; there is no numerical result to reproduce.", "status": "inspected_only", "target": "2019 correction" } ] }, "summary": "The article reports a field experiment in 56 middle schools with 24,191 students. In 28 randomly selected schools, randomly invited students were encouraged to take a public stance against conflict. The article reports fewer disciplinary reports of peer conflict and more influence from highly connected students. 34 entries are classified as errors (1 Major, 10 Moderate, 23 Minor), and 1 is a clarity note; 8 further concerns were not confirmed (one, \\fid{P037-E001}, is listed below) and 3 remain undetermined. The school-level HC0 normal tests reject too often in specified null experiments on the 49-school public-data designs; the study's actual error rate remains unknown (\\fid{P037-N006}). I did not confirm an identification error under the supplement's qualified response model; the counterexample limits what follows from randomization alone and does not establish empirical model failure.", "what_was_checked": "I list here what was read, what was recalculated and what held up. This is not a statement that the article is correct, and the absence of a correction elsewhere is not a finding of correctness.\n\n\\emph{What was read.} I read the article, the SI Appendix, the corrections of 2018 and 2019, the questionnaire, the two codebooks, the registration of 5 July 2013 and all eleven R scripts of the authors' analysis released with the study by ICPSR (ICPSR 37070).\n\n\\emph{What the recalculations use.} Recalculations from individual-level data use the public release (ICPSR 37070, version 2), run through the authors' scripts with adaptations for variable names, missing-value codes and R-version differences, or through a reconstruction using their definitions. The authors' own input files (two data files and two saved sets of simulated assignments) are not public, and ICPSR states that some variables of the release were changed. A value on the public data cannot be assumed to be the value that the authors' files would give, and the original version of the data and the original stream of random numbers cannot be checked.\n\n\\emph{What held up.}\n\\begin{itemize}\n\\item Counts of the design. The authors' rule for selecting records keeps 24,191 students in 56 schools on the public data, 28 treatment and 28 control schools in fourteen blocks of four, with 11,938 students in treatment schools and 728 invited seeds (20 to 32 in a school, 26 on average). Of the 728 seeds, 174 (23.9\\%) declined, which matches ``Twenty-four percent''. The attendance table of the supplement averages 61.7\\%, above the printed ``over 55\\%''.\n\\item The percentage after the correction of 2018. From the printed totals, I obtain $(2695-2012)/2695\\approx 25.3\\%$. On the public data the script of the correction returns 30\\% for the original rounded quotient and 28.7\\%, 25.4\\% and 22.9\\% for the three quotients that the correction reports as 29\\%, 25\\% and 23\\%. The adjusted 25\\% uses 49-school interaction-model predictions; those from the main-effects regression of Table S6, multiplied by 11,938, are 2,695.4 and 2,011.7. The raw 23\\% comparison uses all 56 schools; restricting it to the stated 49-school sample gives 13.6\\% on the public data (\\fid{P037-N009}). The correction of 2019 only adds a grant number.\n\\item Table S6. The four effects and their robust standard errors agree with the public data to two decimals ($0.087$, $-0.014$, $0.1647$, $-0.057$, with standard errors $0.018$, $0.078$, $0.011$, $0.0347$). On the public data the conflict effect gives a two-sided normal Wald $P=0.099$, consistent with the 2018 correction's nonsignificance statement (\\fid{P037-N008}). The two bootstrap rows agree within Monte Carlo error at 5,000 replicates. The regression is the one of SI equation (5); equation (6) adds the interaction used in Table S7. The label of the uniform-weight bootstrap is the subject of \\fid{P037-E005}.\n\\item Table S7. The columns for talking and for wristbands, and the effect on peer conflict at zero social referents, agree with the public data to two decimals, and the $P$ value of the interaction for peer conflict rounds to the printed $0.07$. Four entries do not agree: see below.\n\\item Exposure probabilities and the estimator of the network analysis. The exact expression $\\pi_i(d_k)=\\sum_{z}\\mathbf{1}[f(z,\\theta_i)=d_k]\\Pr(Z=z)$ is correct. The code uses 10,000 simulated assignments for the probabilities and the first 1,000 for the standard errors, as stated, and it weights by the inverse probability of the observed exposure. The printed formulas are corrected in \\fid{P037-E002} and \\fid{P037-E003}, and the simulated assignment of schools in \\fid{P037-N031}.\n\\item Within-school contrasts. On the public data the three wristband contrasts with the group exposed to a social referent seed have $P<0.00001$, the talking contrast with control has $P$ between $0.013$ and $0.017$ (article: $P<0.05$), the contrast in prescriptive norms between exposure to a social referent seed and no exposure has $P$ between $0.003$ and $0.005$ (article: $P<0.01$), and no contrast in disciplinary reports approaches significance. The contrast in prescriptive norms between referent and non-referent seed exposure has $P=0.0101$ with the code's draws and $P=0.0144$ with blocked draws on the public data (article: $P<0.01$). These simulations have Monte Carlo uncertainty and do not settle whether the original test was below $0.01$ (\\fid{P037-E009}, \\fid{P037-N031}).\n\\item Tables S3 and S4. All 48 loadings agree with the public data to two decimals, up to the overall sign of each component.\n\\item Other tables and figures of the supplement. In Table S1 the Total Sample row agrees with the mean of the 14 block means in all 15 columns, and the school-size column equals the block means of the school-size variable of the public data. Fig.~S2 has the 15 covariates of Table S1, one of them with $P<0.05$. The means of Table S2 that were compared equal the raw group means on the public data (its note is corrected in \\fid{P037-N037}). The observed differences of Fig.~S5 agree with the public data; its $P$ values depend on the random draws and were matched only within Monte Carlo error. On the public data 42.3\\% of the first-wave ties persist and the Jaccard index is 0.265 (published: 42.2\\% and 0.264).\n\\end{itemize}\n\n\\emph{What was reproduced, and what was not.} I ran the school-level and network scripts on the public data, with all 10,000 draws for the exposure probabilities, 1,000 for the network variances and 5,000 for each bootstrap. The reproduction is partial.\n\\begin{itemize}\n\\item In agreement on the public data: the percentage calculations of the article and of the correction of 2018, and the two implied totals of the Discussion (to rounding); Table S6 (within Monte Carlo error); the plotted lines and treatment bands of Fig.~2 at graph precision (their description is corrected in \\fid{P037-N026}); the points of Fig.~3 approximately, since its point estimates are not printed numerically, and its intervals within conditional Monte Carlo error, using the historical R sampling convention.\n\\item Not in agreement on the public data: the Results sentence at 20\\% social referents, where the fitted level is $0.095$ against the printed $0.08$ (\\fid{P037-N010}); the corresponding reduction is 57.3\\%, compatible with the printed 60\\% as a coarse approximation. The bands of Fig.~2 are not built as the legend says (\\fid{P037-N026}). Table S7 differs in four entries that no bootstrap seed can change (the two coefficients for prescriptive norms, the robust standard error of their interaction, and the interaction for peer conflict, $-0.84$ against $-0.85$; see \\fid{P037-N011}). In Table S5 several means do not round to the printed values and the four outcomes have samples of 2,457, 2,273, 2,847 and 2,495 students against the stated 2,451 (\\fid{P037-E009}, \\fid{P037-N025}).\n\\item Not reproduced: Fig.~1. No released script draws it and the coordinates of its networks are not in the release. Only its counts of schools (26 control, 23 treatment) and the endpoints of its legend (13 events for a student, a school mean of 0.54) were recalculated on the public data, and they agree.\n\\item Kept as in the released code, and not changed to match the article: the simulated assignment of schools without blocks, the threshold of 0.01 on the exposure probabilities, the single weighted regression with common slopes, the residual-based randomization fits with merged probability controls, and the formulas of the bands of Fig.~2 (\\fid{P037-N031}, \\fid{P037-N025}, \\fid{P037-N033}, \\fid{P037-N034}, \\fid{P037-N026}).\n\\end{itemize}\n\n\\emph{What was not checked.} The scripts for the blocking, the balance test of Fig.~S2 and the display of the design were read and not run in full: the public data report the school-level variables in intervals and lack the school file, the coordinates of the schools, exact ages and the masked ethnicity fields. No released script produces the attendance table, the rule that selects the seed-eligible students, the randomization within schools, the statistics on the stability of the network or the paired $t$ tests of Table S2. The Monte Carlo error of the network estimates that comes from the simulated exposure probabilities was not estimated. The cited works and most of the intervention curriculum were not read.\n\n\\emph{Lean.} The formalization for this work contains 202 entries covering 43 selected claims: 183 statements proved, 9 counterexamples proved, and 10 entries marked not applicable for printed empirical reports, procedure descriptions, captions and definitions. The completed proofs include the least-squares coefficient limit in the constructed example, the exact mean and variance of seed-group composition under the stratified draw, the oracle and contrast Gaussian limits for independent blocks, and the finite regression and variance identities for the wild bootstrap, leverage, HC0 shortfall and recentered refit. They also include the actual pooled and separate WLS minima in the twelve-unit example, the augmented-mean identities, and the general same-controls imputation and variance identities. The proofs also include a positive expected-projection result for fixed complete four-school blocks: it allows arbitrary fixed untreated levels and derives a treatment-moderation bias identity, with common treatment intercepts as a sufficient case. Its composition first moment and expected normal equations are explicit hypotheses. The proofs also cover the normalized bootstrap weight laws, the fitted regression limit for independent blocks, and further design, sample-selection, network and regression claims. The normalized-exponential-to-Dirichlet result is proved conditionally on a published gamma--Dirichlet theorem taken as an explicit hypothesis. Each status applies only to its individual formal statement.\n\nThe qualified conditional-response interpretation of the composition regression is not refuted by the counterexample to identification from randomization alone; its empirical adequacy remains unresolved. For the school tests, raw empirical OLS influence covariance equals HC0. The inspected \\href{https://github.com/DoubleML/doubleml-for-py/blob/6884b5f9816f180324d854a4d81b22e9d334f5e0/doubleml/utils/_estimation.py}{DoubleML nonclustered variance calculation} documents continued use of an unadjusted influence-score variance, without establishing its prevalence or calibration for these school regressions. The school tests reject too often in the specified public-data null experiments; those experiments do not determine the study's actual error rate. The correction retains HC0 and improves only displayed rounding; no calibrated replacement for the original school-level claims is established (\\fid{P037-N006}).", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "Elizabeth Levy Paluck", "Hana Shepherd", "P. M. Aronow" ], "category": "Articles - Empirics", "citation": "Elizabeth Levy Paluck, Hana Shepherd, and P. M. Aronow (2016). Changing Climates of Conflict: A Social Network Experiment in 56 Schools. PNAS 113(3):566–571.", "doi": "10.1073/pnas.1514483113", "edition": { "pages_in_audited_file": 6, "sha256": "572fb2641a05e041820d8881100b5348bdd222ae9c30b7916cb8e6eb38f44a31", "source_url": "https://doi.org/10.1073/pnas.1514483113", "venue": "PNAS", "year": "2016" }, "id": "P037", "later_materials": { "related_entries": [ "P037-E001", "P037-E002", "P037-E003", "P037-E004", "P037-E005", "P037-E006", "P037-E007", "P037-E008", "P037-E009", "P037-E010", "P037-N001", "P037-N003", "P037-N005", "P037-N006", "P037-N007", "P037-N008", "P037-N009", "P037-N010", "P037-N011", "P037-N013", "P037-N018", "P037-N019", "P037-N020", "P037-N021", "P037-N023", "P037-N025", "P037-N026", "P037-N027", "P037-N030", "P037-N031", "P037-N032", "P037-N033", "P037-N034", "P037-N035", "P037-N036", "P037-N037" ] }, "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P037/", "venue": "PNAS", "year": "2016" } }